Motion-I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modeling

被引:4
|
作者
Shi, Xiaoyu [1 ]
Huang, Zhaoyang [2 ]
Wang, Fu-Yun [1 ]
Bian, Weikang [1 ]
Li, Dasong [1 ]
Zhang, Yi [1 ]
Zhang, Manyuan [1 ]
Cheung, Ka Chun [3 ]
See, Simon [3 ]
Qin, Hongwei [4 ]
Dai, Jifeng [5 ]
Li, Hongsheng [1 ,6 ]
机构
[1] CUHK, Multimedia Lab, Shenzhen, Peoples R China
[2] Avolution AI, London, England
[3] NVIDIA AI Technol Ctr, Shenzhen, Peoples R China
[4] SenseTime, Hong Kong, Peoples R China
[5] Tsinghua Univ, Beijing, Peoples R China
[6] Shanghai AI Lab, Ctr Perceptual & Interact Intelligence CPII, Shanghai, Peoples R China
来源
PROCEEDINGS OF SIGGRAPH 2024 CONFERENCE PAPERS | 2024年
基金
国家重点研发计划;
关键词
Diffusion models; Image animation;
D O I
10.1145/3641519.3657497
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
We introduce Motion-I2V, a novel framework for consistent and controllable text-guided image-to-video generation (I2V). In contrast to previous methods that directly learn the complicated image-to-video mapping, Motion-I2V factorizes I2V into two stages with explicit motion modeling. For the first stage, we propose a diffusionbased motion field predictor, which focuses on deducing the trajectories of the reference image's pixels. For the second stage, we propose motion-augmented temporal attention to enhance the limited 1-D temporal attention in video latent diffusion models. This module can effectively propagate reference image features to synthesized frames with the guidance of predicted trajectories from the first stage. Compared with existing methods, Motion-I2V can generate more consistent videos even in the presence of large motion and viewpoint variation. By training a sparse trajectory ControlNet for the first stage, Motion-I2V can support users to precisely control motion trajectories and motion regions with sparse trajectory and region. This offers more controllability of the I2V process than solely relying on textual instructions. Additionally, Motion-I2V's second stage naturally supports zero-shot video-to-video translation. Both qualitative and quantitative comparisons demonstrate the advantages of Motion-I2V over prior approaches in consistent and controllable image-to-video generation. Please see our project page at https://xiaoyushi97.github.io/Motion-I2V/.
引用
收藏
页数:11
相关论文
共 42 条