EchoReel: Enhancing Action Generation of Existing Video Diffusion Models

Paper Code arXiv
Jianzhi Liu  Junchen Zhu  Lianli Gao  Jingkuan Song 
University of Electronic Science and Technology of China
"Imitation is the sincerest form of flattery that mediocrity can pay to greatness." — Oscar Wilde


Abstract

Recent large-scale video datasets have facilitated the generation of diverse open-domain videos of Video Diffusion Models (VDMs). Nonetheless, the efficacy of VDMs in assimilating complex knowledge from these datasets remains constrained by their inherent scale, leading to suboptimal comprehension and synthesis of numerous actions. In this paper, we introduce EchoReel, a novel approach to augment the capability of VDMs in generating intricate actions by emulating motions from pre-existing videos, which are readily accessible from databases or online repositories. EchoReel seamlessly integrates with existing VDMs, enhancing their ability to produce realistic motions without compromising their fundamental capabilities. Specifically, the Action Prism (AP), is introduced to distill motion information from reference videos, which requires training on only a small dataset. Leveraging the knowledge from pre-trained VDMs, EchoReel incorporates new action features into VDMs through the additional layers, eliminating the need for any further fine-tuning of untrained actions. Extensive experiments demonstrate that EchoReel is not merely replicating the whole content from references, and it significantly improves the generation of realistic actions, even in situations where existing VDMs might directly fail. Code and website are available in the supplementary material.



Overview

MY ALT TEXT

Visual Results

Input Text VideoCrafter2 + EchoReel Input Text VideoCrafter2 + EchoReel
"A man is studying in the library" "A man is repairing a car"
"A man is skiing" "A boy plays guitar by the campfire"
"A man is running" "A farmer is harvesting crops"
"Couple walking on the beach" "A child is blowing bubbles"
"A man is carving a stone statue" "A chef making a cake"
"A man is painting a mural" "Looking at diamonds through a magnifying glass"
"A man looking into the distance on the top of a mountain" "A group of people playing beach volleyball"
"A painter is painting" "A female worker is using a sewing machine"
"A man is diving" "A person is making pottery"
"A plane is flying in the sky" "Standing alone in the sunset"

BibTeX


      @article{Liu2024EchoReel,
        title={EchoReel: Enhancing Action Generation of Existing Video Diffusion Models}, 
        author={Jianzhi Liu, Junchen Zhu, Lianli Gao, Jingkuan Song},
        year={2024},
        eprint={2403.11535},
        archivePrefix={arXiv},
      }