LCVD - A Lighting Controllable Portrait Animation Generation Framework Launched by Sichuan University

LCVD (Lighting Controllable Video Diffusion Model) is a high-fidelity, lighting-controllable portrait animation generation framework introduced by Sichuan University. LCVD is based on the separation of intrinsic features (such as identity and appearance) and extrinsic features (such as pose and lighting) of portraits. The reference adapter and shading adapter map these features into different subspaces. During the animation generation process, LCVD combines these feature subspaces and finely controls the lighting effects based on a multi-condition classifier-free guidance mechanism, preserving the identity and appearance of the portrait. The model is based on a stable video diffusion model (SVD), generating high-quality portrait animations that are consistent with the driving video's pose and conform to the target lighting conditions. LCVD significantly outperforms existing methods in terms of lighting realism, image quality, and video consistency, providing strong technical support for fields such as virtual reality, video conferencing, and film production.

Main Functions of LCVD

Portrait Animation: Transforms static portraits into dynamic videos, matching the head movements and expressions in the driving video.

Lighting Control: Re-lights the portrait during the animation generation process according to user-specified or reference image lighting conditions.

Identity and Appearance Preservation: Maintains the identity and appearance features of the portrait during animation and re-lighting, avoiding loss of identity information.

High-Quality Video Generation: The generated videos excel in lighting realism, image quality, and video consistency, suitable for scenarios such as virtual reality, video conferencing, and film production.

Technical Principles of LCVD

Feature Separation: The Reference Adapter maps the intrinsic features (identity and appearance) of the reference portrait to the feature space. The Shading Adapter maps the extrinsic features (lighting and pose) to the feature space. Based on the separation of intrinsic and extrinsic features, the model independently controls lighting and pose during the animation process.

Lighting Controllable Diffusion Model: Based on the Stable Video Diffusion Model, it uses multi-condition classifier-free guidance to adjust lighting effects. By modifying the  guidance strength (such as weight w), it enhances or weakens the influence of lighting cues, achieving fine lighting control.

Motion Alignment and Long Video Generation: Based on the motion alignment module, it ensures that the generated portrait is consistent with the pose of the driving video. Using the diffusion model sampling method, it generates videos of arbitrary lengths, ensuring smooth transitions between video segments based on an overlapping strategy. It ensures consistency in lighting, pose, and identity in the generated videos.

Training and Optimization: During the training phase, self-supervised learning optimizes the adapters and diffusion model using loss functions (such as LPIPS, FID, etc.) to evaluate and optimize the quality of the generated videos.

Project Address of LCVD

arXiv Technical Paper: https://arxiv.org/pdf/2502.19894

Application Scenarios of LCVD

Virtual Reality (VR) and Augmented Reality (AR): Creates realistic virtual characters that naturally integrate with virtual or real scenes.

Video Conferencing: Real-time generation of high-quality portrait animations, reducing bandwidth requirements and enhancing user experience.

Film Production: Quickly generates portrait animations that conform to different lighting conditions, used for special effects and virtual scenes.

Game Development: Generates realistic virtual character animations, enhancing the realism and immersion of games.

Social Media and Content Creation: Supports users in generating personalized dynamic avatars or short videos, enriching content creation forms.