I study how we can create and understand the 3D world. My research spans generative 3D modeling, digital humans, and neural rendering.
I am a Ph.D. candidate at the Multidisciplinary Artificial Reality Studio (MARS), ShanghaiTech University, advised by Prof. Jingyi Yu and Prof. Lan Xu.
SATO generates artist-style meshes with a sequence representation based on triangle strips. Connected faces and explicit UV boundaries help preserve coherent edge flow and meaningful surface partitions. The same representation can produce triangular or quadrilateral meshes, allowing training to combine large triangle datasets with more structured quad examples.
GenPIE reconstructs light transport across space, viewing direction, and time from sparse transient observations. A reconfigurable capture system separates illumination from detection, while generative geometry priors and inverse transient rendering recover regions that are difficult to observe. The resulting representation supports applications such as separating light bounces, correcting time distortion, and changing illumination over time.
TAPESTRY: From Geometry to Appearance via Consistent Turntable Videos
Yan Zeng, Haoran Jiang, Kaixin Yao, Qixuan Zhang, Longwen Zhang, Lan Xu, Jingyi Yu
Research overview
TAPESTRY gives untextured 3D meshes coherent appearance through geometry-conditioned turntable videos. Rendered geometric features guide video generation so that the object remains consistent across viewpoints. A subsequent reconstruction process fills self-occluded areas and transfers the generated appearance into textures or neural representations, connecting video synthesis with complete 3D asset creation.
BANG decomposes 3D objects by generating a sequence of progressively separated parts. A pretrained 3D diffusion model, an exploded-view adapter, and temporal attention produce coherent geometry throughout this process. Spatial prompts let users choose regions to separate. The resulting components support detailed refinement, interactive editing, and workflows that print and reassemble individual parts.
CAST: Component-Aligned 3D Scene Reconstruction from an RGB Image
Kaixin Yao, Longwen Zhang, Xinhao Yan, Yan Zeng, Qixuan Zhang, Lan Xu, Wei Yang, Jiayuan Gu, Jingyi Yu
Research overview
CAST reconstructs a scene as individually editable 3D objects from one RGB image. It combines image segmentation, depth cues, and object relationships with generation that accounts for occlusion. Objects are then aligned to the observed scene, and a physics-aware refinement adjusts their positions to reduce floating geometry and intersections. The reconstructed scenes can support simulation and interactive applications.
Instant Gaussian Splatting Generation for High-Quality and Real-Time Facial Asset Rendering
Dafei Qin, Hongyang Lin, Qixuan Zhang, Kaichun Qiao, Longwen Zhang, Jun Saito, Zijun Zhao, Jingyi Yu, Lan Xu, Taku Komura
Research overview
TransGS converts physically based facial assets into GauFace, a structured Gaussian representation designed for efficient animation and rendering. A diffusion transformer processes patches of the asset, using UV information and aligned sampling to preserve fine visual detail. This approach connects existing facial asset workflows with interactive rendering across desktop, mobile, and virtual-reality platforms.
CLAY generates 3D geometry and physically based materials from text, images, and spatial guidance. A geometry autoencoder and diffusion transformer learn shape priors, while a separate material model produces coordinated texture maps. Controls including multiple views, point clouds, and bounding boxes let users steer the output, from early design ideas to detailed assets that can be rendered under different lighting.
Media2Face: Co-speech Facial Animation Generation With Multi-Modality Guidance
Qingcheng Zhao, Pengyu Long, Qixuan Zhang, Dafei Qin, Han Liang, Longwen Zhang, Yingliang Zhang, Jingyi Yu, Lan Xu
Research overview
Media2Face creates expressive 3D facial animation guided by speech and additional text or image cues. A generalized facial representation separates identity from expression and helps extract motion from videos to build a diverse, annotated training dataset. A diffusion model then generates facial motion in that representation, allowing variations in emotion and style alongside audio-driven performance.
GaussianHair: Hair Modeling and Rendering with Light-aware Gaussians
Haimin Luo, Min Ouyang, Zijun Zhao, Suyi Jiang, Longwen Zhang, Qixuan Zhang, Wei Yang, Lan Xu, Jingyi Yu
Research overview
GaussianHair models individual hair strands as connected cylindrical Gaussian primitives. This representation combines explicit strand structure with efficient differentiable rendering, while a dedicated scattering model captures how hair responds to light. The recovered geometry and appearance support changes to hairstyle, illumination, and motion, linking image-based reconstruction with conventional graphics workflows.
DreamFace: Progressive Generation of Animatable 3D Faces under Text Guidance
Longwen Zhang, Qiwei Qiu, Hongyang Lin, Qixuan Zhang, Cheng Shi, Wei Yang, Ye Shi, Sibei Yang, Lan Xu, Jingyi Yu
Research overview
DreamFace turns written descriptions into personalized 3D faces with geometry, materials, and animation controls. A progressive process refines facial shape and synthesizes appearance using diffusion models. The assets share a topology compatible with established graphics tools, while learned expression priors add person-specific deformation for animating the generated characters.
HACK models the head and neck together, connecting visible appearance and facial expression with the underlying cervical skeleton. Its training data combines ultrasound measurements with multi-view photometric capture. The learned representation separates shape, pose, expression, and larynx motion, giving artists anatomical controls for more expressive head-and-neck animation and motion transfer.
ChatAvatar: Creating Hyper-realistic Physically-based 3D Facial Assets through AI-Driven Conversations
Qixuan Zhang, Longwen Zhang, Lan Xu, Di Wu, Jingyi Yu
Research overview
ChatAvatar creates detailed 3D facial assets through conversational text input. Diffusion-based generation and a dataset of production-ready faces support the creation of geometry and physically based materials. The workflow makes character creation accessible through natural-language descriptions while producing assets intended for conventional computer-graphics pipelines.
SCULPTOR jointly models facial appearance and the skeleton underneath it. Built from paired CT observations in the LUCY dataset, its learned representation connects skull structure, facial geometry, and appearance. Separating shape, pose, and expression enables the creation and editing of faces with attention to both visual realism and anatomical consistency.
This work learns production-oriented facial assets from video, combining dynamic geometry with physically based textures. Separate generative models represent expression, geometry, and appearance while shared mappings keep them coordinated. Fine wrinkle detail supports animated materials, and the resulting assets enable performer capture, motion transfer across identities, and explicit editing within established rendering workflows.
NIMBLE represents the hand through its bones, muscles, and skin rather than surface shape alone. Learned from MRI observations and appearance capture, the model relates internal anatomy to visible deformation and texture. It supports realistic hand animation and rendering, and can also provide synthetic examples or a differentiable representation for estimating hands from images.
Neural Video Portrait Relighting in Real-time via Consistency Modeling
Longwen Zhang, Qixuan Zhang, Minye Wu, Jingyi Yu, Lan Xu
Research overview
This work changes the lighting of portrait video while preserving consistency across frames. A neural model separates structure from illumination and learns from a dynamic dataset with individually controlled lights. Semantic, temporal, and lighting constraints help maintain stable appearance during motion, supporting interactive portrait relighting, including on mobile devices.