Human Motion and Interaction Representation Learning

Created on June 28, 2026

2026   ·   human_motion   physics-informed_modeling

Learning physics-aware human motion is pivotal for advancing the fields of video understanding, prediction, and synthesis. By capturing intrinsic human physics, motion representations enable machines to interpret complex human behaviors effectively, anticipate future actions accurately, and generate realistic motion sequences. These representations integrate hierarchical and relational dynamics, facilitating more robust generalization across diverse activities and contexts. Consequently, physics-aware motion modeling not only enhances human-computer interaction but also holds promise for applications such as robotics, virtual reality, and assistive technologies.

The goal of this research is to learn a precise, human-centric motion representation from noisy videos. We have developed a self-supervised approach H-MoRe [3] for 2D human-centric motion, and H-Flow [1] for 3D human-centric motion learned from videos. These methods have been applied to downstream tasks including action recognition, gait recognition, and video generation.



Dense Human 3D Flow (NeurIPS 2026)
From a single monocular video, H-Flow recovers dense per-point human motion in 3D that generic scene flow misses.
Input Video
H-Flow (Ours) [1]
ZeroMSF



2D Dense Human-centric Flow (CVPR-25 Highlight)
H-MoRe extracts human-centric motion from videos, while generic optical flow is distracted by the background.
Input Video
H-MoRe (Ours) [3]
SEA-RAFT



    Publications
    1. Zhanbo Huang, Xiaoming Liu, and Yu Kong. H-Flow: Self-supervised Human Scene Flow via Physics-inspired Joint Multi-modal Learning. Neural Information Processing Systems (NeurIPS), 2026. PDF, Project
    2. Zhanbo Huang, Dingqiang Ye, Xiaoming Liu, and Yu Kong. Unlocking Motion from Large Vision Models with a Semantic and Kinematic Duality for Gait Recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2026 (Highlight). Project
    3. Ji, Y., Shen, Y., Zhu, S., Kong, Y., & Liu, F. (2026). From Pose to Prose: Explainable, Personalized Fitness Coaching with Biomechanical Context. IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
    4. Zhanbo Huang, Xiaoming Liu, Yu Kong. Learning Human-centric Motion Representation for Action Analysis. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025 (Highlight). Project