Video Understanding and Generation

Created on June 28, 2026

2026   ·   open_set_recognition   action_understanding   video_generation

Trustworthy Video Generation
Recent advances in generative AI have enabled the synthesis of increasingly realistic visual content. However, existing models often struggle to generate outputs that are physically plausible, temporally consistent, and faithfully capture the underlying dynamics and interactions of the real world. Our research investigates how to integrate physical principles together with spatial and temporal knowledge into generative models, enabling the generation of reliable, controllable, and physically consistent visual content for human-centered AI and embodied intelligence.

    Publications
    1. Pu, Y., Cheng, Z., Gong, S., Zhao, H., & Kong, Y. (2026). LogicDirector: Enforcing Temporal Composition in Text-to-Video Generation. Neural Information Processing Systems (NeurIPS).
      2. Project
    2. Pu, Y., & Kong, Y. (2026). NoisEasier: Test-Time Noise Optimization for Text-to-Video Generation. European Conference on Computer Vision (ECCV).
      3. Project
    3. Pu, Y., Huang, Z., Boddeti, V., & Kong, Y. (2026). Show Me: Generating Instructional Videos with Diffusion Models. Winter Conference on Applications of Computer Vision (WACV).
      4. PDF Project


Open-world Action Recognition
How can AI robustly understand unseen human activities?


Open-set recognition problem assumes that there are many unknown classes in the testing data, which are not appearing in the training data. From a feature space perspective, closed-set recognition pre-define a fixed list of classes in training, and all we need to learn is the class boundaries separating these classes. While for open-set recognition, the list of classes are not fixed, and new classes may appear in testing. We hope to learn tight boundaries for known training classes so that we can separate them from new unknown classes in testing. [Figure credit: Scheirer, et. al., TPAMI-12]

    Publications
    1. Bao, W., Li, K., Patel, D. A., Chen, Y., & Kong, Y. (2025). Exploiting VLM Localizability and Semantics for Open Vocabulary Action Detection. Winter Conference on Applications of Computer Vision (WACV). PDF
    2. Bao, W., Chen, L., Huang, H., & Kong, Y. (2024). Prompting Language-Informed Distribution for Compositional Zero-Shot Learning. European Conference on Computer Vision (ECCV). PDF
    3. Dengxiong, X., & Kong, Y. (2023). Ancestor Search: Generalized Open Set Recognition via Hyperbolic Side Information Learning. IEEE/CVF Winter Conferences on Applications of Computer Vision (WACV), 4003–4012. PDF
    4. Bao, W., Yu, Q., & Kong, Y. (2022). OpenTAL: Towards Open Set Temporal Action Localization. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2969–2979. PDF
    5. Neupane, K. P., Zheng, E., Kong, Y., & Yu, Q. (2022). A Dynamic Meta-Learning Model for Time-Sensitive Cold-Start Recommendations. Association for the Advancement of Artificial Intelligence (AAAI), 7868–7876. PDF
    6. Bao, W., Yu, Q., & Kong, Y. (2021). Evidential Deep Learning for Open Set Action Recognition. Proceedings of the IEEE International Conference on Computer Vision (ICCV), 13349–13358. PDF


Human Activity Understanding
How to learn robust representations of human activities to support recognition, localization, and anticipation?

Human activities are inherently complex, exhibiting large variations in appearance, motion, temporal dynamics, and environmental context. Learning robust representations of human activities is therefore fundamental to enabling AI systems to recognize, localize, and anticipate human actions across diverse real-world scenarios. Our research investigates representation learning methods that capture the spatial, temporal, and semantic structures of human activities, providing a foundation for a wide range of applications, such as human-robot interaction, entertainment, and autonomous driving vehicles.

    Publications
    1. Kong, Y., & Fu, Y. (2022). Human Action Recognition and Prediction: A Survey. International Journal of Computer Vision (IJCV), 130(3), 1366–1401. PDF
    2. Chen, J., Mittal, G., Yu, Y., Kong, Y., & Chen, M. (2022). GateHUB: Gated History Unit with Background Suppression for Online Action Detection. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 19893–19902. PDF
    3. Kong, Y., Tao, Z., & Fu, Y. (2020). Adversarial Action Prediction Networks. IEEE Transaction on Pattern Analysis and Machine Intelligence (T-PAMI), 42(3), 539–553. PDF
    4. Kong, Y., Tao, Z., & Fu, Y. (2017). Deep Sequential Context Networks for Action Prediction. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 1473–1481. PDF
    5. Kong, Y., & Fu, Y. (2016). Max-Margin Action Prediction Machine. IEEE Transaction on Pattern Analysis and Machine Intelligence (T-PAMI), 38(9), 1844–1858. PDF
    6. Kong, Y., Kit, D., & Fu, Y. (2014). A Discriminative Model with Multiple Temporal Scales for Action Prediction. European Conference on Computer Vision (ECCV), 596–611. PDF
    7. Kong, Y., Jia, Y., & Fu, Y. (2014). Interactive Phrases: Semantic Descriptions for Human Interaction Recognition. IEEE Transaction on Pattern Analysis and Machine Intelligence (T-PAMI), 36(9), 1775–1788. PDF
    8. Kong, Y., Jia, Y., & Fu, Y. (2012). Learning Human Interaction by Interactive Phrases. European Conference on Computer Vision (ECCV), 300–313. PDF
    9. Kong, Y., & Fu, Y. (2017). Max-Margin Heterogeneous Information Machine for RGB-D Action Recognition. International Journal of Computer Vision (IJCV), 123(3), 350–371. PDF
    10. Kong, Y., & Fu, Y. (2015). Bilinear Heterogeneous Information Machine for RGB-D Action Recognition. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 1054–1062. PDF