hiw3c 日报精选
arXiv cs.AI/CL/LG 网页快照

单一先验,双重动态:将视频扩散变换器重新用于自监督潜在动作学习

原文标题 · One Prior, Two Dynamics: Repurposing Video Diffusion Transformers for Self-Supervised Latent Action Learning

作者: Yuxin Jiang, Wenqiang Sun, Jiaqi Li | 视频揭示了世界如何变化,却很少展示导致这些变化的行为。潜在动作学习能从视觉转换中恢复此类监督,但往往从零开始就学习底层动态。我们探讨预训练视频生成器能否转而为逆向和正向动态提供先验。我们介绍O$^{2}$LaF,其构建基于一个简单的原则:一个先验,

来源:arXiv cs.AI/CL/LG · arxiv.org

正文为英文,可一键机器翻译(仅首次需要等待)

Computer Science > Computer Vision and Pattern Recognition

Title: One Prior, Two Dynamics: Repurposing Video Diffusion Transformers for Self-Supervised Latent Action Learning

Abstract: Videos reveal how the world changes, but rarely the actions that cause those changes. Latent action learning recovers such supervision from visual transitions, yet often learns the underlying dynamics largely from scratch. We ask whether a pretrained video generator can instead provide the prior for both inverse and forward dynamics. We introduce O$^{2}$LaF, built on a simple principle: one prior, two dynamics. We initialize both models from the same pretrained video DiT and adapt them to their distinct roles: a feed-forward transition encoder with learnable action queries for inverse dynamics, and a causal, action-conditioned video generator for forward dynamics. By controlling information access and optimizing both dynamics end-to-end through a compact bottleneck, O$^{2}$LaF learns transferable actions from unlabeled videos. Across navigation and robotic manipulation, O$^{2}$LaF sets a new state of the art in few-shot action probing and enables stronger zero-shot action transfer across visual contexts and embodiments. Beyond generative video priors, we further extend our framework to predictive representation priors such as V-JEPA2, achieving competitive performance. Our findings suggest a broader paradigm for latent-action learning: repurposing pretrained video models to discover transferable actions and predict their consequences.

Submission history

Access Paper:

Additional Features

Current browse context:

References & Citations

BibTeX formatted citation

Bookmark

Bibliographic and Citation Tools

Code, Data and Media Associated with this Article

Demos

Recommenders and Search Tools

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .