VideoSSM: Autoregressive Long Video Generation with Hybrid State-Space Memory

Yifei Yu*, Xiaoshan Wu*, Xinting Hu, Tao Hu, Yangtian Sun, Xiaoyang Lyu, Bo Wang, Lin Ma, Yuewen Ma, Zhongrui Wang, Xiaojuan Qi
The University of Hong Kong · PICO, ByteDance · Southern University of Science and Technology
* Equal contribution

arXiv · 2025

TL;DRA local attention cache preserves recent detail while recurrent state-space memory carries evolving context beyond the window.

Original overview: hybrid memory, long-video examples and training. Figure 1 of the paper.
Original overview: hybrid memory, long-video examples and training. Figure 1 of the paper.Click to enlarge ↗ Original PDF ↗

Overview

VideoSSM augments autoregressive diffusion with a hybrid memory: a sliding-window cache for local detail and an evolving state for longer-range context. It supports minute-long generation and prompt switching without retaining an ever-growing history of keys and values.

Method

Remember the past. Keep the scene moving.

01

Local cache

Recent tokens retain fine motion and appearance.

02

Global state

Evicted tokens update a compact recurrent memory.

03

Adaptive fusion

A router combines local and global information.

Hybrid memory module. Original Figure 5.
Hybrid memory module. Original Figure 5.Click to enlarge ↗

Results

Consistency without freezing the video.

MetricSelf ForcingLongLiveVideoSSM
Subject consistency88.2591.0992.51
Background consistency91.7393.2393.95
Dynamic degree35.0037.5050.50

Selected metrics from Table 2, evaluated on 60-second videos. Higher is better. See the paper for the complete comparison and evaluation protocol.

Read the experimental protocol ↗

BibTeX

@misc{yu2025videossm,
  title = {VideoSSM: Autoregressive Long Video Generation with Hybrid State-Space Memory},
  author = {Yifei Yu and Xiaoshan Wu and Xinting Hu and Tao Hu and Yangtian Sun and Xiaoyang Lyu and Bo Wang and Lin Ma and Yuewen Ma and Zhongrui Wang and Xiaojuan Qi},
  year = {2025},
  eprint = {2512.04519},
  archivePrefix = {arXiv},
  primaryClass = {cs.CV},
  url = {https://arxiv.org/abs/2512.04519}
}