Substream Recollection

Code for our Substream Recollection paper. It runs the streaming-recall eval on open-weight checkpoints and computes per-response FLOPs. The sequence-model training code and the synthetic-video generator that built the dataset are here too. The figure-generation code is not part of this release.

@inproceedings{anonymous2026substream,
  title     = {Substream Recollection: A Controlled Benchmark for Long-Context
               Video and Sequence Streaming},
  author    = {Anonymous},
  booktitle = {Under review},
  year      = {2026}
}

Setup

Two virtualenvs, because MiniCPM-V 2.6 pins an older Transformers/torch that conflicts with everything else.

git clone https://github.com/anonstreammem/beyond-long-context.git
cd beyond-long-context

# full-stack: every model except MiniCPM-V 2.6
python3.11 -m venv .venv-full-stack
source .venv-full-stack/bin/activate
pip install -e ".[full-stack]"
deactivate

# hf450: MiniCPM-V 2.6 only (--model minicpm)
python3.11 -m venv .venv-hf450
source .venv-hf450/bin/activate
pip install -e ".[hf450]"
deactivate

The two extras live in pyproject.toml. flash-attn comes in as a prebuilt wheel; if your CUDA/torch doesn't match the pinned wheel, edit the URL there before installing. Activate the matching env before a run.

Don't set CUDA_VISIBLE_DEVICES yourself. The wrappers hand the model to accelerate and expect to see every GPU, so partition at the SLURM/container level (--gres=gpu:N).

Data

On HuggingFace. Mirror it locally:

huggingface-cli download anonstreammem/substream-recollection \
    --repo-type dataset --local-dir ./data

Four configs: text/, synthetic_video/, natural_video/, easyhuman/. Each has a manifest.json that main.py reads directly (and flat questions.parquet/questions.json copies if you only want to inspect rows). Buckets are keyed by carrier length and entropy band, e.g. UNIFORM_EVAL_L008_ELOW; natural-video uses nat_<N>_frames. Point --asset-root at the config directory so the relative clip paths resolve. The streaming datasets API won't hand you the on-disk video files the wrappers open with decord/torchvision, so use the download above for real runs.

Quick start

Three questions on the shortest bucket with InternVL3.5 (8B), ~2 min on 2x A100/H100 once the checkpoint is cached:

source .venv-full-stack/bin/activate
python main.py ./data/synthetic_video/manifest.json \
    --asset-root ./data/synthetic_video \
    --input-mode video --eval-mode spatial \
    --model internvl-3-5 \
    --enable_metrics --verbose \
    --question-log-csv ./logs/smoke/iv8b_L008_ELOW.csv \
    --state-file       ./logs/smoke/iv8b_L008_ELOW.state \
    --max_frames 4100 --max_tokens 4096 \
    --limit 1 --limit_questions 3 \
    --bucket-filter UNIFORM_EVAL_L008_ELOW \
    --resume-state

Running an evaluation

A run is a manifest, a --model, and one or more --bucket-filter. State and CSV are written incrementally; --resume-state continues a stopped run. --bucket-filter is repeatable and matches any manifest path containing it as a substring.

Video, single bucket:

python main.py ./data/synthetic_video/manifest.json \
    --asset-root ./data/synthetic_video \
    --input-mode video --eval-mode spatial --model glm45v \
    --question-log-csv ./logs/glm45v/L032_ELOW.csv \
    --state-file       ./logs/glm45v/L032_ELOW.state \
    --bucket-filter UNIFORM_EVAL_L032_ELOW \
    --max_frames 4100 --max_tokens 4096 --resume-state

Text-token streams instead of video:

python main.py ./data/text/manifest.json \
    --asset-root ./data/text \
    --input-mode sequence --eval-mode sequential --sequence-format comma \
    --model qwen3_full \
    --question-log-csv ./logs/qwen3vl8b/seq_L256.csv \
    --state-file       ./logs/qwen3vl8b/seq_L256.state \
    --bucket-filter UNIFORM_EVAL_L256_ELOW \
    --max_tokens 4096 --resume-state

The CSV is append-only, one row per (video, question): video_id, question_id, video_entropy, correct_answer, model_answer, plus a few derived columns. The state file is JSON keyed by bucket:video_index; resuming skips the pairs it already has.

python main.py --help has the rest. --qwen3-thinking switches Qwen3-VL to the Thinking variant. --restart_on_oom (with --max_oom_retries) re-inits and retries the current video on a CUDA OOM. --max_gpu_mem overrides the per-GPU checkpoint budget in GiB. --qwen3-omni-fast-processor uses the fast Qwen3-Omni processor where the build ships it.

Supported models

Model	--model	Wrapper
Qwen2.5-VL (7B)	qwen_full	models/qwen2_5_vl.py
Qwen3-VL (8B), --qwen3-thinking for the Thinking variant	qwen3_full	models/qwen3_vl.py
Qwen3-Omni (30B-A3B)	qwen3_omni	models/qwen3_omni.py
GLM-4.5V (104B-A12B)	glm45v	models/glm45v.py
InternVL3.5 8B / 30B-A3B / 38B (+ -thinking for each)	internvl-3-5, internvl-3-5-30b-a3b, internvl-3-5-38b	models/internvl_3_5.py
MIMO-VL (7B)	mimo-vl	models/mimo_vl.py
Phi-4-MM (6B)	phi_multimodal	models/phi_4_mm.py
MiniCPM-V 4.5 (9B)	minicpm-4-5	models/minicpm_v_4_5.py
MiniCPM-V 2.6 (8B), needs .venv-hf450	minicpm	models/minicpm_v_2_6.py
LongVILA (7B)	longvila	models/longvila.py

M3-Agent (m3_agent) and TimeChat-Online (timechat) have wrappers too. They run but weren't part of the reported numbers.

FLOPs

flops_estimator/ predicts per-response FLOPs in closed form (matmul-only), one function per model. Constants come from each model's HuggingFace config.json, checked against the reference modeling code. No polynomial fits.

from flops_estimator.flops_all_models import MODEL_FUNCTIONS, DISPLAY_NAMES, _get_total

frames = [{"height": 448, "width": 448}] * 8
fn = MODEL_FUNCTIONS["glm45v"]
result = fn(frames, n_in_text_tokens=128, n_out_text_tokens=64)
print(DISPLAY_NAMES["glm45v"], _get_total(result))

_get_total(result) is the total (keyed total or total_flops per model). The dict also has the per-stage breakdown and *_elementwise mirrors. flops_estimator/README.md has the counting convention and the per-model caveats. The one that bites: Qwen3-VL and GLM-4.5V silently downsample frames past a few hundred.

Layout

Path
main.py	the eval driver -- CLI, state manager, per-mode dispatch
models/	one file per checkpoint family, over HuggingFace/vLLM
frame_samplers/
processors/	question / sequence / video preprocessing
datasets/	manifest loader
metrics/	curve fitting, FLOPs helpers
flops_estimator/	the predictor above
extras/	multi-question + anomaly drivers
tests/
external/	vendored LongVILA / TimeChat bits
utils/	GPU monitor, memory, logging, frames
training/	backbone training (below)
data_gen/	video generator (below)

Training the backends

training/ is the training setup for the decoder-only QA models with the memory backends from the paper. One main.py entrypoint drives PyTorch Lightning over synthetic or file-backed data, with helper scripts in training/scripts/. Install from training/pyproject.toml first. training/README.md covers dataset prep, the backend registry, and the flags.

Generating the videos

data_gen/ renders the controlled-entropy video buckets from template YAML scenes and rule-based vocabularies (RGB tensors or MP4s), via a main.py driver and a seq2vid package. See data_gen/README.md for the demo and the bucket plans in data_gen/configs/.

License

MIT, see LICENSE.