Video-MME-Logical : A Controlled Diagnostic Benchmark for Video Temporal-Logical Reasoning

Benchmark at a Glance

Why Video-MME-Logical

Existing video benchmarks often conflate temporal-logical reasoning with general temporal understanding, scene recognition, or uncontrolled visual variation. This leaves three gaps: reasoning categories are often under-specified, difficulty is hard to interpret because it co-varies with natural-video complexity, and final-answer-only evaluation cannot verify whether a model follows the correct temporal evidence trace. Video-MME-Logical addresses these gaps with operation-centric tasks, controlled difficulty, and verifiable intermediate states.

Taxonomy

We organize Video-MME-Logical around five temporal-logical operations. State Tracking tests whether models maintain hidden or latent object states across visual transformations. Sequential Counting requires accumulating discrete evidence over time. Temporal Ordering asks models to recover the order of state changes, revealed symbols, or event sequences. Dynamic Spatiality evaluates geometric and motion-based inference, while Structural Composition requires composing spatial structures across viewpoints, occlusions, and partial observations.

Construction Pipeline

Each task category is implemented as an executable program with four components: temporal transition, scene configuration, metadata construction, and video rendering. The recorded metadata supports video generation, question construction, exact answer computation, difficulty control, and intermediate-state supervision. Easy, medium, and hard settings are defined by increasing temporal horizon and reasoning complexity.

Leaderboard

SFT Scaling

Supervised fine-tuning improves Video-MME-Logical performance with more generated data, but the gains saturate before closing the human gap. The thinking variant peaks at 39.2% overall accuracy with 375K samples and drops to 37.7% at 500K, while both SFT variants remain far below the 95.9% human reference.

Task Gallery

Each card shows one representative example from a fine-grained task category. Use the arrows to browse the examples in a single horizontal row.

Intermediate-State Diagnostics

Final-answer accuracy can hide intermediate-state failures. In Video-MME-Logical-S, models must output structured intermediate information in the same answer tag, and predictions are scored by exact match against program-recorded states. The qualitative example shows that a model can predict the correct final location while producing an incorrect swap trace, whereas a successful model must recover both the intermediate sequence and the final answer.

Key Findings

• Human performance reaches 95.9% overall accuracy, while the strongest evaluated zero-shot model, Gemini-3.1 Pro, reaches 28.6%.
• Controlled difficulty exposes sharp degradation: GPT-5.4 drops from 31.7% on easy tasks to 16.1% on hard tasks, while Gemini-3.1 Pro drops from 33.1% to 20.6%.
• Intermediate-state evaluation remains difficult: GPT-5.4 reaches 17.4% and Gemini-3.1 Pro reaches 10.8% on Video-MME-Logical-S, far below the 96.1% human reference.
• Explicit thinking does not consistently improve temporal-logical reasoning; the reasoning trace must remain grounded in the correct visual evidence.
• SFT scaling improves performance up to 39.2%, but the full 500K setting drops to 37.7%, suggesting that naive supervised scaling saturates.

Citation

@misc{videommelogical2026,
  title = {Video-MME-Logical: A Controlled Diagnostic Benchmark for Video Temporal-Logical Reasoning},
  author = {Kwan, Hohin and Li, Hongyu and Zhang, Ray and Zhang, Manyuan and Kong, Xianghao and Rao, Anyi and Xie, Jiahao and Liu, Si},
  year = {2026},
}