Where Omni-Reason succeeds
- Restoration: 30/31 Full tasks.
- Geometry: 18/21.
- Cross-modal: 9/11.
These are published task-level results, not a claim that every sample is available as a video.
Compare capability families →Bench
Evaluating generalist video models across diverse computer vision tasks. What does a successful operation look like?
Choose a task from the list to inspect its benchmark sample.
Omni-Reason-Bench evaluates whether a generated video completes the requested visual operation. Full, Mini, and Additional use different samples, denominators, and comparison scopes, so their scores are reported separately.
Full170 tasks · 850 samples · 8 models · task pass rate
Mini141 tasks · 141 samples · 12 models · sample pass rate
Additional170 task types · 849 samples · transfer beyond project fine-tuning
01
Condition
Provide the input video, instruction, and reference when the task is RV2V.
02
Generate
The model produces a target video intended to complete the requested operation.
03
Judge
Task Core Criteria and the automatic judge assess functional completion.
04
Aggregate
Full requires at least 3 of 5 passing samples; Mini uses one sample per task.
A sample passes when its main functional outcome is useful and mostly complete. Identity, counts, action order, and correspondence matter. PSNR, SSIM, LPIPS, and CLIP similarity are reported separately and do not replace task completion.
Choose a task in the explorer to open its sample-level viewer in this area. The comparison keeps Input and Ground Truth tied to the verified sample manifest; Model Outputs are shown whenever a video path is available for that task and sample.
Source bindingOnly a matched task/sample output can support a comparison. Missing links do not establish that an output file does not exist.
101 / 170Full task passes in the November 1 manuscript; separate from archive playback availability.
Historical Omni-Reason outputs that cannot be paired to a current Full170 sample are not placed beside unrelated Input or Ground Truth videos.
Where it struggles
- View: 4/12.
- Future and conditioned action: 5/16.
A plausible-looking view can still lose object relations; tracking can mark points while losing identity.
Read the limitations →Additional reports transfer relative to the project fine-tuning roster: 16/170 task types passed against the fixed planned roster. Of 849 planned samples, 827 are evaluable across 166 tasks; 22 samples are unsupported. The manuscript does not establish that these tasks were unseen during base pretraining, so this is not labeled strict zero-shot or out-of-distribution generalization.
Automated judges and Task Core Criteria support scale, but they do not remove evaluator sensitivity. Models may also receive different inputs when a reference is unavailable. Some prediction videos are not present in the current release; that affects evidence availability, not the November 1 manuscript’s 101/170 Full score.