omnireasonDESIGN PREVIEW
OMNI-REASON / A VISUAL REASONING BENCHMARK

A world to see.
A world to reason about.

OmniReason.

Beyond recognizing a frame.
Understanding what moves, what changes,
and what comes next.

340 tasks 9 capability families One atlas

THE WORLD BEHIND THE WORDS IS ALREADY MOVING.INPUT → TARGET · REAL TASK EXAMPLES

A WORLD OF VISUAL QUESTIONS

Follow a question
through the world.

A ball changes direction. A camera moves through space. Weather grows into a storm. Explore the scenes and transformations that make each task distinct.

ONE WORLD. A DIFFERENT WAY TO SEE IT.
SWIPE TO EXPLORE01 / 32
INPUT What the model seesWhat the task asks for TARGET
INPUT
TARGET
Drag the line. Reveal the transformation.
TASK 47 / UNDERSTAND SPACE

See the depth in a scene.

Turn each frame into a map of near and far, without losing the scene’s geometry.

DRAG THE LINE TO COMPARE · SWIPE THE CARD TO EXPLORE
MINI INPUT → TARGET · NOT A MODEL OUTPUT

32 windows into the atlas. Swipe in either direction — the last flows back to the first.
Open any task to inspect its evidence.

02 / FROM PIXELS TO STRUCTURETRACKING · GEOMETRY · VIEWPOINT

Not just pixels.
Paths. People. Space.

Follow the same points through time.
Recover the structure behind what you see.

Released Mini task examples · short browsing excerpts · targets are not model predictions.

THE OMNI-REASON TASK MAP

One model.
Nine capabilities.

Omni-Reason organizes 340 video tasks into nine capability families. Each node opens that family in the task wall, so you can move from the overview straight into examples.

340tasks in the atlas
9capability families
170evaluated tasks

Choose a family to filter the task wall.

Counts show tasks in the atlas. 170 of the 340 tasks are evaluated in Full170.

02 / CHANGE THE SCALEMEDICINE · WEATHER · ASTRONOMY

From a
heartbeat
to a star.

A moving body. An evolving storm.
A different view of the Sun.
Reasoning reaches beyond the everyday.

And this is only the beginning ↓

02 / THE COMPLETE TASK UNIVERSE

340 tasks.
Every point opens a world.

Nine capability families. One connected view.
Pick a point, or search the complete directory.

340 TASKS / 9 FAMILIES
Drag to rotate · select a point to inspectCAPABILITY-GROUPED LAYOUT

Colors indicate the published capability families. Point positions are a browsing layout, not a measure of similarity, difficulty or model performance. This is the frozen 340-task atlas; 170 tasks have Full170 model outputs.

Browse and search all 340 tasks +

THE EXAMPLES EXPLAIN THE QUESTION. THE OUTPUTS SHOW THE RESPONSE.

Look closer.
Judge for yourself.

Explore the original model outputs with their quality notes and input-protocol disclosures. A playable result is not necessarily a correct one.

RESEARCH / THE PAPER

Omni-Reason:A General Video Model That Could Solve 100 Computer Vision Tasks

THE QUESTION / THE APPROACH / THE EVIDENCE

Can one video model carry out very different visual operations by following a task-specific instruction? Omni-Reason studies this with a shared video-to-video interface: provide an Input, a Prompt and sometimes a Reference; the model returns a video response to compare with the task’s Target. This page walks through the task design, dataset, model, evaluation and what the reported results can—and cannot—establish.

340task definitions in the full dataset34 sub-capabilities · 9 families
30.65Mcanonical Input–Target pairsMedian 5,634 pairs per task
2.31Mencoded training recordsPrepared from the selected 170-task subset
170 × 5Bench170 held-out examples850 samples across all 9 families

Keep the inventories distinct. The full dataset has 340 task definitions. Bench170 selects 170 tasks and five held-out samples per task. The paper’s Mini100 setting uses one sample from each of 100 tasks; the separate Table 3 pass-rate report uses 141 fixed common samples. These are different evaluation scopes.

Paper PDF soonDataCode
01 / DATA

A common task language

Convert diverse visual operations into Input, Prompt, optional Reference and Target records, while retaining the rules that define a correct answer.

02 / MODEL

One video generator

Give each conditioning stream an explicit role and temporal relationship so the same transformer can learn different operations.

03 / EVALUATION

Task-specific evidence

Study whether an output performs the requested operation and preserves the evidence it depends on, alongside visual-similarity measurements.

Figure 1: Omni-Reason task map. Nine capability families connect representative visual tasks around a shared video generation formulation, covering 340 tasks and 30.65 million sample pairs.
Figure 1. Omni-Reason: Unifying computer vision through video generation. Across nine capability families, Omni-Reason-Data casts 340 visual tasks into a shared video-to-video formulation.

How to read Figure 1. The colored branches group representative operations into nine capability families. The thumbnails illustrate task types; they are not model comparisons. The center summarizes the shared video-to-video framing and dataset scale, not an evaluation score.

ABSTRACT

Can one video model perform many different computer-vision tasks when told what to do in a prompt? Omni-Reason studies this by expressing each task as a video-generation problem: provide the visual evidence and instruction, then generate a video that demonstrates the requested operation. The challenge is temporal and relational. A useful answer may need to preserve an object, follow the same point through motion, change a viewpoint or show a plausible future state.

TASK FORMAT

What the model receives and generates

For each task, the model receives an Input video and a task-specific Prompt. Some tasks also provide a Reference image or video for extra guidance. The model generates a video response, which can then be compared with the task’s Target. For example, “unfold this paper in reverse order” and “track this exact point” use the same input-and-prompt format but ask for different visual operations.

InputWhat the model sees+PromptWhat it must do+ReferenceOptional guidance→Generated videoThe model’s response

The Target is the expected answer used for training or comparison; it is not given to the model as an input.

The research question. Can one model follow different instructions while respecting the specific objects, relationships and changes in each input video? The benchmark makes that question concrete by defining what the model sees and what a correct video should show for each task.

WHY VIDEO-TO-VIDEO?

Many vision tasks already describe a transformation between visual states: restore a damaged frame, track an object through time, change a viewpoint or predict what happens next. A generated video can represent these answers while preserving their temporal structure.

WHAT MAKES IT HARD?

The same output format hides different rules. Tracking must preserve identity on the Input timeline; prediction must continue beyond it; editing must change only what the Prompt allows. A plausible-looking clip can still violate the task.

DATA / TASK INTERFACE & COVERAGE

Each task pairs a video with an instruction and an expected result. The collection contains 340 task definitions; Bench170 selects 170 tasks and five held-out examples per task. Some tasks use only a video and prompt, while others also provide a reference. The Target is the expected result used for comparison, not an input to the model.

Paper Figure 2: TV2V paper folding and RV2V surface-point tracking, with their Input, Prompt, optional Reference and Target roles.
Figure 2 / A shared interface, different temporal relations. Paper unfolding generates a later interval; point tracking aligns outputs to the Input timeline. The shown Target sequences are task examples, not model predictions.

How to read Figure 2. In the paper-unfolding example, the Target is a later interval: the model must reverse the fold order and preserve the punched holes. In point tracking, a first-frame Reference identifies which point to follow, while the Target marks that same point frame by frame. A Reference specifies the query; it does not provide the future trajectory.

Paper Figure 3: dataset capability taxonomy, observed domains, sample-count distribution, TV2V/RV2V task mix, and task versus sample-pair shares.
Figure 3 / Dataset composition and scale. The draft reports 340 tasks and 30.65M canonical pairs; the selected train/benchmark scope contains 170 tasks (146 TV2V, 24 RV2V), a 2.31M-record training corpus, and 850 held-out samples (170 × 5).

How to read Figure 3. Panel (a) names the nine families and 34 sub-capabilities; its equal-angle sectors are a taxonomy, not proportions. Panel (b) counts scene domains, which can overlap because one task may cover several domains. Panel (c) shows a long-tailed sample distribution: the median task has 5,634 canonical pairs, while a small number of tasks contribute many more. Panels (d) and (e) separate task counts from sample-pair counts. The 170-task subset is 146 TV2V tasks and 24 RV2V tasks; its 850 benchmark examples are separate from the 2.31M-record training corpus.

WHY SELECT 170 TASKS?

The experimental subset prioritizes visual operations and inference while accounting for duration and inference cost. Tasks dominated by textual explanations, or too costly for this round, remain in the full collection. The selected subset spans all nine families and 31 of the full collection’s 34 sub-capabilities.

FULL DATASET / BENCH170

Coverage by capability family

Counts use each task’s primary family. The benchmark column is a subset of the full task directory.

CAPABILITY FAMILYFULL · 340 TASKSBENCH170
Semantic perception and understanding4218
Identity, motion and interaction5128
Spatial structure and geometry4021
View synthesis and transformation2012
Visual restoration and completion5031
Content and appearance editing2916
Future prediction and conditioned action3916
Constraint reasoning, assessment and correction3517
Cross-modal and representation translation3411

Family counts sum to 340 tasks in the full collection and 170 in Bench170. They count task definitions, not sample pairs or model outputs.

THE TASK DIRECTORY340 tasks

The complete task collection. The current public showcase has 1,697 task examples; tasks do not all have the same number of examples.

THE TRAINING CORPUS~2.31M records

Encoded training records prepared from the selected 170 tasks. These records are not the held-out benchmark examples.

THE BENCH170 SET850 samples

Five held-out examples for each of 170 tasks. This is the fixed evaluation scope described in the paper.

THE FULL170 VIDEO MANIFEST5,950 outputs

Seven open-source model IDs × 850 samples. The Main170 metric report has eight rows; the report and video manifest are separate inventories.

01 / TASK COVERAGE

Nine capability families

The full collection spans 34 sub-capabilities and 13 observed domains. One hundred tasks span at least two domains. Figure 3 reports task share and sample-pair share separately; these distributions are not interchangeable. A large sample share means more examples, not more task types.

02 / CONSTRUCTION ROUTES

Pipeline + Generator

Pipeline converts and standardizes existing sources. Generator creates paired examples from specified scenes, parameters and rules. Both routes produce the shared task format.

METHOD / MODEL

Omni-Reason uses a video diffusion transformer to generate a response video from the task instruction. It can condition on an input video and, when the task needs it, a reference image or video. The model is trained to distinguish these input streams from the video it must generate, so one model can handle different visual operations.

Paper Figure 5: video DiT architecture with role, alignment, modality and position embeddings, masked visual self-attention, and bimodal logit-normal noise sampling.
Figure 5 / Model architecture and noise schedule. Typed structural conditioning marks each visual token’s role, temporal alignment, modality and input slot. Bimodal sampling allocates 15% to low-noise detail updates and 85% to the global denoising trajectory.

How to read Figure 5. On the left, Role, Alignment, Modality and Position tell the video transformer what each visual stream represents and how it relates in time. This matters when, for example, a reference image identifies an object but is not synchronized with every generated frame. On the right, the two noise peaks combine broad generation steps (85%) with low-noise refinement (15%) for spatially precise details such as masks, lines and boxes.

01 / INPUT STRUCTURE

Label every visual stream

Each visual token is marked by its role (source, reference or target), temporal alignment, modality (video or mask) and slot. The draft model accepts at most one source video and up to three references.

02 / CONDITIONING

Use evidence without corrupting it

Directed attention allows source and reference conditions to inform one another and the generated target. Noisy target tokens cannot flow back into the clean condition representations.

03 / VIDEO GENERATION

Generate across scales of detail

Flow matching drives the video transformer from noise toward a target video. A 15% low-noise component gives additional training attention to precise structures such as masks, thin lines and boxes.

Time and viewpoint matter. Alignment-aware 3D positional encoding distinguishes a source synchronized with the Target from an unaligned Reference. This lets a task say whether the output should correspond frame by frame or depict a later interval.

Training setup in the draft. The paper describes a 13.1B-parameter, 48-layer video DiT trained on one AWS trn2.48xlarge node with 16 Trainium2 chips, using TP4 × FSDP16 parallelism. These are the reported research-training details, not hardware requirements for running the website or released inference code.

RESULTS / TASK COMPLETION + VISUAL SIMILARITY

We report two kinds of evidence. Task-pass judgments ask whether the requested operation was completed; full-reference metrics measure resemblance to the Target. They have different scopes and should be read separately.

FULL BENCHMARK / OMNI-REASON-157.1%

Full Task Pass Rate · 170 tasks × 5 samples. A task passes when at least 3 of its 5 samples pass.

MINI COMMON SET / OMNI-REASON-158.9%

Sample Pass Rate · 141 fixed common samples, one sample per task. This is not the Mini100 similarity table.

See the full-model breakdown and Mini task-completion scope
Separate reports and versions. The Main170 automatic-metrics report covers 170 tasks × 5 samples and eight model rows. It names its model row “Omni-Reason A10K”; Table 3 names the task-completion model “Omni-Reason-1”, and their checkpoint mapping is unspecified. The paper draft’s Mini100 tables for automatic similarity and human preference contain no numeric results. The 141-sample Table 3 Mini result and the website’s Mini100 score file are separate inventories.
PSNR ↑21.42 dBhigher is better
MSE ↓18.368 × 10⁻³lower is better
SSIM ↑0.7557higher is better
MS-SSIM ↑0.7960165 tasks / 825 samples
LPIPS ↓0.2937lower is better
CLIP ↑0.9131higher is better
READING THE RESULT

The A10K row leads all six similarity columns in the supplied Main170 report. This supports better agreement with the reference videos on that report’s scope. A generated video can still miss a task constraint, so completion judgments and example inspection remain necessary.

Explore model comparisons and report versions

Metric coverage. Five metrics cover 170 tasks / 850 samples; MS-SSIM covers 165 tasks / 825 samples.

Read the direction. Higher is better for PSNR, SSIM, MS-SSIM and CLIP. Lower is better for MSE and LPIPS.

ANALYSIS / WHAT THE SCORES SAY

The six metrics measure different kinds of visual similarity between each generated video and its Target. They help compare results at scale, but none directly checks whether the model followed every part of a task instruction. Read the scores as measures of visual similarity, within the stated task and sample scope.

PSNR · MSE

Measure pixel-level differences. They are sensitive to exact color and position changes between generated and Target frames.

SSIM · MS-SSIM

Compare image structure at one or multiple scales. MS-SSIM is available for 165 of the 170 tasks in this report.

LPIPS · CLIP

Compare learned visual features. LPIPS is a perceptual distance; CLIP is a feature-similarity score. Neither checks the task instruction directly.

What the metrics can miss. A small spatial shift can lower pixel and structure scores even when an answer is plausible. Conversely, a visually similar clip can still track the wrong object or violate a prompt constraint. Read the table as one view of visual fidelity; it does not directly measure task completion.

What to keep in mind. A score can reward visual resemblance while missing a task detail, such as following the same physical point through an occlusion or reversing every paper fold in the correct order. The metric table should therefore be interpreted as evidence about visual similarity, not as a direct measure of instruction following.

Two result inventories. The Main170 metric report contains eight model rows, including Omni-Reason. The published Full170 video manifest contains 5,950 outputs across seven open-source model IDs. The report rows and video files are not yet bound to one another record by record.

FINDINGS / WHAT THE MODEL LEARNS

The paper’s task-by-task review shows why a single aggregate score is incomplete: performance varies with the kind of visual operation, the need to ground language, and how much identity or physical structure must persist over time.

108tasks reviewed manually in the draft

Reviewers described roughly 60 as solved, about 15 as partly solved, and the remainder as failed. These approximate qualitative counts are not the formal Bench170 task-pass rate.

STRONGER IN THE REVIEW

Dense, aligned visual operations

Reviewers often found restoration, enhancement, interpolation and dense prediction more reliable when the task had a direct visual target, such as dehazing, depth, masks or point tracking. This is a qualitative pattern, not a per-family benchmark score.

RECURRING FAILURE

Meaning and identity

Text and symbolic overlays may appear in the right place but contain meaningless labels. Language-grounded tasks can act on the wrong object, while long clips expose identity switches, forgotten objects and broken object–hand relationships.

OPEN CHALLENGE

Reason over change

Human-action continuation and multi-step physical relationships remain difficult in the reported review. Plausible motion or a familiar scene is not enough when an answer must preserve order, geometry, identity or a causal relation.

TRAINING INSIGHT / DRAFT ANALYSIS

The draft compares three peak learning rates using the same data order and seed. It reports no setting that wins on every task: operations that preserve most of the source can favor a smaller update, while masks and depth require a larger change in output representation. The draft also observes interference between tasks with similar output forms. These observations motivate studying data mixtures and task-specific behavior beyond an aggregate score.

What the evidence does not establish. The review is a draft qualitative analysis of 108 tasks, not an exhaustive independent audit of all 340 definitions. Five samples per benchmark task provide a limited view of each task. Pixel similarity, visual plausibility, automated acceptance and task completion are distinct forms of evidence.

VALIDATION / DATA & EVALUATION

The data pipeline reviews examples before scaling up and reviews a sample again after generation. The report also records automated acceptance checks for paired results. Passing those checks confirms that records met the reported checks; it does not prove that a generated video completed its task correctly.

Paper Figure 4: data sources pass through Pipeline or Generator, shared sample formatting, validation Gate B, full generation, review Gate C and separate training and benchmark outputs.
Figure 4 / Build, review, then split. Pipeline standardizes existing data; Generator creates samples from task definitions. Both routes pass through review before data is assigned to training or benchmark use.

How to read Figure 4. Follow either construction route into the shared Input / Prompt / optional Reference / Target record. Gate B checks a small validation set before full generation; Gate C samples the scaled outputs for review. Only after these steps are records assigned to training or benchmark use. “Sample-disjoint” means the benchmark examples are kept separate from training examples.

  1. 01

    Define the task

    Specify what the model receives, what the prompt asks it to do and what a correct Target should show.

  2. 02

    Review a small batch

    A task Owner and an independent Human Evaluator review validation examples at Gate B.

  3. 03

    Review after generation

    At Gate C, reviewers inspect samples from the scaled output set for quality and task fit.

  4. 04

    Hold out benchmark samples

    The 850 benchmark samples are separate from the 2.31M-record training corpus.

Report snapshot: 6,800 / 6,800 automated acceptance checks passed. The report notes that 48 paired results were recomputed and the original results retained. Passing acceptance checks means the supplied report accepted those records; it is not a human judgment of task success.

TASKS / VIDEO DEMOS

Each row pairs an Input with its expected Target; some tasks also include a visual Reference. These clips illustrate the task definition and supervision format. Targets are not Omni-Reason predictions.

INPUT
TARGET
REFERENCE
No reference
INPUT
TARGET
REFERENCE
Reference image for Task 322 · Reference 2
INPUT
TARGET
REFERENCE
No reference
INPUT
TARGET
REFERENCE
No reference

Copy the BibTeX entry.

@misc{omni_reason,
  title  = {Omni-Reason: A General Video Model That Could Solve 100 Computer Vision Tasks},
  author = {Wang, Maijunxian and Zhao, Zehong and Gao, Qingying and Jiang, Zhengze and Zhang, Yilan and Qin, Huayi and An, Siyuan and Ju, Hanxuan and Wu, Yuhan and Wang, Yue and Huang, Junruo and Zhao, Lunyu and Zhao, Xin and Liu, Huayu and Li, Yihan and Wu, Jiarui and Xiao, Xintong and Wu, Shuqi and Liu, Jiajun and Luo, Evan and Zhou, Yikai and Song, Ziyan and Zhang, Haotian and Biran, Yahav and Gandhi, Kavish and Singh, Adarsh and Gray, John and Agha-Ebrahim, Armin and Burtoft, Jim and Luo, Dezhi and Yu, Fengyuan and Ji, Ran and Deng, Hokin}
}

OMNI-REASON / PEOPLE & INSTITUTIONS

Contributors

Omni-Reason is built by a research team spanning computer vision, video generation and evaluation. Authors are listed in paper order. * Project Lead.

33paper authors
12affiliated institutions

01 / THE RESEARCH TEAM

Authors

Superscript numbers map each author to an institution below.

  1. Maijunxian Wang1,*
  2. Zehong Zhao2,*
  3. Qingying Gao3,*
  4. Zhengze Jiang4,*
  5. Yilan Zhang5,*
  6. Huayi Qin6
  7. Siyuan An7
  8. Hanxuan Ju6
  9. Yuhan Wu3
  10. Yue Wang7
  11. Junruo Huang7
  12. Lunyu Zhao8
  13. Xin Zhao7
  14. Huayu Liu9
  15. Yihan Li7
  16. Jiarui Wu7
  17. Xintong Xiao7
  18. Shuqi Wu8
  19. Jiajun Liu7
  20. Evan Luo7
  21. Yikai Zhou4
  22. Ziyan Song9
  23. Haotian Zhang10
  24. Yahav Biran11
  25. Kavish Gandhi11
  26. Adarsh Singh11
  27. John Gray11
  28. Armin Agha-Ebrahim11
  29. Jim Burtoft11
  30. Dezhi Luo12
  31. Fengyuan Yu6
  32. Ran Ji2
  33. Hokin Deng7

02 / AFFILIATIONS

Institutions

Universities and organizations represented in the paper.

1University of California, Berkeley2University of California San Diego3Johns Hopkins University4Columbia University5University of California, Los Angeles6Nanyang Technological University7Carnegie Mellon University8Duke University9University of California, Irvine10University of Southern California11Amazon Web Services12University of Michigan
OMNI-REASON / TASK EXPLORER