Specify the operation in text.
TV2V
Input video and instruction define the task. The model generates the requested visual result.
Can video generation become a general interface for computer vision?
A visual answer can be a segmentation map, a moving skeleton, or a restored scene. Omni-Reason studies whether these different operations can be learned through one video-generation interface.
The research idea
Express tasks as conditional video transformations, train a shared model across them, and evaluate whether the requested operation is completed.
Based on the November 1 manuscript. Explore the method · Examine the evidence
The answer does not have to look like a natural video. Segmentation produces category labels; skeleton annotation produces joints and bones; restoration produces a recovered scene. Each can be represented as a sequence of visual frames.
Input video + instruction + optional references → output video
References add visual information when the instruction alone does not specify the desired object, appearance, or view.
| Input | Requested operation | Visual answer |
|---|---|---|
| A scene video | Label the visible semantic categories. | A frame-aligned segmentation video. |
| A person moving | Annotate the body’s skeleton. | The video with joint and bone annotations. |
| A video + a view reference | Reproduce the action from the specified view. | A video of the action from that viewpoint. |
Add a visual reference.
RV2V
An image or video provides extra conditioning—for example, a desired viewpoint or the identity of points to track.
Supervision becomes generation.
Target versus output
The dataset target supplies the desired answer during training. At inference, the model generates an output; it is not given that target answer.
Omni-Reason-1 starts from a video-only checkpoint converted from Lightricks’ LTX-2.3 open weights. Its approximately 13.125B-parameter video Transformer is fine-tuned across tasks, which use one shared set of trained weights.
- Represent the input compactly.A video encoder (VAE) turns the input and visual references into latents: smaller numerical representations of the frames. The instruction provides the requested operation.
- Generate an answer under those conditions.The Transformer starts with random target noise and progressively transforms it into a target latent, using the input, instruction, and optional references as conditions.
- Turn the answer back into frames.The VAE decoder converts the generated target latent into the visible output video.
This is the inference path. The target video is supervision during training, not an input supplied to answer a new example.
Give each input a clear role.
Design 01 · Identify
The model needs to distinguish the source scene, references, and target being generated. It also needs to know which inputs correspond spatially and temporally.
Typed embeddings
Role, slot, alignment, and modality embeddings mark these distinctions. Aligned inputs share coordinates; unaligned inputs use separate coordinate offsets.
Let the answer read the evidence.
Design 02 · Condition
The target can attend to the input conditions. The attention pattern prevents noisy target tokens from feeding back into the condition tokens.
Directed attention
Condition-to-target information flow is directional: target tokens read conditions, while condition tokens cannot attend to the noisy target. This separates the supplied evidence from the evolving answer.
Train on the requested answer.
Design 03 · Learn
The learning objective applies to target tokens. Training examples from different tasks update the same model parameters.
Target-only flow matching
The model learns the target velocity used to transform noise into the desired target latent, conditioned on the clean inputs. Conditional flow-matching loss is computed only on target tokens.
Initialization and model provenance
The upstream LTX-2.3 development model is described as 22B; Omni-Reason uses an approximately 13.125B converted video Transformer. These are different model scopes. The manuscript documents the converted checkpoint, but the original conversion execution was not retained for full numerical-equivalence verification.
Omni-Reason-Data contains 30,655,520 canonical input–target pairs across 340 task types, nine capability families, and 34 sub-capabilities. The collection includes 282 TV2V and 58 RV2V task types. The fine-tuning corpus is a selected subset of this collection.
Specify the operation
01 · Define
Human task design establishes what the input provides, what should change, what must remain consistent, and what counts as success.
Build paired examples
02 · Construct & review
Pipelines convert existing sources; generators render controlled examples. Both produce the shared format and undergo sample validation before scaling.
Prepare the model corpus
03 · Select & encode
170 selected task types—146 TV2V and 24 RV2V—yield 2,308,875 encoded training records. These records are distinct from the full collection of 30.66M canonical pairs.
Training first establishes a 100-task model, expands the mixture to 170 tasks, then refines that shared model. The final checkpoint is used for both the main comparison and transfer evaluation.
100 tasks
Stage 1
1.241M encoded records · 32,000 iterations · batch size 16. The stage endpoint passes 68 main tasks and 10 additional tasks.
170 tasks
Stage 2
2.309M encoded records · 60,000 iterations · batch size 16. The expanded model passes 98 main tasks and 16 additional tasks.
Final refinement
Stage 3
10,000 iterations · batch size 32. The final Omni-Reason-1 checkpoint passes 101 main tasks and 16 additional tasks.
| Training stage | Main / 170 | Held-out / 170 |
|---|---|---|
| Initial · 100 training tasks | ||
| Expanded · 170 training tasks | ||
| Final refinement |
Stage endpoints from Figure 6. Main coverage grows most during expansion; held-out coverage reaches 16 and shows limited further gain. These are three endpoints, not the full 14-checkpoint trajectory.
The paper follows 14 checkpoints along the final model’s training path. A separate 50K / 100K / 200K experiment finds lower validation loss with more training data, but also more compute; it does not isolate a data-only scaling law.
A sample passes when the requested functional outcome is useful and mostly complete. A Full or Additional task requires at least three passing samples. Similarity metrics complement this judgment; a visually similar output is not sufficient evidence of correct execution.
| Setting | Scope | Final result | Interpretation |
|---|---|---|---|
| Full | 170 tasks × 5 samples · 8 models | 101 / 170 tasks · 59.4% | Seen task types; evaluation samples separate from training. Strongest baseline: MiniMax H3, 19 / 170 (11.2%). |
| Mini | 141 fixed common samples · 12 models | 84 / 141 samples · 59.6% | Availability-selected comparison including proprietary systems. Strongest baseline: Kling O3 Standard, 34 / 141 (24.1%). |
| Additional | 170 planned task types · 849 samples | 16 tasks passed | Excluded from project fine-tuning; same weights and inference settings. 166 tasks / 827 samples have evaluable outputs; 22 samples are unsupported. |
Breadth: Omni-Reason-1 leads all nine families in Full and eight in Mini. Full restoration reaches 30 / 31 tasks; geometry reaches 18 / 21. Mini future prediction is led by Grok Imagine Video (42.9% versus 35.7%).
Transfer: the 16 additional successes comprise eight restoration/completion tasks, five editing tasks, two future/action tasks, and one identity/motion task. “Excluded from project fine-tuning” does not establish that the pretrained base model never encountered related data.
Protocol: GPT-6 Astra judges functional completion. Omni-Reason uses CFG 1.5 and 50 inference steps. Full informed development and branch selection. Some baselines cannot consume every reference input; the paper records these input-adaptation limits. Human–judge agreement results are not provided on this page.
Source: manuscript Table 1, Sections 5.1–5.3 and Appendices E–F. The Leaderboard uses these manuscript results. Videos in Bench retain their earlier release identities and are not evidence of this final output batch.
Restoration & completion
8 tasks
Half of the additional successes fall in this family.
Content & appearance editing
5 tasks
Transfer includes changing appearance under new task definitions.
Future, action & identity
3 tasks
Two future/action tasks and one identity/motion task extend the repertoire beyond restoration and editing.
Reference-guided colorization, wildlife identity masks, and action-conditioned continuation are reported examples. The 16 passed task types comprise 15 TV2V and one RV2V task. This is task-type separation from project fine-tuning, not a claim of no related pretraining exposure.
Source: Section 5.2 and Appendix F.2. Of 170 planned tasks (849 samples), 166 tasks (827 samples) have evaluable outputs. No cross-model Additional ranking is reported.
Remove the overlay. Keep the building.
Selective editing · reported success
Request: remove the holographic blueprint from a construction video.
Observe: Omni-Reason retains the building and crane. The displayed Hunyuan OmniWeaving output removes those physical structures too.
Lesson: deletion alone is insufficient; unrelated content must survive.
Close S2. Light B2.
Rule-guided execution · reported success
Request: generate the bulb states implied by the circuit switch operation.
Observe: B2 lights while six already lit bulbs stay on.
Lesson: success depends on the circuit’s relationships and the correct state change.
The viewpoint changes. The stacks break.
View transformation · reported failure
Request: reproduce a stacking action from the reference head-camera view.
Observe: blue and yellow blocks remain separate from their intended red and orange supports.
Lesson: a plausible view can lose the action’s relational outcome.
Text summaries of manuscript Figure 7, not the earlier Mini clips shown on the homepage. In the separate bell-on-scale example, the model moves the bell onto the platform, but complete gripper release remains unclear.
A broad repertoire does not mean reliable execution on every task or every sample.
Structure can break
Precision
View synthesis passes 4 / 12 Full tasks, future/action 5 / 16, and constraint reasoning 5 / 17. Fine geometry, object relations, and action order remain challenging.
Evidence is uneven
Transfer
Additional successes concentrate in restoration and editing. There is less evidence for transfer in geometry, viewpoint transformation, and constraint reasoning.
Read the scope
Evaluation
Full is development-informed, Mini is availability-selected, and Additional includes unsupported outputs. These settings do not establish universal out-of-distribution competence.
Paper: November 1 manuscript · PDF forthcoming.
Published video examples: earlier Mini release. Final evaluation: November 1 manuscript, Table 1 and Appendices E–F.
2026 research manuscript · PDF forthcoming.