TimelineBench 1.0
Can agents cut a finished video from raw footage and a client brief?
- Tasks
- 56
- Collections
- 4
- Setups
- 19
- Slots (19 × 56)
- 1,064
- 169 of 1,064 resolved
- 15.9%[13.7, 18.2]
- Video editors
- 43
- Blind judgments
- 2,589
| Rank | Model | Harness | Resolved | |||
|---|---|---|---|---|---|---|
| 1 | Claude Opus 5.5Anthropic | Claude Codeguidance off | 32.1% [20.3, 46.0] | 18/56 | 49/56 | — |
| 2 | GPT-6 AstraOpenAI | Codexguidance curated | 26.8% [15.8, 40.3] | 15/56 | 52/56 | 24.4% |
| 3 | GPT-6.1 SolOpenAI | Codexguidance off | 25.0% [14.4, 38.4] | 14/56 | 48/56 | — |
| 4 | Claude Opus 5Anthropic | Claude Codeguidance off | 23.2% [13.0, 36.4] | 13/56 | 50/56 | 23.2% |
| 5 (tied) | GPT-6 AstraOpenAI | Codexguidance off | 21.4% [11.6, 34.4] | 12/56 | 48/56 | 23.8% |
| 5 (tied) | GPT-6 AstraOpenAI | OpenCodeguidance off | 21.4% [11.6, 34.4] | 12/56 | 48/56 | 22.3% |
| 5 (tied) | Claude Sonnet 5.5Anthropic | Claude Codeguidance off | 21.4% [11.6, 34.4] | 12/56 | 50/56 | — |
| 8 (tied) | Claude Fable 5.1Anthropic | Claude Codeguidance off | 17.9% [8.9, 30.4] | 10/56 | 49/56 | 23.5% |
| 8 (tied) | Claude Fable 5.1Anthropic | OpenCodeguidance off | 17.9% [8.9, 30.4] | 10/56 | 50/56 | 22.9% |
| 8 (tied) | Claude Opus 5Anthropic | OpenCodeguidance off | 17.9% [8.9, 30.4] | 10/56 | 46/56 | 19.0% |
| 8 (tied) | GPT-5.6 SolOpenAI | OpenCodeguidance off | 17.9% [8.9, 30.4] | 10/56 | 45/56 | 17.6% |
| 12 | Grok 4.6xAI | OpenCodeguidance off | 12.5% [5.2, 24.1] | 7/56 | 39/56 | 17.0% |
| 13 (tied) | Gemini 3.8 FlashGoogle | OpenCodeguidance off | 10.7% [4.0, 21.9] | 6/56 | 42/56 | 13.1% |
| 13 (tied) | GLM 5.3 FlashZhipu AI (Z.ai) | OpenCodeguidance off | 10.7% [4.0, 21.9] | 6/56 | 41/56 | 9.5% |
| 15 | GPT-5.6 SolOpenAI | Codexguidance off | 8.9% [3.0, 19.6] | 5/56 | 45/56 | 13.1% |
| 16 | DeepSeek FlashDeepSeek | OpenCodeguidance off | 7.1% [2.0, 17.3] | 4/56 | 43/56 | 13.0% |
| 17 (tied) | GPT-6 AstraOpenAI | Codexguidance offcomputer use | 3.6% [0.4, 12.3] | 2/56 | 35/56 | 8.2% |
| 17 (tied) | Qwen 3.8 MaxAlibaba (Qwen) | OpenCodeguidance off | 3.6% [0.4, 12.3] | 2/56 | 35/56 | 4.0% |
| 19 | Gemini 3.1 Pro PreviewGoogle | OpenCodeguidance off | 1.8% [0.0, 9.6] | 1/56 | 19/56 | 9.0% |
Task resolution rate on 56 TimelineBench tasks, one attempt each; exact 95% CI. Ranked by resolution rate; ties share a rank.
Overall: 169 of 1,064 slots resolved, 15.9% [13.7, 18.2]. 834 slots pass all their tests. A missing output resolves nothing.
VerifiedRun and scored by the TimelineBench team. Every entry is verified: the paper's 16 setups and those added since.
Rank: by resolution rate; setups with the same number of resolved tasks share a rank, and ranks are recomputed over the setups shown. How scoring works.
A task is resolved when the agent's cut passes all of its tests and clears the quality bar. Counts are the 1,064 slots still standing after each layer.
- 01 / 04
4 delivery tests
The task's unchanged tests/test.sh: the file exists; video codec, raster and frame rate; duration inside the window; audio stream.
1,025slots pass · 96.3% - 02 / 04
6 content guards
Not mostly silent or frozen, no looped footage, no dead air over 5 s, no one-sided sound, faces not cut by the frame.
932still standing · 87.6% - 03 / 04
Brief must-haves
What the brief requires in so many words, 2–7 per task (215 in all), checked by code or a 2-of-3 judge majority.
834pass all tests · 78.4% - 04 / 04
Quality bar
A three-lab judge panel scores the cut against the professional cut of the same job; the gap must reach the collection's tie margin.
169resolved · 15.9%
Disclosures
- One run per task.
- The reference edits are professional edits, not certified ground truth.
Validated on editors
43 video editors made 2,589 blind judgments of agent cut against professional cut (2,582 assessable). With each setup's own votes held out, the resolution rate tracks the editors' win-or-tie rate with a mean error of 3.0 points and Spearman 0.93.