About
TimelineBench evaluates video-editing agents from raw footage to final cut.
01What it is
TimelineBench is a benchmark of 56 real video-editing jobs. Each gives an agent a client brief and a frozen source pack (camera originals, recorded voice-over and dialogue, music, graphics, scripts and shot lists) and asks for one finished video, cut to the brief's delivery specification. Every task is a Harbor task, and a Python package scores the result in four layers: delivery tests, content guards, the brief's must-haves and a quality bar against the professional cut.
The headline metric is the task resolution rate: the share of the 56 tasks an agent resolves in one attempt, with an exact 95% interval. Across 16 setups, agents resolve 125 of 896 slots (14.0%); the best setup resolves 15 of 56. See the leaderboard, the tasks and the docs.
02Collections
| Collection | Tasks | Raster / fps | Output window | Inputs |
|---|---|---|---|---|
| EditStockProfessional productions from EditStock’s stock library: commercials, brand films, profiles and a narrative short. | 11 | 1920×1080, 3840×2160, 4096×216023.976 / 24 fps | 29–305 s | 1,707 files99.7 GB |
| CinestudyCinestudy's free "EDIT THIS" practice scenes: comedy, drama, horror, documentary and a trailer. | 15 | 1920×108023.976 / 24 / 25 fps | 35–315 s | 407 files155.6 GB |
| UGCPresenter-led 9:16 portrait ads for products, services and personal brands. | 15 | 1080×192025 fps | 25–55 s | 1,080 files163.1 GB |
| CommercialProduct, lifestyle and travel commercials. | 15 | 1920×108024 fps | 44–125 s | 1,809 files254.3 GB |
03Design principles
- 01Complete work productsEach task asks for one finished video from a full production pack, not a clip-level edit. Sources, sequence and sound have to work together.
- 02Obligations fixed, solution openBriefs state what must appear and the delivery spec, the way a client would. How to structure, pace and mix the piece is left to the agent; matching the professional cut is not the goal.
- 03Frozen, verifiable inputsEvery one of the 5,003 input files is pinned by size and sha256, and a trial does not start on unverified inputs. Briefs, tests and oracles are byte-identical to the evaluated release.
- 04Tests with an oracleLike Terminal-Bench, every task has pass/fail tests. A mechanical oracle proves the delivery contract can be met, and the professional cut validates the content tests: a check it fails is dropped.
- 05A bar calibrated on editorsThe quality bar compares each cut with the professional cut through a three-lab judge panel whose margins are fitted on blind editor judgments, and validated with each setup’s own votes held out.
- 06Ranks with intervals; setups, not cutsOne attempt per task gives intervals of about ±11 points, so neighbouring ranks on the leaderboard are often not statistically separable. The verifier is valid per setup, not as a per-cut judge.
- 07Reproducible numbersEvery per-slot result is released with scripts that recompute the paper’s numbers without media, network or model calls. Two independent recomputes found no discrepancies.
- 08No footage publishedThe site shows briefs, specs, rubrics and results, never video, stills or thumbnails. The source packs are not distributed. Fill in the access form and the maintainers will get in touch about running your agent on the benchmark internally.
04People
TimelineBench is under double-blind review. During the review period the authors are listed as the TimelineBench Team, and contributors will be listed after the review period.
43 video editors made the 2,589 blind judgments of the human study.
05Contact
Security reports, requests to remove a listed model, and media-access problems go to the maintainers privately through the contact form, not through public issues. To have an agent run on the benchmark, use the access form. The source packs are not distributed; the maintainers get in touch about running the agent internally (see Inputs & media access). During review the code is an anonymized mirror at /code.