GPT-6 Astra

OpenAI · Codex CLI 0.155.1 · guidance off

Rank 3 of 16 (tied)Verified · run and scored by the TimelineBench teamCodex CLI 0.155.1guidance off

Resolution rate

21.4%[11.6, 34.4] exact 95% CI

12 of 56 tasks resolved, one attempt each

Tests passed: 48/56; delivery + guards + must-haves
Must-haves passed: 51/56; 91.1% [80.4, 97.0]
Editors' win-or-tie: 23.8%; [16.2, 31.4] · 168 votes
Rank · bootstrap interval: 3 (tied) · 1–9; by resolution rate, of 16 setups
Videos delivered: 56/56; 0 missing
Median runtime: 19.6 min; 19.5 h in total
Auto Elo: 792; [676, 884] · pro cut 1000
Editor Elo: 786; [703, 852]
Cost: Not recorded; recorded for Claude Code only

Where the cuts drop out

56 slots, in order

How scoring works

01 / 04
Delivery tests
56 videos delivered; the four tests of each task's tests/test.sh.
56pass · 100.0%
02 / 04
Content guards
Silent, frozen, looped, dead air, one-sided sound, faces cut.
53still standing · 94.6%
03 / 04
Brief must-haves
Every validated must-have of the brief.
48pass all tests · 85.7%
04 / 04
Quality bar
Panel gap against the professional cut reaches the collection's tie margin.
12resolved · 21.4%

By collection

EditStock3/11resolved · 9/11 pass tests · bar 0.21

Cinestudy4/15resolved · 11/15 pass tests · bar 0.30

UGC1/15resolved · 14/15 pass tests · bar 1.54

Commercial4/15resolved · 14/15 pass tests · bar 0.52

Per task

first failing layer, panel gap and editor votes

Per-task results for GPT-6 Astra · Codex · guidance off: delivery tests, content guards, must-haves, quality gap against the collection tie margin, outcome and editor votes.
Task	Delivery	Guards	Must-haves	Gap vs bar	Outcome	Editors A·T·P
EditStock · 3/11 resolved · tie margin 0.21
Babiators	4/4	6/6	Pass · 12/15 items	+0.43 / 0.21	Resolved	0 · 1 · 2
BOS Parkour	4/4	6/6	Pass · 9/9 items	+0.23 / 0.21	Resolved	1 · 0 · 2
Evolve By Training Camp	4/4	6/6	Pass · 8/9 items	−0.36 / 0.21	Below bar	0 · 0 · 3
Incubate	4/4	6/6	Fail · 8/15 items	0.00 / 0.21	Failed a test	2 · 0 · 1
J-Pops	4/4	6/6	Pass · 13/15 items	−0.15 / 0.21	Below bar	0 · 0 · 3
Klug Fitness Brand Story	4/4	6/6	Fail · 14/15 items	+0.45 / 0.21	Failed a test	0 · 0 · 3
Klug Fitness Next Time	4/4	6/6	Pass · 14/15 items	+0.48 / 0.21	Resolved	0 · 0 · 3
Klug Fitness Railroad Track	4/4	6/6	Pass · 15/15 items	−0.01 / 0.21	Below bar	1 · 0 · 2
Two Player	4/4	6/6	Pass · 10/15 items	−0.91 / 0.21	Below bar	1 · 0 · 2
Whisper Fan	4/4	6/6	Pass · 7/7 items	+0.03 / 0.21	Below bar	0 · 0 · 3
Wild Planet	4/4	6/6	Pass · 8/8 items	−0.29 / 0.21	Below bar	1 · 0 · 2
Cinestudy · 4/15 resolved · tie margin 0.30
Car Chase	4/4	6/6	Pass · 13/13 items	−0.14 / 0.30	Below bar	0 · 1 · 2
Color Grading	4/4	6/6	Pass · 13/15 items	+0.22 / 0.30	Below bar	1 · 0 · 2
Compression Of Time	4/4	Mostly frozen	Pass · 15/15 items	−0.40 / 0.30	Failed a test	2 · 0 · 1
Cutting On Motion	4/4	6/6	Pass · 9/14 items	+0.14 / 0.30	Below bar	3 · 0 · 0
Documentary Format	4/4	6/6	Pass · 14/15 items	+0.28 / 0.30	Below bar	1 · 0 · 2
Editing A Sequence	4/4	6/6	Fail · 13/15 items	+1.18 / 0.30	Failed a test	2 · 0 · 1
Horror Movie	4/4	6/6	Pass · 13/15 items	+0.22 / 0.30	Below bar	0 · 0 · 3
Hospital Scene	4/4	6/6	Pass · 15/15 items	+0.19 / 0.30	Below bar	0 · 1 · 2
License Exam	4/4	Repeated footage	Pass · 15/15 items	+0.37 / 0.30	Failed a test	2 · 0 · 1
Movie Trailer	4/4	6/6	Pass · 13/15 items	+0.41 / 0.30	Resolved	1 · 0 · 2
Podcast Documentary	4/4	6/6	Pass · 13/15 items	−0.29 / 0.30	Below bar	0 · 0 · 3
Refractory	4/4	6/6	Pass · 14/15 items	+0.31 / 0.30	Resolved	1 · 0 · 2
Relationship Card	4/4	6/6	Pass · 15/15 items	+0.69 / 0.30	Resolved	0 · 0 · 3
Restaurant Scene	4/4	6/6	Pass · 12/15 items	+0.53 / 0.30	Resolved	1 · 0 · 2
The Intern Mockumentary	4/4	Dead air	Pass · 14/14 items	+0.15 / 0.30	Failed a test	2 · 1 · 0
UGC · 1/15 resolved · tie margin 1.54
Breathex Nose Patches	4/4	6/6	Pass · 12/15 items	−0.01 / 1.54	Below bar	0 · 0 · 3
CUPNEX	4/4	6/6	Pass · 12/15 items	+0.68 / 1.54	Below bar	0 · 0 · 3
Dazzle and Charm	4/4	6/6	Pass · 13/15 items	+2.06 / 1.54	Resolved	0 · 0 · 3
Feel Comfy	4/4	6/6	Pass · 12/14 items	+0.66 / 1.54	Below bar	0 · 0 · 3
Fitness Venue	4/4	6/6	Pass · 13/15 items	+0.96 / 1.54	Below bar	0 · 0 · 3
Global Physiotherapy Clinic	4/4	6/6	Pass · 13/15 items	+0.64 / 1.54	Below bar	0 · 0 · 3
House of Hue	4/4	6/6	Pass · 10/15 items	+0.69 / 1.54	Below bar	1 · 0 · 2
LeadLoop	4/4	6/6	Fail · 11/15 items	+0.65 / 1.54	Failed a test	0 · 0 · 3
LedgeFin	4/4	6/6	Pass · 12/15 items	+0.80 / 1.54	Below bar	1 · 0 · 2
MotionMix Studio	4/4	6/6	Pass · 14/15 items	+0.67 / 1.54	Below bar	0 · 0 · 3
Nova Sole	4/4	6/6	Pass · 14/15 items	+0.33 / 1.54	Below bar	0 · 0 · 3
Off Case	4/4	6/6	Pass · 12/15 items	−0.11 / 1.54	Below bar	0 · 0 · 3
Personal Branding	4/4	6/6	Pass · 12/15 items	+0.51 / 1.54	Below bar	0 · 0 · 3
Personal Introduction	4/4	6/6	Pass · 13/15 items	+1.37 / 1.54	Below bar	0 · 0 · 3
Portable Side Bag	4/4	6/6	Pass · 12/15 items	+0.52 / 1.54	Below bar	0 · 0 · 3
Commercial · 4/15 resolved · tie margin 0.52
Aroma Candle	4/4	6/6	Pass · 14/15 items	0.00 / 0.52	Below bar	0 · 0 · 3
Bean & Roast	4/4	6/6	Pass · 15/15 items	+0.82 / 0.52	Resolved	2 · 0 · 1
Beauty of Delhi	4/4	6/6	Pass · 13/15 items	+0.93 / 0.52	Resolved	1 · 0 · 2
Bike Ride	4/4	6/6	Pass · 15/15 items	+0.87 / 0.52	Resolved	0 · 0 · 3
Camera	4/4	6/6	Pass · 13/15 items	−0.28 / 0.52	Below bar	2 · 0 · 1
Fortunex	4/4	6/6	Pass · 13/15 items	+0.01 / 0.52	Below bar	0 · 0 · 3
Hard Life Of Athlete	4/4	6/6	Pass · 12/15 items	−0.13 / 0.52	Below bar	0 · 0 · 3
Loom Craft	4/4	6/6	Pass · 14/15 items	+0.01 / 0.52	Below bar	1 · 0 · 2
Motionmix In Charge	4/4	6/6	Pass · 12/13 items	+1.01 / 0.52	Resolved	0 · 0 · 3
Pro Bite	4/4	6/6	Pass · 13/15 items	+0.25 / 0.52	Below bar	0 · 0 · 3
Quickshine	4/4	6/6	Pass · 9/14 items	−0.05 / 0.52	Below bar	2 · 0 · 1
Shadow Airpod	4/4	6/6	Fail · 13/15 items	+0.57 / 0.52	Failed a test	1 · 0 · 2
Standcup	4/4	6/6	Pass · 12/15 items	+0.39 / 0.52	Below bar	1 · 0 · 2
Street Wear	4/4	6/6	Pass · 15/15 items	−0.48 / 0.52	Below bar	0 · 0 · 3
Voltix Bulb	4/4	6/6	Pass · 13/15 items	+0.47 / 0.52	Below bar	2 · 0 · 1

ResolvedPassed tests, below barFailed a testNo outputEditors A·T·P: blind votes for the agent cut, a tie, or the professional cut.

Missing outputs

Every task produced a video.

Delivery failures

Every delivered video passes its four delivery tests.

Guard failures

Dead air1
Mostly frozen1
Repeated footage1

Reproduce

Run this setup

# GPT-6 Astra · Codex CLI 0.155.1 · guidance off
# source packs are not distributed; the maintainers run the agent internally
harbor run -p tasks/<id> -a codex -m openai/gpt-6-astra
# after all 56 tasks, one attempt each:
timelinebench aggregate <jobs-dir>

Setup id codex__gpt-6-astra__skills-off · model gpt-6-astra · run guide

Disclosures

One run per task.
The reference edits are professional edits, not certified ground truth.

Where the cuts drop out

Delivery tests

Content guards

Brief must-haves

Quality bar