Claude Fable 5.1
Anthropic · Claude Code 2.1.278 · guidance off
Rank 5 of 16 (tied)Verified · run and scored by the TimelineBench teamClaude Code 2.1.278guidance off
Resolution rate
17.9%[8.9, 30.4] exact 95% CI
10 of 56 tasks resolved, one attempt each
- Tests passed
- 49/56
- delivery + guards + must-haves
- Must-haves passed
- 51/56
- 91.1% [80.4, 97.0]
- Editors' win-or-tie
- 23.5%
- [15.8, 31.2] · 167 votes
- Rank · bootstrap interval
- 5 (tied) · 3–10.5
- by resolution rate, of 16 setups
- Videos delivered
- 56/56
- 0 missing
- Median runtime
- 57.0 min
- 62.5 h in total
- Auto Elo
- 713
- [554, 817] · pro cut 1000
- Editor Elo
- 780
- [694, 848]
- Cost
- $2,311.07
- $19.15 median per task
- 01 / 04
Delivery tests
56 videos delivered; the four tests of each task's tests/test.sh.
56pass · 100.0% - 02 / 04
Content guards
Silent, frozen, looped, dead air, one-sided sound, faces cut.
54still standing · 96.4% - 03 / 04
Brief must-haves
Every validated must-have of the brief.
49pass all tests · 87.5% - 04 / 04
Quality bar
Panel gap against the professional cut reaches the collection's tie margin.
10resolved · 17.9%
By collection
EditStock1/11resolved · 9/11 pass tests · bar 0.21
Cinestudy6/15resolved · 13/15 pass tests · bar 0.30
UGC0/15resolved · 13/15 pass tests · bar 1.54
Commercial3/15resolved · 14/15 pass tests · bar 0.52
Per task
first failing layer, panel gap and editor votes| Task | Delivery | Guards | Must-haves | Gap vs bar | Outcome | Editors A·T·P |
|---|---|---|---|---|---|---|
| EditStock · 1/11 resolved · tie margin 0.21 | ||||||
| Babiators | 4/4 | 6/6 | Pass · 12/15 items | −0.05 / 0.21 | Below bar | 2 · 0 · 1 |
| BOS Parkour | 4/4 | 6/6 | Pass · 9/9 items | −0.22 / 0.21 | Below bar | 1 · 0 · 2 |
| Evolve By Training Camp | 4/4 | 6/6 | Pass · 9/9 items | −0.60 / 0.21 | Below bar | 1 · 0 · 2 |
| Incubate | 4/4 | 6/6 | Fail · 9/15 items | +0.22 / 0.21 | Failed a test | 0 · 0 · 3 |
| J-Pops | 4/4 | 6/6 | Pass · 12/15 items | −0.17 / 0.21 | Below bar | 1 · 1 · 1 |
| Klug Fitness Brand Story | 4/4 | 6/6 | Pass · 15/15 items | +0.75 / 0.21 | Resolved | 1 · 0 · 2 |
| Klug Fitness Next Time | 4/4 | 6/6 | Pass · 14/15 items | +0.03 / 0.21 | Below bar | 1 · 0 · 2 |
| Klug Fitness Railroad Track | 4/4 | 6/6 | Pass · 15/15 items | −0.15 / 0.21 | Below bar | 0 · 0 · 3 |
| Two Player | 4/4 | 6/6 | Pass · 10/15 items | −1.12 / 0.21 | Below bar | 1 · 0 · 2 |
| Whisper Fan | 4/4 | 6/6 | Pass · 7/7 items | −0.09 / 0.21 | Below bar | 1 · 0 · 2 |
| Wild Planet | 4/4 | One-sided sound | Pass · 8/8 items | −0.24 / 0.21 | Failed a test | 0 · 0 · 3 |
| Cinestudy · 6/15 resolved · tie margin 0.30 | ||||||
| Car Chase | 4/4 | Faces cut by the frame | Pass · 13/13 items | −0.20 / 0.30 | Failed a test | 1 · 0 · 2 |
| Color Grading | 4/4 | 6/6 | Pass · 15/15 items | +0.22 / 0.30 | Below bar | 1 · 1 · 1 |
| Compression Of Time | 4/4 | 6/6 | Fail · 14/15 items | +0.11 / 0.30 | Failed a test | 1 · 0 · 2 |
| Cutting On Motion | 4/4 | 6/6 | Pass · 10/14 items | −0.09 / 0.30 | Below bar | 2 · 0 · 1 |
| Documentary Format | 4/4 | 6/6 | Pass · 14/15 items | +0.44 / 0.30 | Resolved | 0 · 0 · 3 |
| Editing A Sequence | 4/4 | 6/6 | Pass · 14/15 items | +1.23 / 0.30 | Resolved | 1 · 1 · 1 |
| Horror Movie | 4/4 | 6/6 | Pass · 14/15 items | +0.51 / 0.30 | Resolved | 0 · 0 · 3 |
| Hospital Scene | 4/4 | 6/6 | Pass · 15/15 items | −0.16 / 0.30 | Below bar | 1 · 0 · 2 |
| License Exam | 4/4 | 6/6 | Pass · 15/15 items | +0.36 / 0.30 | Resolved | 1 · 0 · 2 |
| Movie Trailer | 4/4 | 6/6 | Pass · 13/15 items | +0.69 / 0.30 | Resolved | 1 · 0 · 2 |
| Podcast Documentary | 4/4 | 6/6 | Pass · 13/15 items | −0.82 / 0.30 | Below bar | 1 · 0 · 2 |
| Refractory | 4/4 | 6/6 | Pass · 15/15 items | +0.61 / 0.30 | Resolved | 1 · 0 · 2 |
| Relationship Card | 4/4 | 6/6 | Pass · 14/15 items | −0.13 / 0.30 | Below bar | 1 · 0 · 2 |
| Restaurant Scene | 4/4 | 6/6 | Pass · 12/15 items | +0.17 / 0.30 | Below bar | 1 · 0 · 1 |
| The Intern Mockumentary | 4/4 | 6/6 | Pass · 13/14 items | −0.09 / 0.30 | Below bar | 0 · 0 · 3 |
| UGC · 0/15 resolved · tie margin 1.54 | ||||||
| Breathex Nose Patches | 4/4 | 6/6 | Pass · 12/15 items | −0.14 / 1.54 | Below bar | 0 · 0 · 3 |
| CUPNEX | 4/4 | 6/6 | Fail · 11/15 items | +0.44 / 1.54 | Failed a test | 1 · 0 · 2 |
| Dazzle and Charm | 4/4 | 6/6 | Pass · 13/15 items | +1.45 / 1.54 | Below bar | 1 · 0 · 2 |
| Feel Comfy | 4/4 | 6/6 | Pass · 12/14 items | +0.08 / 1.54 | Below bar | 0 · 0 · 3 |
| Fitness Venue | 4/4 | 6/6 | Pass · 13/15 items | +0.73 / 1.54 | Below bar | 0 · 0 · 3 |
| Global Physiotherapy Clinic | 4/4 | 6/6 | Pass · 12/15 items | +0.65 / 1.54 | Below bar | 0 · 0 · 3 |
| House of Hue | 4/4 | 6/6 | Pass · 11/15 items | +0.67 / 1.54 | Below bar | 0 · 0 · 3 |
| LeadLoop | 4/4 | 6/6 | Fail · 11/15 items | +0.64 / 1.54 | Failed a test | 0 · 0 · 3 |
| LedgeFin | 4/4 | 6/6 | Pass · 12/15 items | +0.43 / 1.54 | Below bar | 1 · 0 · 2 |
| MotionMix Studio | 4/4 | 6/6 | Pass · 13/15 items | +0.35 / 1.54 | Below bar | 0 · 0 · 3 |
| Nova Sole | 4/4 | 6/6 | Pass · 14/15 items | +0.18 / 1.54 | Below bar | 0 · 0 · 3 |
| Off Case | 4/4 | 6/6 | Pass · 13/15 items | −0.25 / 1.54 | Below bar | 0 · 0 · 3 |
| Personal Branding | 4/4 | 6/6 | Pass · 12/15 items | +0.51 / 1.54 | Below bar | 0 · 0 · 3 |
| Personal Introduction | 4/4 | 6/6 | Pass · 14/15 items | +1.45 / 1.54 | Below bar | 0 · 0 · 3 |
| Portable Side Bag | 4/4 | 6/6 | Pass · 12/15 items | +0.43 / 1.54 | Below bar | 0 · 0 · 3 |
| Commercial · 3/15 resolved · tie margin 0.52 | ||||||
| Aroma Candle | 4/4 | 6/6 | Fail · 13/15 items | +0.18 / 0.52 | Failed a test | 1 · 0 · 2 |
| Bean & Roast | 4/4 | 6/6 | Pass · 15/15 items | +0.34 / 0.52 | Below bar | 0 · 0 · 3 |
| Beauty of Delhi | 4/4 | 6/6 | Pass · 13/15 items | +1.25 / 0.52 | Resolved | 0 · 0 · 3 |
| Bike Ride | 4/4 | 6/6 | Pass · 15/15 items | +0.83 / 0.52 | Resolved | 1 · 1 · 1 |
| Camera | 4/4 | 6/6 | Pass · 13/15 items | −0.94 / 0.52 | Below bar | 1 · 0 · 2 |
| Fortunex | 4/4 | 6/6 | Pass · 11/15 items | +0.09 / 0.52 | Below bar | 0 · 0 · 3 |
| Hard Life Of Athlete | 4/4 | 6/6 | Pass · 12/15 items | −0.65 / 0.52 | Below bar | 1 · 0 · 2 |
| Loom Craft | 4/4 | 6/6 | Pass · 14/15 items | −0.40 / 0.52 | Below bar | 1 · 0 · 2 |
| Motionmix In Charge | 4/4 | 6/6 | Pass · 12/13 items | +1.03 / 0.52 | Resolved | 1 · 0 · 2 |
| Pro Bite | 4/4 | 6/6 | Pass · 13/15 items | +0.23 / 0.52 | Below bar | 0 · 1 · 2 |
| Quickshine | 4/4 | 6/6 | Pass · 9/14 items | −0.52 / 0.52 | Below bar | 1 · 0 · 2 |
| Shadow Airpod | 4/4 | 6/6 | Pass · 14/15 items | +0.24 / 0.52 | Below bar | 0 · 0 · 3 |
| Standcup | 4/4 | 6/6 | Pass · 12/15 items | +0.38 / 0.52 | Below bar | 1 · 0 · 2 |
| Street Wear | 4/4 | 6/6 | Pass · 15/15 items | −1.24 / 0.52 | Below bar | 0 · 0 · 3 |
| Voltix Bulb | 4/4 | 6/6 | Pass · 13/15 items | +0.33 / 0.52 | Below bar | 1 · 0 · 2 |
ResolvedPassed tests, below barFailed a testNo outputEditors A·T·P: blind votes for the agent cut, a tie, or the professional cut.
Missing outputs
Every task produced a video.
Delivery failures
Every delivered video passes its four delivery tests.
Guard failures
- One-sided sound1
- Faces cut by the frame1
Reproduce
Run this setup
# Claude Fable 5.1 · Claude Code 2.1.278 · guidance off
# source packs are not distributed; the maintainers run the agent internally
harbor run -p tasks/<id> -a claude-code -m anthropic/claude-fable-5-1
# after all 56 tasks, one attempt each:
timelinebench aggregate <jobs-dir>Setup id claude-code__claude-fable-5-1__skills-off · model claude-fable-5-1 · run guide
Disclosures
- One run per task.
- The reference edits are professional edits, not certified ground truth.