Lab 11. Harness architectures compared on cost, time and success
Part 4 · Scheduled 17 Jan 2027 · 2-hour lab
Upcoming course outline. This class has not been recorded as studied.
Lab brief
Write an 8-field run contract; run a coding agent on 3 staff tasks, bare vs harnessed (the team's own class 7 config), k = 3; one extension arm on the system the team's pathway studies (own agent or coding agent), 3 tasks, k = 3; runs in the background 6 at a time with --max-turns 15 and --max-budget-usd; check three 'passed' patches by hand, layer rules included; tag data-frozen.
What to build and measure
- Each team writes an 8-field run contract; the stop condition is a script that exits 0 or 1.
- Run a coding agent on 3 staff tasks, bare vs harnessed (the team's own class 7 configuration), k = 3; one extension arm on the system the team's pathway studies (own agent or coding agent).
- Runs go in the background, 6 at a time, --max-turns 15 and --max-budget-usd; check three 'passed' patches by hand meanwhile, layer rules included; record minutes per accepted task.
- Tag data-frozen at the end; write the paper draft after class (due Fri 22 Jan).
- Stretch: three-way review recall on 5 seeded bugs; an orchestrator with two worktree workers.
Why this class matters
- Industry: A table of pass^3, tokens and cost per accepted task per arm is the business case for or against a harness.
- Academic: Frozen data with a tag is the reproducibility requirement for class 12 and for publication.
- Elsewhere: Students learn that a 'passed' patch still needs a human read.
In-room pass check
Stop condition is a script exiting 0/1; table of pass^3, tokens, cost and minutes per accepted task per arm; patch review logged; data-frozen tag exists.
Required materials
- 3 staff coding tasks with hidden tests (ready by 3 Jan).
- API budget: the largest of the course; low-cost model option or local agent as fallback.
- About 60 min of background runtime inside the lab.
These are materials the draft assigns to the course team. They are not bundled with these notes. See the course guide for the test-first verification log and the deliverables for part boundaries.
Results ledger
[Measurements, test output, run-record paths, and verification log]
Sources
- Claude Code docs: headless mode (-p, --max-turns, --max-budget-usd)
- Arena.ai: HarnessTax