Lab 08. Specs, locked tests and layer rules as delivery gates
Part 3 · Scheduled 27 Dec 2026 · 2-hour lab
Upcoming course outline. This class has not been recorded as studied.
Lab brief
The feature is the EXACT / IN_ORDER / ANY_ORDER trajectory matchers for part2-ref (class 10 uses them). Write specs/feature.md with >= 5 Gherkin scenarios that name the zone; tests from them; CI red gate, mutation gate and layer-rule gate; lock tests for the whole run (PreToolUse hook + deny Edit(tests/**), sandbox denyWrite on tests/, CI checksum). In pairs run the coding agent 3x free-form and 3x from the spec in the background (--max-turns 20, --max-budget-usd); each run is a row with transcript path and minutes. Last 10 min: write the team research question.
What to build and measure
- The feature is the EXACT / IN_ORDER / ANY_ORDER trajectory matchers for part2-ref; class 10 uses them.
- Write specs/feature.md with 5 or more Gherkin scenarios (2 edge cases, 1 'shall continue to') that name the zone; derive tests.
- Add a CI red gate, a mutation gate (mutmut) and a layer-rule gate; a staff patch that passes tests but breaks a layer must fail.
- Lock tests for the agent run: PreToolUse hook, deny rule Edit(tests/**), sandbox denyWrite on tests/, CI checksum on tests/.
- In pairs: 3 free-form runs and 3 spec runs in the background with the same visible tests; hidden tests run outside the sandbox; each run is a row with transcript path and minutes.
- Last 10 min: each Part 4 team writes its research question.
Why this class matters
- Industry: Locked tests and a red gate are the controls that make autonomous runs safe to merge.
- Academic: Each run is a row (arm, hidden result, tokens, cost); the class run sheet is the dataset for Part 4.
- Elsewhere: Students learn that a passing test the agent wrote is not evidence.
In-room pass check
Tests fail at the starter commit and pass at HEAD; a staff test-weakening commit fails CI; a staff patch that passes tests but breaks a layer rule fails CI; a scripted sed -i on tests/ fails under the sandbox; mutation score meets threshold; each run is a row in the class run sheet.
Required materials
- Staff feature repo on the Part 2 reference with hidden tests, layer rules and the test-lock config (ready by 13 Dec).
- API budget per pair (--max-turns 20, --max-budget-usd); run-sheet schema fixed with the Part 4 instructor.
- mutmut run once on the reference before class.
These are materials the draft assigns to the course team. They are not bundled with these notes. See the course guide for the test-first verification log and the deliverables for part boundaries.
Results ledger
[Measurements, test output, run-record paths, and verification log]
Sources
- Google/Kaggle 2026, Day 5: Spec-Driven Production Grade Development
- Claude Code docs: permissions
- Claude Code docs: sandboxing