Lab 07. Silent harness faults and configuration fitness checks
Part 3 · Scheduled 20 Dec 2026 · 2-hour lab
Upcoming course outline. This class has not been recorded as studied.
Lab brief
The seeded repo is part2-ref with a seeded coding-agent configuration and one fault in the student's own tool guard (it fails open on an exception). Find the 4-6 silent faults with the agent's doctor and status commands and a load-time hook, in 10 turns or fewer; add one CI check per fault, for both agents. Write a skill with 10 should + 10 should-not prompts plus 10 peer held-out rephrasings; run it alone and with 5 distractor skills.
What to build and measure
- The seeded repo is part2-ref with a seeded coding-agent configuration; 4-6 silent faults, number and names hidden, one of them in the student's own tool guard (it fails open on an exception).
- Find each with claude doctor, /hooks, /permissions, /mcp, /skills, /memory, / context and an InstructionsLoaded hook; 10 agent turns maximum.
- Add one CI check per fault, for both agents; each check fails on the seeded repo and passes on the fixed one.
- Write a skill and 20 trigger prompts; a peer adds 10 held-out rephrasings; run with and without 5 distractor skills (60 calls).
Why this class matters
- Industry: CI checks on agent configuration are new practice; students leave with a reusable check set.
- Academic: Held-out prompts and distractors are the eval design the Part 4 paper needs.
- Elsewhere: Students stop shipping skills that fire on the wrong prompt.
In-room pass check
Found / seeded >= 0.8; each CI check fails on the seeded repo and passes on the fixed one; the own-agent guard now blocks on exception; held-out trigger accuracy >= 90% with distractors and zero false fires, or the skill stays draft with evidence.
Required materials
- Staff seeded configuration on the Part 2 reference and the fixed reference (ready by 13 Dec).
- API budget: 60 calls per student at --max-turns 1; local agent if the choice changes.
- A harness worksheet for students who missed it.
These are materials the draft assigns to the course team. They are not bundled with these notes. See the course guide for the test-first verification log and the deliverables for part boundaries.
Results ledger
[Measurements, test output, run-record paths, and verification log]
Sources
- Google/Kaggle 2026, Day 3: Agent Skills
- Claude Code docs: hooks
- Claude Code docs: settings and precedence
- Claude Code docs: skills