Lab 10. Calibrated graders and trajectories as evaluation evidence
Part 4 · Scheduled 10 Jan 2027 · 2-hour lab
Upcoming course outline. This class has not been recorded as studied.
Lab brief
Run the class 8 matchers on the student's own run records and on the class 8 transcripts; add one local LLM judge; compute kappa against 30 hand labels done before class, revise the judge once, recompute; gate CI on pass^k minus margin; submit a one-page analysis plan.
What to build and measure
- Run the class 8 matchers on the student's own run records and on the class 8 transcripts; add one local LLM judge to the eval.
- Compute kappa against 30 hand labels; revise the judge once; compute kappa again.
- Gate CI on pass^k minus the margin; a seeded regression must fail.
- Flag a planted run that deleted a failing test.
- Each team submits a one-page analysis plan: variable, metric, k, test.
Why this class matters
- Industry: A judge with a measured kappa is the difference between an eval and an opinion.
- Academic: The analysis plan is pre-registration; it stops p-hacking in the Part 4 paper.
- Elsewhere: Students see how much a judge changes after one revision.
In-room pass check
Matchers pass the staff 12-case file and run on the student's records; both kappa values reported; a planted run that deleted a failing test is flagged; a seeded regression fails CI; the plan names variable, metric, k and test.
Required materials
- 30 staff outputs labelled by each student before class (45 min).
- Staff 12-case trajectory file and the seeded regression (ready by 3 Jan).
- Local judge model on the LAN server.
These are materials the draft assigns to the course team. They are not bundled with these notes. See the course guide for the test-first verification log and the deliverables for part boundaries.
Results ledger
[Measurements, test output, run-record paths, and verification log]
Sources
- Google/Kaggle 2026, Day 3: Agent Skills
- Anthropic: Demystifying evals for AI agents