Lecture 10. Calibrated graders and trajectories as evaluation evidence

Part 4 · The Architecture as an Evaluated Claim · Scheduled 10 Jan 2027 · 1-hour lecture

Upcoming course outline. This class has not been recorded as studied.

Learning objectives

Task, trial, grader, transcript; code, model and human graders; judge bias; kappa calibration.
pass@k vs pass^k with intervals; infrastructure noise up to 6 points on Terminal-Bench 2.0.
Output vs trajectory; seven dimensions and the method matrix (Day 4 pp. 29-33); traces as the production view of an agent.
Terminal-Bench task format; reward hacking (Stanford CS329Z L10, L11).

Industry: Teams that ship agents need an eval they trust; calibrated judges and intervals are that eval.
Academic: This is the methods chapter of a thesis on agents; it makes a result credible to a reviewer.
Elsewhere: Students read benchmark claims with the right scepticism.

These are preparation requirements in the draft; staff materials are not bundled with this site.

Stanford CS329Z L10: Guest lecture: Terminal-Bench (slides)
Stanford CS329Z L11: What data do agents need? (slides)
Stanford CS329Z L12: Data selection and quality (schedule entry, slides not published)
Stanford CS329Z L13: Evaluation fundamentals and benchmark design (schedule entry, slides not published)
Stanford CS329Z L15: LLM-as-judge and evaluation infrastructure (schedule entry, slides not published)
Anthropic: Demystifying evals for AI agents
Google/Kaggle 2026, Day 3: Agent Skills
Google/Kaggle 2026, Day 4: Vibe Coding Agent Security and Evaluation
Zhu et al., Rigorous Agentic Benchmarks (2025)

The linked reference library contains the source URLs extracted from the course PDF. Availability notes above reflect the draft of 4 November 2026.

Continue with Lab 10: Calibrated graders and trajectories as evaluation evidence. Every lab contributes an artifact to the same repository and a measurement to the results ledger.

[Lecture notes and observations]