Lab 03. The agent loop as a tested, evaluated component
Part 1 · Scheduled 22 Nov 2026 · 2-hour lab
Part 1 is currently in progress. This page records the supplied class plan; personal notes and results have not been added.
Lab brief
Grow the staff loop.py into a 3-tool agent over the class 2 notices; the loop writes a JSONL run record that classes 6, 8, 10 and 11 read. Add scripted-LM unit tests and a 20-task eval with code graders, k = 3, on the local model. Tag part1-ref.
What to build and measure
- Grow the staff loop.py into a 3-tool agent over the class 2 corpus; the loop writes a JSONL run record per task.
- Scripted-LM tests: malformed reply, tool exception, step cap.
- A 20-task eval with code graders, k = 3, on the local model; one command runs it.
- Tag the repo as the Part 1 deliverable (part1-ref).
Why this class matters
- Industry: A tested agent with a one-command eval is the artifact an employer wants to see in a portfolio.
- Academic: The 20-task eval is the baseline that later experiments (context policy, harness) compare against.
- Elsewhere: Students own a small working agent that runs without paid APIs.
In-room pass check
Scripted tests pass for a malformed reply, a tool exception and the step cap; the eval runs with one command; pass^3 and a run record per task are reported.
Required materials
- Staff loop.py: a Python agent loop over the class 1 wrapper.
- Scripted-LM test file and the 20-task set with graders.
- Local model from class 1; LAN server for slow laptops.
These are materials the draft assigns to the course team. They are not bundled with these notes. See the course guide for the test-first verification log and the deliverables for part boundaries.
Results ledger
[Measurements, test output, run-record paths, and verification log]
Sources
- Anthropic: Demystifying evals for AI agents