Lab 04. Retrieval as a tested subsystem
Part 2 · Scheduled 29 Nov 2026 · 2-hour lab
Upcoming course outline. This class has not been recorded as studied.
Lab brief
Compare BM25, dense and hybrid RRF on the same corpus with 50 labelled queries; plug the best into the agent and re-run the class 3 20-task eval.
What to build and measure
- Compare BM25, dense and hybrid RRF on the staff bilingual corpus with 50 labelled queries.
- Add a CI gate: fail if Recall@10 drops below the threshold.
- Write a test that proves a restricted document never reaches a user without access.
- Plug the best method into the agent and re-run the class 3 20-task eval: component metric up is not system metric up.
Why this class matters
- Industry: A recall gate in CI and a permission test are what separate a prototype from a product.
- Academic: Three methods, one corpus, one metric table: a reproducible comparison.
- Elsewhere: Students see how a bad index silently breaks an agent.
In-room pass check
Metric table for three methods; CI fails if Recall@10 drops below threshold; a test proves a restricted document never reaches an unauthorised user; pass^3 before and after the retriever is reported.
Required materials
- Staff bilingual corpus with 50 labelled queries (ready by 22 Nov).
- Dense embeddings computed before class with the staff script (20 min on CPU).
- part1-ref for students who missed Part 1.
These are materials the draft assigns to the course team. They are not bundled with these notes. See the course guide for the test-first verification log and the deliverables for part boundaries.
Results ledger
[Measurements, test output, run-record paths, and verification log]
Sources
- rank_bm25
- sentence-transformers