Lecture 12. Reproduced results, validity threats and thesis paths

Part 4 · The Architecture as an Evaluated Claim · Scheduled 24 Jan 2027 · 1-hour lecture

Upcoming course outline. This class has not been recorded as studied.

Learning objectives

Threats to validity: contamination, judge bias, small n, model drift, the harness as a hidden variable.
Source criticism: grade each number in Day 1 (the 3-10x crossover is 'illustrative, not measured').
How to review a paper; venues: MSR, LLM4Code, BLP.
Five thesis pathways from the course data.

Industry: Engineers who can tell a measured claim from a marketing number buy the right tools.
Academic: This class turns the course project into a thesis proposal; it is the research flavour of the course made concrete.
Elsewhere: Students can review a colleague's experiment with a form and a reason.

The course research-rigor rubric and the five thesis pathways.
Stanford CS329Z L11 slides, L16 schedule and the project rubric.
The paper (Kapoor et al. 2024) as pre-reading; Google Day 1 2026 for the claims check.

These are preparation requirements in the draft; staff materials are not bundled with this site.

The linked reference library contains the source URLs extracted from the course PDF. Availability notes above reflect the draft of 4 November 2026.

Continue with Lab 12: Reproduced results, validity threats and thesis paths. Every lab contributes an artifact to the same repository and a measurement to the results ledger.

[Lecture notes and observations]