Week 4: Debugging AI's Work
Use | Output that looks right is not necessarily right
Learning objectives
By the end of this week, you should be able to:
- Identify failure modes in AI-generated analytic code without being told what to look for
- Quantify the exact numerical impact of each bug on key business metrics
- Build an automated test suite (
run_validation()) that catches data quality and logic errors - Apply three test habits: row-count checks, data-description checks, code/unit checks
- Use lightweight versioning to track changes
- Conduct a peer code review using both manual inspection and automated test suites
Sessions
Five failure trends in AI-generated code, the three test routines, then a bug hunt on a provided dataset: you'll find bugs in an AI-generated script and compute the exact dollar and percentage impact of each one. "Revenue is wrong" is not enough, you need to say by how much.
Build an automated test function that runs 10+ validation checks and produces a pass/fail report. Then peer-review a classmate's Week 3 notebook using both manual inspection and your test suite comparing what each method catches.
Prepare before class
Before Session 1:
- Make sure you have access to the Week 3 v2 dataset (
class5-orders-data.csv) - Review your Week 3 notebook - you’ll use it during Session 2’s peer review
Before Session 2:
- Bring your Week 3 notebook ready to share with a partner
- Review the Python for analysts guide section on assertions
Key concepts
| Concept | What it means |
|---|---|
| Three test routines | Row-count checks, data-description checks, code/unit checks - run every time you use AI code |
| Quantitative impact | Not “the bug made revenue wrong” but “the bug overstated revenue by $12,400 (8.3%)” |
| Test function | A reusable run_validation() function with 8+ checks that produces a pass/fail report - infrastructure, not one-off assertions |
| Peer review | Manual checklist + automated test suite, comparing what each catches |
Readings and resources
- Békés, Gábor. Data Analysis with AI, Week 3: Data Wrangling & Debugging - the three kinds of tests, “silent failures,” and trust-but-verify discipline that this week’s bug hunt and test suite are built on (Session 1)
- Git and GitHub 101 guide - the lightweight versioning you’ll use in Session 2
Deliverable
Bug report with quantitative impact analysis, fixed script with an automated test function, and peer review combining manual and automated checks.
Full prompt and rubric →Looking ahead
Next week: AI as Analytic/Research Companion - you’ll use AI to help design an analysis, not just execute one. Shifting from “write me code” to “help me think through an approach.”