Week 5: AI as Analytic Companion

Module: Use | Deliverable: Analytic design critique + implementation (due before Week 6 begins) | See assignment →

Learning objectives

By the end of this week, you should be able to:

Use AI to propose, critique, and refine an analytic design (four prompt roles)
Apply the helpful/adversarial pairing technique
Conduct outlier analysis and determine appropriate data transformations
Implement customer segmentation with formal model selection (elbow + silhouette + gap statistic) and holdout validation
Or implement a linear mixed model with AIC/BIC comparison, residual diagnostics, and confidence intervals
Report results with effect sizes and statistical evidence, not just point estimates

Sessions

Session 1 · Class 9

AI as design critic

The shift from "write me code" to "help me think." Four prompt roles (helpful analyst, adversarial reviewer, domain expert, methods reviewer), the helpful/adversarial pairing technique, and an activity where you propose and attack an analysis plan.

Four prompt roles Helpful/adversarial pairing Propose and attack activity

Session 2 · Class 10

From design to rigorous implementation

Implement your analysis with formal model selection: for clustering, we'll focus on three metrics (elbow, silhouette, gap statistic), outlier analysis, and holdout validation. For the LMM track, AIC/BIC comparison across 4+ specifications with residual diagnostics and confidence intervals. Then a group design critique.

Python: segmentation or LMM Formal model selection Holdout validation Group design critique

Prepare before class

Before Session 1:

Read Békés, Data Analysis with AI, Week 4: Econometrics with AI - focus on the helpful/adversarial prompt pattern (the controls section walks through it directly)
Think about a business question for the customer dataset

Before Session 2:

Review your Session 1 reconciliation
Skim the sklearn KMeans docs or statsmodels mixedlm docs depending on your track

Key concepts

Concept	What it means
Helpful/adversarial pairing	Two separate AI interactions: one proposes, the other attacks and you get to reconcile
Gap statistic	Compares clustering quality to random data - tests whether clusters are real or noise
Holdout validation	Fit on 70%, evaluate on 30% - tests whether clusters generalize
AIC/BIC	Information criteria for model comparison: lower is better, BIC penalizes complexity more
ICC	Intraclass correlation (ICC): proportion of variance that’s between-group vs. within-group

Readings and resources

Békés, Gábor. Data Analysis with AI, Week 4: Econometrics with AI - explicitly teaches the helpful/adversarial pairing this week’s design activity is built on: one AI session proposes, a second attacks, you reconcile
sklearn KMeans docs or statsmodels mixedlm docs, depending on your track (Session 2)

Deliverable

Analytic Design Critique + Implementation

3 components | 30 points | Due before Week 6's first class | Submit via Canvas

Design document with adversarial critique, Python implementation with formal model selection (3 metrics or AIC/BIC), holdout validation, and AI use log.

Full prompt and rubric →

Looking ahead

Next week closes out the Use module with Text as Data with LLMs - classification, network analysis, and the question of when an LLM adds value over a simpler method.