Week 5: Analytic Design Critique + Implementation
3 components | Due before Week 6's first class | Submit via Canvas
Assignment
Use AI as a design partner, then implement with formal model selection and validation.
Component 1: Design document (1–2 pages, submit as PDF)
Four sections, in this order:
- Business question - the question your analysis will answer, framed for a business audience
- AI-proposed plan - prompt an AI tool as a helpful analyst; include its proposed approach
- Adversarial critique - prompt an AI tool (the same one or a different one) as an adversarial reviewer attacking the plan; include its critique
- Your reconciliation - what you kept, changed, or rejected from each round, and why
Component 2: Python implementation (choose one track)
Submit as a single Jupyter notebook (.ipynb) rendered as .html that runs top to bottom without errors. Use markdown cells to label each required piece (model selection, validation, etc.) so it’s easy to find.
Segmentation track: RFM features from a customers dataset, outlier analysis, train/test split, model selection with three metrics (elbow, silhouette, gap), justified k, holdout validation, ANOVA, cluster profiles.
LMM track: 4+ specifications, AIC/BIC comparison table, justified final model, residual diagnostics, ICC, effect sizes with 95% confidence intervals.
Start from the Class 10 notebook, but don’t stop there. The in-class notebook is a worked example on a generic question. You may reuse its code, but your submission must implement your plan from Component 1, not the class defaults.
- Your question drives the analysis. The first markdown cell restates your business question. Your final interpretation answers it.
- Make at least two analytic choices of your own, each traceable to your reconciliation.
- Segmentation examples: feature set beyond basic RFM, transformation, outlier handling, scaling method, range of k tested, a different clustering algorithm as a robustness check.
- LMM examples: outcome variable, predictors, grouping structure, random slopes, interaction terms.
- Decision log. Near the top, add a table with these columns: Decision | Class default | My choice | Why | Source (AI helpful / AI adversarial / my data / my judgment).
- Fill in every “Your decision” cell in your own words, citing your output (specific metric values, AIC/BIC numbers, diagnostic plots).
A notebook that reproduces the class defaults without its own question, choices, and justification earns at most “Adequate” on Model selection and Interpretation.
Component 3: AI use log
See the AI use log guide.
Rubric
| Criterion | Excellent (5) | Adequate (3) | Needs revision (1) |
|---|---|---|---|
| Design process | All four roles, substantive reconciliation | Some roles | No design process |
| Model selection | 3+ metrics or AIC/BIC with justified choice | One metric | No formal selection |
| Validation | Holdout or diagnostics | Some validation | None |
| Statistical rigor | ANOVA/CI/effect sizes, assumptions checked | Some statistics | Descriptive only |
| Interpretation | Named clusters or interpreted effects in context | Some interpretation | None |
| AI use log | Detailed | Present but incomplete | Missing |
Total: 30 points