Week 6: Text as Data with LLMs

Module: Use | Deliverable: Text classification pipeline (due before Week 7 begins) | See assignment →

Learning objectives

By the end of this week, you should be able to:

Hand-code a validation set and compute inter-annotator reliability (Cohen’s κ)
Build and cross-validate an sklearn text classifier
Run an LLM classifier with structured JSON output at scale
Evaluate classifiers formally: precision, recall, F1 per class, confusion matrix
Compute LLM classification costs and project to production scale
Construct a topic co-occurrence network and test community structure against a null model
Compare what classification reveals vs. what network analysis reveals

Sessions

Session 1 · Class 11

Turning text into data

The text-as-data pipeline with a text corpus: hand-code 25+ texts for validation, build an sklearn baseline with cross-validation, and run an LLM classifier with structured output. You will be tracking tokens and timing for the next class's cost analysis.

Hand-coding + calibration Python: sklearn with cross-validation Python: LLM structured classifier

Session 2 · Class 12

Evaluation, costs, and networks

Compute inter-annotator reliability with a partner, run formal evaluation (precision/recall/F1 per class), analyze LLM costs projected to production scale, then build a topic co-occurrence network with community detection tested against a random null model.

Inter-annotator kappa Precision/recall/F1 per class Cost analysis and projections Network modularity vs. null model

Prepare before class

Before Session 1:

Read Békés, Data Analysis with AI, Week 5: Text as Data - text-to-data pipelines and evaluation
Review the Python for analysts guide sections on sklearn and networkx

Before Session 2:

Review your Session 1 classification results
Coordinate with a partner: you’ll both need to independently code the same 20 texts for the kappa exercise

Key concepts

Concept	What it means
Inter-annotator reliability	Cohen’s κ between two human coders on the same texts; if κ < 0.6, the coding scheme needs revision
Precision/recall/F1 per class	Not just overall accuracy, performance broken down by category to identify systematic weaknesses
Cost analysis	Per-text LLM cost from actual token counts, projected to production scale, compared to human coding
Modularity	How well a network’s communities separate tested against a random null model to check significance

Readings and resources

Békés, Gábor. Data Analysis with AI, Week 5: Text as Data - the classification and validation pipeline for Session 1: hand-rating first, structured-output prompting, agreement against a labeled sample
networkx community detection docs - modularity and the detection algorithms behind Session 2’s network analysis

Deliverable

Text Classification Pipeline

4 components | 30 points | Due before Week 7's first class | Submit via Canvas

Hand-coded validation with inter-annotator kappa, sklearn + LLM classifiers with formal per-class evaluation, cost analysis projected to production scale, topic co-occurrence network with modularity significance testing, and comparison memo.

Full prompt and rubric →

Looking ahead

Next week we enter the Evaluate module with Evaluating GenAI Outputs - rubric design, systematic scoring, and the question of when to trust LLM-as-judge.