Week 2: LLMs, Harnesses, and Setup

Module: Understand | Deliverable: Tool setup and benchmarking report (due before Week 3 begins) | See assignment →

Learning objectives

By the end of this week, you should be able to:

Explain what tokens, context windows, and next-token prediction are
Distinguish closed/frontier models from open-weights models
Explain the harness concept: same model, different interface, different workflow
Set up VS Code, Claude Code, and/or GitHub Copilot
Design and run a controlled model benchmarking experiment
Compute performance metrics (response time, token usage, cost) and quality scores with a rubric
Make a data-driven tool recommendation with quantitative evidence

Sessions

Session 1 · Class 3

What LLMs are and why it matters for your work

We'll look inside LLMs (tokens, next-token prediction, context windows) and explore the model landscape (closed vs. open-weights). Then we'll unpack the harness concept: same model, very different experience depending on the interface. Includes a tokenization activity and a qualitative surface comparison that previews Session 2's quantitative benchmarking.

Tokens and prediction Context windows Model landscape Engine vs. car (harnesses) Python: tokenization explorer

Session 2 · Class 4

Setting up and benchmarking your tools

We'll finish setting up your development environment, then run a systematic benchmarking experiment. It includes 5 analytic prompts × 2 model configurations, measuring response time, token usage, cost, and output quality scored against a rubric. You'll compute summary statistics and make a data-driven tool recommendation.

Tool setup workshop Python: systematic model benchmarking Quality scoring with rubric Data-driven recommendation

Prepare before class

Before Session 1:

Read your Week 1 use case reflection feedback (posted on Canvas)
Skim Békés, Data Analysis with AI, Week 1 - focus on the “LLMs and harnesses” framing

Before Session 2:

Start the developer setup guide - install VS Code and get Python working before class (hopefully this is already complete from Week 0). We’ll finish the AI tool setup and troubleshoot in class, but arriving with the basics ready saves time for the benchmarking activity.

Key concepts

Concept	What it means
Tokens	Subword pieces that LLMs process. Token count determines cost and what fits in a prompt.
Next-token prediction	The core mechanism: given preceding tokens, predict the most probable next one. The model computes probabilities, not necessarily truth.
Context window	Everything the model can see at once: prompt, documents, conversation, response. Typically 128K–200K tokens.
Closed / frontier models	Claude, GPT-4o, Gemini - accessed via API, strongest general capabilities.
Open-weights models	Llama, Mistral, Qwen - weights downloadable, more control, often smaller.
Harness	The interface wrapping the model. Chat, API, IDE agent, and CLI are different harnesses for the same engine.
Benchmarking	Systematic comparison of model configurations using controlled inputs and measured outputs

Readings and resources

Békés, Gábor. Data Analysis with AI, Week 1: LLMs, Models & Harnesses + Setup - the conceptual backbone for this week
Microsoft. Generative AI for Beginners, Lessons 2–4 - tokens, prompting basics, and model selection
Course setup guide: Developer setup guide

Deliverable

Tool Setup and Benchmarking Report

~2 pages + code | 25 points | Due before Week 3's first class | Submit via Canvas

Confirm your dev environment works, run a systematic benchmarking experiment with quantitative metrics (response time, token usage, cost, quality scores), and make a data-driven tool recommendation citing specific numbers.

Full prompt and rubric →

Looking ahead

Next week we move into the Use module with From Raw Data to Report - you’ll use AI to explore a real dataset, document it, clean it, and produce a directed report with formal statistical analysis. The tools you set up this week are the foundation for everything from Week 3 onward.