One word at a time
The loop a model runs to write every reply.
We gave Claude Haiku 5.5 a file of 40 customer messages from Pinewick, a coffee company we made up for this course, and asked for a summary. In all three recordings, the reply stopped partway through. The video follows one of them to show how a chat model writes: one Token A piece of text, often a word or part of one, that a model reads and writes. at a time. A token is a short piece of text, like a word, part of a word or a punctuation mark.
Read the transcript
Here's everything Claude Haiku 5.5 was given in one of our recordings. First, two sentences of instructions from the app. Then a file with forty customer messages, which we made up. Last, the request: summarize this feedback.
Its reply was eight hundred tokens long. A token is a short piece of text: a word, part of a word, or a punctuation mark.
It wrote them one at a time, in a loop. Here's one real round, recorded from another model, GPT-4.1 mini, asked to copy this sentence and continue it.
First, it reads all the text so far. Then it scores every possible next token, giving each one its odds of coming next.
One token is picked, usually a likely one, and added to the end. Now the text is one token longer, so it all runs again with that token in it.
Claude Haiku 5.5 went round that loop eight hundred times to write the summary, in about four seconds. Each pick could draw on everything before it: the instructions, all forty messages, the request, and the summary so far. It stopped in the middle of a list, because eight hundred tokens was the limit we set.
Each round picks a likely next token, and nothing in the loop makes sure a count was checked. Under shipping, the summary says roughly thirteen messages, then cites ten different ones. Before you repeat a number like that, check it against the messages.
Now run that loop yourself, on the sentence from the video.
That loop is the engine of a Large language model (LLM) A model that writes text one token at a time by predicting what comes next., or LLM. After The most common complaint from our coffee subscribers this week was about the, the explorer rated delayed highest, at about 45 percent, though mini never saw a single message: the odds come from how sentences like that one usually go. Here's the summary from the video, played back at the speed it arrived. You don't need to read it all: skip to the end and notice where it stops.
System prompt
You are an assistant at Pinewick Coffee, a company that ships coffee beans by subscription. The user works in customer operations.
You
Summarize this feedback.
Claude Haiku 5.5
Summary of Customer Feedback (40 messages)
Overall sentiment: Mixed. Customers consistently love the coffee and appreciate the personal touches, but shipping reliability, billing accuracy, and app functionality are driving frustration and cancellations.
Positive feedback (about 8 messages)
- Coffee quality is praised, with specific mentions of the Ethiopia Guji and the Guatemala (#2, #26, #15, #23)
- Handwritten notes and the compostable bag are appreciated (#10, #31)
- Fast, responsive customer service was noted (#40)
- Gift subscriptions are driving new signups (#36)
All three samples stop mid-reply, because each one reached the 800-token limit we set. A reply ends when the model reaches a natural stopping point, or when it hits a limit like that one.
A colleague sketches how a chat model writes a reply. Which sketch is right?
What to remember
A chat model writes by predicting one Token A piece of text, often a word or part of one, that a model reads and writes., adding it, and predicting again, until it reaches a stopping point or a limit.
Each token is picked from learned odds. Nothing in the loop makes sure a fact was checked, so fluent text can still be wrong.
Every lesson’s takeaways are on the course cheat sheet.