Part 1: The maths behind the stochastic parrot.
Every time you interact with a large language model—say, ChatGPT—you’re not just getting a response; you're seeing the result of a massive, complex calculation of probabilities. At its heart, an LLM is a masterful "probability machine."
That’s both the source of its power and unpredictability. It makes LLMs capable of generating new “things,” but also makes them notoriously difficult to train or debug.
To understand how they work, we need to start with the building blocks of probability itself. (For more on how words become numbers, check out Unlocking AI’s Understanding: The Magic of Embeddings from the Napkin Sketch series.)
1. Thinking in Probabilities
How randomness, patterns, and learning from data help machines make sense of language.
Let’s develop some intuition with relatable, real-world examples.
1.1 Probability: The Art of Knowing the Unknown
The world is full of uncertainty. Probability helps us navigate it.
Imagine you have a single coin. When you flip it, what are the chances it lands on heads? You'd say "50-50," or more formally, a probability of 0.5.
Probability is a way to quantify uncertainty— think of it as a number between 0 and 1:
A probability of 0 means an event is impossible.
A probability of 1 means it's guaranteed to happen.
A probability of 0.5 means it's equally likely to happen or not happen.
Probability is a tool to make sense of unpredictability, to turn “I’m not sure” into “I’m 12.5% confident it’ll happen.”
When you think about it we use probability to reason about uncertain events all the time, from predicting the weather to guessing the outcome of a sports game.
What’s the chance it rains tomorrow?
What’s the chance someone says “thank you” after you hold the door?
For an LLM, the "uncertainty" it's trying to measure is what the next word in a sentence should be. When a language model picks the next word in a sentence, it doesn't know—it just assigns probabilities to all possible options.
“I’m going to the…” →
60% “store”, 20% “gym”, 5% “moon”...
It’s not guessing randomly. It’s using probability to make an informed choice.
1.2 Probability Distributions: Finding Patterns in Randomness
Probability isn’t just about one event—it’s about all the possibilities.
When you toss a coin it’s a 1-in-2 chance of head/tail, now say you roll a die. Each number (1–6) has an equal 1-in-6 chance, and if you roll the die a thousands time you are likely to see each faces about 166 times; these are called uniform distribution.
But not everything in life is equal. Test scores often follow a bell curve (a normal distribution):
Most people score around the average.
Few score really low or really high.
Height of people is an other classic example of a "bell curve.". Measure a large group of people, you'd find most people are around the average height (the peak of the bell). Very few people are extremely tall or extremely short (the thin tails of the bell). This curve is a probability distribution for human height—it shows the pattern in a seemingly random set of data.
Why all of this matters ? Because distribution = the shape of possibility.
Probability distributions describe how likely outcomes are across many possibilities.
If probability is a single measurement, a probability distribution is a comprehensive map of all possible outcomes. It shows how likely each outcome is, giving us a complete picture of the randomness.
Well, language isn’t random either.
“the” is super common
“aardvark” is rare
In the world of language, a distribution might show how often certain words appear. In a collection of texts, you might find that the word "the" has a very high probability, while a rare word like "soliloquy" has a very low probability. The distribution maps out this entire pattern of word usage.
A language model tracks how often words appear—this is its word distribution. Like tracking which notes are most common in a song.
1.3 Conditional Probability: Leveraging Context
What’s the chance of B, if A already happened?
This is a crucial concept for language models. Conditional probability asks: "What is the probability of event B happening, given that event A has already happened?" It’s all about dependencies.
For example, if the sky is cloudy (condition A, rain (event B) is more likely.
A great real-world example is the weather:
The probability of rain on a random day might be low, say 0.2.
But the probability of rain, given that the sky is already full of dark clouds, is much higher, maybe 0.7.
The condition ("the sky is full of dark clouds") changed the probability of the outcome ("rain").
This is the same for a language model.
The probability of the word "cat" is very low on its own.
But the probability of the next word being "cat," given that the previous words were "The dog was chasing a...", suddenly becomes very high.
That’s conditional probability—probability that depends on context; and how a model builds coherent sentences— context is so important that context windows are key performance attributes of LLMs.
1.4 Sampling: Turning Probability into Action
Probability tells you what could happen. Sampling decides what does.
Sampling means choosing outcomes according to their probabilities—like rolling dice to move in a board game. Sampling lets us turn probabilities into actual events.
Imagine you're playing a board game and need to roll a six-sided die. The probability distribution says each number has a 1/6 chance of appearing. When you roll the die, you are performing a sample. You are taking the probabilities and turning them into a single, concrete outcome.
Sampling is how an LLM moves from thinking to acting. After calculating the conditional probabilities for every possible next word in a sentence, it has a long list of words, each with a likelihood score. Let's say the context is "The chef added fresh". The model predicts the next word in a sentence like this:
When generating text, it “samples” from this distribution—like spinning a weighted roulette wheel. Sometimes you get the most likely word (“herbs”), but sometimes you get variety (“salt”), but importantly both are still possible outcomes.
Temperature is the parameter influencing our imaginary weighted roulette wheel- when low, the model is more likely to choose "herbs" as the next word. With a higher temperature, it might be more likely to choose "garlic" or even "salt" — this is how hallucinations starts.
Sampling adds creativity and diversity to LLM outputs. Without it, every answer would sound the same.
That’s how LLMs can surprise you—within bounds set by probability
1.5 Training: Learning Probability from Data
Machines don’t start smart—they learn from examples. Lots of them.
Training an LLM is a lot like teaching a child. You don't hand them a list of words and tell them their probabilities. Instead, you let them observe the world. The computer does the same thing, but on a massive scale.
Training an LLM involves feeding it trillions of words from books, articles, and websites. Unlike a child though, the computer doesn't "understand" the text in the human sense. Instead, it reads this enormous dataset and painstakingly builds a complex probability map. It learns that after the words "The dog chased the," the word "ball" appears far more often than the word "teapot." This is how it learns the conditional probabilities and dependencies of language.
The more data the model sees, the better its sense of what’s likely.
It’s not memorizing. It’s building a probability map of language. That’s why it can write poems, summarize articles, or finish your sentence—just by predicting the most likely next word, over and over again.
On a side note: this touch the basis of the recent rulings in favor of Anthropic and Meta in copyright infringement cases, no one book is important to train an LLM, no one text is there to be bluntly replicated, no LLM’s is writing the book you were meant to write in your words and ideas, but LLMs do need loads of books to be trained and that is, for now, fair use Anthropic and Meta Decisions on Fair Use.
Take aways:
The intuition to develop here is that each of the technique we saw essentially changes how probabilities guide the model’s output toward desired results:
They quantify uncertainty
Track patterns
Adapt to context
Make probabilistic decisions
And learn from data to get better over time
And by now you should be thinking:
“Wait… predicting words is just probability with a really smart map.”
In the next part of our series, we'll dive into how these five core concepts of probability come together to create the incredible "language machine" we know today.