Transcription
Every day, millions of people type prompts into ChatGPT, Claude, or Grok, and get responses that feel almost human. But most people don't realize the model has no idea what it's about to say. Not the full sentence, not even the next word. It's generating your response one piece at a time, and each piece is a probabilistic guess from over a 100,000 options.
In this video, we'll see exactly what happens from the moment you hit send to the moment text appears step by step. So, five things happen when you send a prompt.
One, tokenization. Your text becomes pieces.
Two, embeddings. Those pieces become meaningful vectors.
Three, the transformer. Context gets processed through attention.
Four, probabilities. Every possible next token gets a score.
Five, sampling. One token is selected, then it loops.
Let's look at each one in a bit more detail.
Step one, tokenization. LLMs don't read words, they read tokens. Here's OpenAI's tokenizer. I type "I love programming. It's awesome." And I get seven tokens. Notice most tokens are for the words, but there are separate tokens for the period. This isn't random. Tokenizers are trained on text data to find efficient patterns. This happens before the model ever sees your input. It's a pre-processing step, not the neural network deciding how to split. Common words like 'the' get one token. Uncommon or long words get broken into subword pieces. So "indistinguishable," that's four tokens. 'the' just one. Why does this matter to you? When an API says max 4,096 tokens, that's not 4,000 words. It's roughly 3,000 words of English. Tokens are smaller units. Every token gets a number, a token ID. So, "I love programming. It's awesome." becomes a sequence of seven numbers, seven integers. That's what enters the model.
But numbers alone don't carry meaning. That's step two, embeddings. A token ID is just a number. The model needs to understand what it means. So, every token gets converted into a vector, a list of numbers representing its meaning. These vectors have thousands of dimensions. GPT-3 uses over 12,000 numbers per token. And these aren't random numbers. They're coordinates in a meaning space. Think of it like this. Words with similar meanings end up near each other. "King" is near "queen." "Python" the language is near "JavaScript." "Python" the snake is somewhere completely different. There's a famous demonstration. If you take the vector for "king," subtract "man," add "woman," you land near "queen." The model learned gender relationships just from text patterns. Let me show you a more practical example. Look at this embedding space for programming terms. "Function," "method," "procedure" clustered. "Variable," "parameter," "argument" clustered nearby. "Database," "SQL," "query" different cluster entirely. This is how the model understands that JavaScript and Python are related. Not because anyone told it, but because they appear in similar contexts.
These rich vectors now flow into the transformer. Step three, the transformer. Your embedding vectors enter a neural network with billions of parameters. But I want to focus on the one mechanism that makes it all work. Attention. Imagine a spotlight operator at a concert. The music shifts. The operator decides which musician to highlight. During a guitar solo, spotlight on the guitarist. During vocals, spotlight on the singer. Attention works similarly. When processing each token, the model decides which other tokens to focus on. Take this sentence: "The cat sat on the mat because it was tired." What does "it" refer to? The cat, not the mat. This is what attention does. When the model processes "it," it assigns high attention weight to "cat" and low weight to "mat." Even though "mat" is closer in the sentence, the model learned this from patterns across millions of examples. "It was tired." Patterns match with animals, not objects. This attention calculation happens multiple times in parallel through what are called attention heads. Different heads can capture different relationships. And then this whole layer repeats. GPT-3 has 96 layers stacked. Llama 3's 70 billion parameter model has 80. Each layer refines the representation. Each layer builds more abstract understanding. What comes out? Vectors that now encode not just individual token meanings, but rich contextual information about the entire input.
Now we need to predict the next token. Step four, probabilities. The transformer has processed your input. Now it needs to answer what token comes next. The final layer produces a score for every token in the vocabulary. Every single one. Llama 3 has 128,000 tokens in its vocabulary. Each gets a score. These raw scores are called logits. We apply a function called softmax to convert them into probabilities that sum to one. So for our input, we might get "is" 23% probability, "really" 14%, "the" 9%, "love" 6%, and 127,996 more tokens with smaller probabilities. This is the core reality of LLM generation. The model doesn't decide what to say. It produces a probability distribution over all possible next tokens. Your final response is just one path through an enormous space of possibilities.
Now, how do we choose? Step five, sampling. This is where you have control. The simplest approach, greedy decoding. Pick the highest probability token every time. Consistent? Yes. Boring? Often. That's where temperature comes in. Temperature adjusts how confident the distribution is. Same prompt: "What is Python?" with different temperatures. Low temperature sharpens the distribution. Safe, predictable choices dominate. High temperature flattens it. Unlikely tokens get a real chance. But push it too high and outputs often become incoherent. That 1.5 example is already getting strange. Then there's top P, also called nucleus sampling. Top P says only sample from the smallest set of tokens whose probabilities add up to P. If top P is 0.9, you might be choosing from just 15 tokens or 500, depending on how confident the model is. Quick reference: writing code, temperature 0.2 to 0.4. You want precision. General tasks, temperature 0.7 to 1.0. Balanced. Creative writing, temperature 1.0 or higher. Embrace variation. When you set these parameters in an API call, you're directly shaping this selection process.
One token selected. Great. But we've only generated one token. Last piece, the loop. We generated one token. Now we append it to the input and run the entire process again. Tokenize, embed, transform, probabilities, sample for every single token. "What is Python?" First pass selects "Python." Now we have "What is Python? Python." Second pass selects "is." Now we have "What is Python? Python is." Third pass selects "a," and this continues until the model produces an end-of-sequence token or hits a length limit. This is why generation slows down for longer outputs. Every new token requires attention over all previous tokens. And this is why the model genuinely doesn't know what it will say in advance. There's no hidden script, no planned sentence waiting to come out at token 10. Token 50 hasn't been determined yet. Each word is decided only when it's that word's turn, based on everything that came before.
Now you understand what's actually happening. Three insights you can use right away. First, when LLMs hallucinate, they're not lying. They're generating text that pattern-matches to what a confident, true-sounding response looks like. The probability distribution doesn't know truth from plausibility. The implication: always verify factual claims, especially when the model sounds confident. Second, temperature doesn't make models more creative. It makes them more likely to select lower-probability tokens. Creativity is a human interpretation of that randomness. The implication: for deterministic tasks, coding, extraction, formatting, use low temperature. Don't leave it to chance. Third, context limits aren't arbitrary product restrictions. They're computational reality. Attention has quadratic complexity. Every token must attend to every other token. The implication: when you hit context limits, it's not the company being stingy. It's the architecture.
The next time you use an LLM, see what's happening. Tokens in, meaning vectors, attention connecting context, probabilities over a 100,000 options, one selection at a time. It's not magic. It's mechanism. Understanding the mechanism makes you a better builder. If this helped, don't forget to like and share. Thanks for watching.