How LLMs Actually Work

An LLM is not a database of facts and not a search engine. It is a machine that predicts the next token — the next chunk of text — given everything before it. Everything else (fluency, reasoning, lying confidently) emerges from that one behavior at enormous scale.

The strawberry mystery is the perfect gateway, because it exposes the fundamental unit of everything: the token.

Tokens: the atoms of the model's universe

Text is split into tokens by a tokenizer trained with byte-pair encoding: start from bytes, merge the most common pairs, repeat. Common words become single tokens; rare words get split. The model never sees letters — only token IDs. Watch what that does to 'strawberry':

import tiktoken

enc = tiktoken.get_encoding("o200k_base")   # GPT-4o / GPT-5 family

tokens = enc.encode("strawberry")
print(len(tokens))   # 3
print([enc.decode_single_token_bytes(t).decode() for t in tokens])
# ['str', 'aw', 'berry']

# s-t-r-a-w-b-e-r-r-y has THREE r's, but the tokenizer merged them:
# 'berry' is ONE token — the double-r is invisible to the model.
# It predicts tokens, not letters. It's not stupid. It's token-blind.

Rule of thumb: English ≈ 0.75 words per token ≈ 4 characters. That's your unit for cost math (module 8) and for understanding why 'count the letters' tasks are the classic failure.

Playground: next-token prediction, demystified

Here's the whole 'AI' in miniature — a char-level bigram model. It counts which letters follow which in a sentence, then samples the next letter from those counts. This is literally what GPT does, minus a few hundred billion parameters:

from collections import defaultdict
import random

text = "the quick brown fox jumps over the lazy dog"

# count: which letter follows each letter?
counts = defaultdict(lambda: defaultdict(int))
for a, b in zip(text, text[1:]):
    counts[a][b] += 1

def sample_next(letter):
    options = list(counts[letter].items())
    total = sum(n for _, n in options)
    r = random.random() * total
    acc = 0
    for ch, n in options:
        acc += n
        if r <= acc:
            return ch
    return " "

out = random.choice(list(counts))     # start anywhere
for _ in range(40):
    out += sample_next(out[-1])
print(out)

# every LLM is this loop: predict the next token, feed it back.

Inside the transformer: attention is a lookup, not a search

A transformer is a stack of layers; each layer runs self-attention (each token looks at every earlier token and decides how much each matters) followed by a feed-forward network. Parameters are the learned weights of those operations. Attention is what lets a model use its 1M-token context — it's a weighted lookup across all previous tokens, computed in parallel.

At inference, the model generates ONE token at a time: predict the next token's probability, sample from it, feed it back in. That loop is the whole show. 'Reasoning models' just run more of this loop internally before answering.

def generate(model, prompt_ids, max_new_tokens=100, temperature=0.7):
    ids = prompt_ids[:]
    for _ in range(max_new_tokens):
        logits = model(ids)                    # forward pass: scores for
                                               # every token in the vocab
        logits = logits / temperature          # temperature: 0 = greedy,
        probs = softmax(logits)                # 1 = pure distribution
        next_id = sample(probs)                # top-p keeps the top p% mass
        ids.append(next_id)
    return ids

How the model got smart: three training stages

Stage 1 — pretraining: predict the next token over ~15 trillion tokens of internet text (Llama 3). This is where knowledge and language live, and it costs millions — DeepSeek V3 famously trained for $5.6M by being clever with sparse MoE architecture. The result is a fluent but unhelpful base model.

Stage 2 — SFT: fine-tune on (instruction, response) pairs so it follows instructions. Llama 3.1 used 25 million curated examples. This teaches format more than facts.

Stage 3 — alignment: RLHF/DPO/GRPO optimize for what humans prefer. Reasoning models (DeepSeek R1, o-series) add a fourth stage: reinforcement learning over thinking traces with verifiable rewards — the model learned to 'think' because thinking got rewarded when it led to correct answers.

Prompting   -> cheapest, instant, no training.
               Use when the model already knows the skill.
               Cost: $0.    Quality ceiling: the model's knowledge.

RAG         -> give the model facts at call time (docs, prices, policies).
               Use when the answer lives in YOUR data, not the model's.
               Cost: embeddings + retrieval. No training needed.

Fine-tuning -> teach the model a NEW behavior or format (tool-call JSON,
               your brand voice, a domain style).
               Use when prompting + RAG can't reach the behavior.
               Cost: training hours + a good dataset.