Hello LLM
  • English

    • // how large language models work

    Watch a language model think, one token at a time.

    Tokenizers, vectors, attention, backpropagation and the tricks behind today's models, each explained with a tiny example you can play, pause and poke.

    Start readingJump to attention
    • 15 chapters
    • every figure is live
    • runs in your browser
    tokens intransformer layers ×4next tokenThe·cat·sat·on·themat 41%floor 18%sofa 14%·mat

    The whole idea in one loop

    A language model reads the text so far, scores every token it knows, picks one and appends it. Then it does it again. Press play, or step through it.

    Chapter 1 explains each step →
    The·cat·sat·on·the
    1. Read
    2. Layers
    3. Scores
    4. Pick
    5. Append
    1/25

    The model sees 5 tokens. It never sees letters, only these pieces (as numbers).

    Contents

    From text to numbers

    1. 01What an LLM doesOne job: guess the next token. Then do it again.
    2. 02TokenizationByte-pair encoding, merge by merge, and a real GPT tokenizer.
    3. 03Vectors and embeddingsWords as points in space, and why direction means meaning.
    4. 04PositionSinusoids and RoPE: how the model knows word order.

    Inside the transformer

    1. 05AttentionQueries, keys, values and a softmax, step by step.
    2. 06The transformer blockResidual stream, normalization, MLP, stacked many times.

    Learning

    1. 07Loss, gradient descent, scalingRolling downhill on a loss surface, and what more compute buys.
    2. 08BackpropagationThe chain rule, flowing backwards through a graph.

    Using the model

    1. 09SamplingTemperature, top-k and top-p shape the dice.
    2. 10Fast inferenceKV cache, grouped queries, speculative decoding, GPUs.
    3. 11CachingPrompt and prefix caching, paged memory, shrinking the KV cache.
    4. 12Long contextWhy a million tokens is hard, and what breaks along the way.

    Modern techniques

    1. 13Mixture of experts, quantization, LoRABigger models that cost less to run and tune.
    2. 14Beyond the classic transformerLatent attention, state-space models, diffusion, multimodal tokens.
    3. 15From base model to assistantFine-tuning, RLHF, DPO, reasoning, tools and RAG.