Learning

An AI primer: tokens, parameters, and new facts

In progress

This is an outline, not a finished note. I want the words people use around models — token, parameter, context, cutoff — to mean something I can check, the way I would check a JEP before I relied on it. Nothing here is a project I ship at work. It is the background I want before I write a later note that actually calls a model from Java.

Where a line says TODO, that part is still mine to write after I have run the tool, not a figure I am willing to invent.

Why this note exists

A context window is not a page count, and a "175 billion parameter" model is not 175 billion facts. The bill and the limit are both in tokens. The weights do not update because I typed a question. Those three sentences are the whole primer. The rest is what I would want in front of me when I am about to trust one of them.

Tokens

A model does not read words. It reads tokens. A token is a piece of text from a fixed vocabulary, often a subword: a whole short word, a fragment of a longer word, a bit of punctuation, or a space glued to the next piece. The usual way to build that vocabulary is byte-pair encoding, or something close to it. The idea, from Sennrich, Haddow, and Birch, is to start from small pieces and repeatedly merge the pair that shows up most often in the training text, until the vocabulary is the size you chose (Neural Machine Translation of Rare Words with Subword Units). Frequent English words become one token. Rare strings stay broken up.

OpenAI's own counting guide gives the rule of thumb I will use until I have measured it myself: for common English, one token is about four characters, or about three-quarters of a word, so 100 tokens is about 75 words. They also say this is an estimate, and that another language can sit at a different ratio (Understanding and counting tokens).

A sketch, not a dump from one tokenizer. The sentence A Java record is a class. might be cut as A, Java, record, is, a, class, . — seven pieces, with the leading space kept on the word that follows it. A different tokenizer can merge class with the full stop, or split record if that string was rare in its training text. I am not claiming a model emits that split. I am claiming the split is into pieces, and that two models need not agree on the pieces.

That is why non-English text and code often cost more tokens for the same meaning. A German compound such as Parkraumbewirtschaftung is one word to me and, if the merge list rarely saw it whole, several tokens to the model. A Java identifier such as booleanPointInPolygon is one name in the source and usually several tokens as well, because the letters in the middle were not a frequent merge. The same note in English prose is cheaper than the same note in source code, even when a human would say they are the same length.

The context window is a token budget, not a character budget. What I send and what the model sends back both spend it. Input tokens are the prompt: instructions, the conversation so far, any page I pasted, the question. Output tokens are what the model generates back. APIs price those two separately, and the output side is often the expensive one. The hard limit is that they have to fit in the window together. OpenAI also describes reasoning tokens, which a reasoning model can spend before the visible answer and which still count (the same help page). If I only budget for the answer I can see, I will under-count.

TODO: run one short Java method through a published tokenizer — tiktoken for an OpenAI encoding, or the Llama tokenizer — and paste the real pieces under the sketch. Keep the sentence that says the sketch was only a sketch.

Parameters, the weights

A parameter here is a learned number in the network: a weight or a bias. Training moves those numbers. After training they are the model. A billion parameters is a billion of those numbers, not a billion stored sentences.

The counts I am willing to write down are the ones a lab published.

GPT-2's largest published model is a 1.5 billion parameter Transformer (Language Models are Unsupervised Multitask Learners). GPT-3 is the 175 billion parameter model in the few-shot paper (Language Models are Few-Shot Learners). Meta published Llama 3.1 at 8B, 70B, and 405B; the 405B model is the dense one they describe as their largest in that family (Introducing Llama 3.1). DeepSeek-V3 is a mixture-of-experts model: 671 billion parameters in total, of which 37 billion are activated for each token (DeepSeek-V3 Technical Report, and the model table). Total and active are different numbers. A dense model uses all of its weights on every token. A mixture-of-experts model stores more weights than it uses for one token, and a router picks a few experts. Comparing 671B with 405B as if they did the same work per token is the wrong comparison. 37B active against a dense model's full count is closer, and it is still not the whole story.

Current frontier models from OpenAI, Anthropic, and Google do not come with a published parameter count. I am not going to fill that gap with a rumour. If a lab has not written the number, this note does not have the number.

The other things people call parameters are not weights. They are request settings.

Temperature is how sharply the model picks the next token. Lower is more repetitive and more predictable. Higher lets less likely tokens through. Top-p (nucleus sampling) cuts the list of next tokens to the smallest set whose probabilities add up to p, then samples inside that set. Max tokens, under whatever name the API uses (max_tokens, max_completion_tokens), is a cap on how many tokens the model may generate for this reply. It is not a cap on how much it knows. OpenAI documents all three on the chat completions reference. I treat them as call settings, the way I treat a query timeout, not as the size of the model.

TODO: after one small call from Java, write which of those three I actually changed, and what the answer did. One sentence. No table of folklore settings.

How a model hears something new

The weights are the result of a training run. Pretraining reads a corpus and stops. The date people call the knowledge cutoff is the end of the data that run saw, not a switch the model checks. When the run is finished the weights are frozen. A chat does not train them. If I tell a model my name, the next request does not know it because the weights moved. It knows it only if the product sends that earlier turn back in the prompt, or stores it somewhere and inserts it later.

New facts get into a product by one of these routes, not by the chat itself editing the network.

pretraining corpus
        |
        v
   frozen weights  ---- new training run ----> a new model version
        |
        |   this request only:
        |   instructions + earlier turns
        |   + retrieved pages, tool results, search snippets
        v
   context window  ----> answer tokens

A new model version is another training run: more pretraining, a fine-tune on labelled examples, or a preference stage such as RLHF. That ships as a new set of weights. It is not my session teaching the old weights.

Retrieval-augmented generation leaves the weights alone and puts documents into the prompt. The model reads them the way it reads anything else in the window. Tools and web search are the same shape: something outside the model fetches, and the result comes back as tokens. If the fetch fails, the model is back to the frozen weights.

A long context window is a larger budget for that one request. It is not a memory. A memory feature, when a product has one, is storage outside the weights: notes, files, a summary of older chats. The product chooses some of that text and puts it in a later window. Useful, and still not learning. The model did not update. The prompt got longer.

TODO: one paragraph, after I have used a search tool and a plain chat on the same question, on which answer I would trust for a fact that changed this year. Name the product. Do not generalise past that trial.

Takeaway

I will count in tokens, not in words, and I will expect code and German to cost more than plain English. I will treat a published parameter count as a size of the weight file, and I will not invent one for a lab that did not publish it. Temperature, top-p, and max tokens are settings on the call. And I will not expect the weights to learn from a chat: either the fact was in training, or some other program put it into this window.

TODO: the tokenizer paste, and the one call from Java. Those two are the next edits. Until they are here, this note stays in progress.

All notes