# How AI works

> An LLM is a next-word guesser at giant scale: text becomes numbers, flows through billions of learned weights, and out comes a probability for every possible next token. All it "knows" was pressed into those weights by predicting trillions of words.

Text goes in as numbers. The model first chops your words into *tokens* — common chunks of characters — and each token becomes a long list of numbers called a vector. From here on, there are no words inside the machine at all. Only arithmetic.

The engine is a *transformer* (Vaswani et al., 2017): a stack of layers in which every token's numbers are mixed with every other token's numbers through learned weights — this mixing is called *attention*, and it is how "the bank of the river" ends up meaning something different from "the bank on the corner". Nothing else is inside: no grammar rules, no fact database. Just billions of multiply-and-add operations whose constants were learned.

Where do the constants come from? Training. The model reads text and plays one game, trillions of times: *guess the next token, measure the surprise, nudge every weight to be slightly less surprised next time.* To get good at that game across all human writing, the weights are forced to absorb spelling, grammar, facts, style, and reasoning patterns — knowledge as a side effect of prediction. Chat assistants are this same engine with an extra layer of training where humans rate answers, teaching it to prefer helpful continuations.

When you talk to it, the model produces a probability for every token in its vocabulary, one token is sampled, appended, and the loop runs again — your whole answer is generated one guess at a time.

This one mechanism explains both the magic and the failure mode. The fluency is real: prediction at this scale genuinely captures how ideas connect. But the model always produces a *plausible* next token — including where it has no ground truth — which is why it can state falsehoods with perfect confidence. It is not lying; it is guessing well. That is also why this site attaches numbers and sources to every claim: plausibility is not evidence.

## Evidence
- GPT-3 has 175 billion learned weights and was trained by next-token prediction over roughly 300 billion tokens of text (Brown et al., 2020)
- Llama 3 models were trained on more than 15 trillion tokens (Meta, 2024)
- At every step the model outputs a probability for each entry in its whole vocabulary — 50,257 possible tokens in GPT-2/GPT-3's tokenizer (Radford et al., 2019)

Sources:
- Brown et al. — Language Models are Few-Shot Learners — https://arxiv.org/abs/2005.14165 (accessed 2026-09-10)
- Meta — The Llama 3 Herd of Models — https://arxiv.org/abs/2407.21783 (accessed 2026-09-10)
- Radford et al. — Language Models are Unsupervised Multitask Learners — https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf (accessed 2026-09-10)
- Vaswani et al. — Attention Is All You Need — https://arxiv.org/abs/1706.03762 (accessed 2026-09-10)

---
Published 2026-09-10 · ai, fundamentals · AO — Abstract Objective · https://abstractobjective.dev/knowledge/how-ai-works/