Most of that material turns out to be useful far beyond the engineering floor. If you buy software development services, a handful of these concepts explain things you have probably already noticed. They explain why AI usage has a monthly bill attached, why an AI assistant cannot simply “know” your whole project, and why the same question gives a different answer on Tuesday than it did on Monday. This article walks through the five topics from that session with one question after each: what does this mean when you buy software development?
How an LLM writes: one word at a time
A language model does exactly one thing, over and over. Given the text so far, it scores every entry in its vocabulary, roughly 100,000 possible chunks, and assigns each one a probability. For the sentence “The cat chased the ___”, the word “mouse” might score around 94 percent, “rat” about 2 percent, and the remaining tens of thousands of options next to nothing. The model takes a word from the top of that distribution, appends it, and runs the whole scoring process again on the new, slightly longer text. A paragraph, an essay, or a working function is that single move repeated many times.
The output stays coherent because of a mechanism called attention. Every time the model scores the next word, all the earlier words pull on the prediction. In “The cat chased the mouse,” the word “cat” pulls “mouse” toward the animal meaning; in “I clicked the mouse,” the word “clicked” pulls it toward the computer device. One neighboring word can decide the meaning, and this weighing happens for every word at every step.
Our training closes this chapter with a list of what the model does not do, and that list matters more for buyers than the mechanics. The model does not look anything up. It does not plan the whole answer before it starts. It does not check whether what it says is true. It does not store what you told it.
For a software buyer, this reframes what “AI-generated” means. When an answer is brilliant, the context made the right words likely; when an answer is wrong, the context made the wrong words likely. A vendor who understands this treats AI output as a strong draft that needs verification, and builds review steps into the process rather than treating the model as an oracle.
Tokens: the unit you actually pay for
The model does not read words or letters. Before it does anything, it chops your text into chunks called tokens, drawn from a fixed vocabulary of roughly 100,000 entries in older models and about 200,000 in newer ones. Common words like “favorite” earn a single token. Rarer letter sequences get assembled from pieces, which is why “strawberry” arrives at the model as three chunks: “Str,” “aw,” “berry.”
This one technical detail explains some famous AI failures. Asked how many r’s are in “strawberry,” models often answer two, because the letters are fused inside chunks the model never sees individually. Asked whether 9.9 or 9.11 is bigger, models regularly pick 9.11, because the numbers arrive as token chunks and the chunk “11” can look bigger than the chunk “9.” Character counting and precise decimal comparison fight against how the text was cut up.
Tokens are also the unit of billing, and the exchange rate is not the same for every language. In our training we use a concrete example: “Strawberry is my favorite fruit” is about 8 tokens in English. The same sentence in Vietnamese, “Dâu tây là loại trái cây tôi yêu thích nhất,” is about 26 tokens for identical meaning, because the vocabulary was built mostly from English text. Even images become tokens. A picture is cut into tiles of roughly 512 pixels, and each tile is converted into tokens you pay for, which is why our engineers extract text from screenshots instead of sending the image whenever the picture itself is not needed.
If you are the one paying the bill, tokens are where AI capability meets your budget. A task that runs at scale in a non-English language can cost roughly three times what the English version costs. An AI feature that processes documents, chat histories, or images has a per-token cost that a competent partner can estimate and optimize before you commit. If a vendor cannot explain what drives the token count of a proposed feature, the running costs of that feature are a guess.
The context window: what the AI can know about your project
The context window is the model’s working memory. It holds everything the model can read at one moment: the question, the conversation so far, any documents provided, and the answer being written. On today’s top models that window is around one million tokens, roughly a few thousand pages. That sounds like enough to hold a whole project, and this is where two important caveats from our training come in.
The first caveat is that the window is working memory, not storage. Tell a model your name in one chat, open a fresh chat, and ask for it back; the answer is “I don’t know.” Nothing was ever stored. When a chat product seems to remember a conversation, the application around the model is quietly re-sending the entire history with every new message so the model can re-read it from scratch. The remembering belongs to the app, and you pay for it, because every re-sent token is billed again. Long conversations get more expensive with each message and eventually start losing the thread.
The second caveat is that fitting into the window is not the same as being found. In research our training cites, a needed fact placed at the start of a long input was used correctly about 75 percent of the time and about 72 percent at the end. Placed in the middle, accuracy sagged to about 55 percent, barely above the 56 percent the model scored with no document at all. The pattern is reliable enough to have a nickname, “lost in the middle,” and it replicated across 18 different models. Our trainer’s comparison in the live sessions was the smartphone battery: screens and apps improved faster than batteries, and context windows have grown faster than the attention mechanism that has to search them.
For a software buyer, this is the concept behind every “the AI knows our whole codebase” claim. It does not know the codebase; it knows whatever a well-designed system placed into its window for this specific task, positioned so it will actually be noticed. Teams that understand this curate context instead of dumping everything in, and they place stable instructions where the provider can cache them. Cached input is billed at a fraction of the normal rate, around one tenth on the Anthropic pricing discussed in our sessions, which keeps per-request costs predictable. The difference between a demo that impresses and a system that works in production is largely context management.
Sampling and temperature: why the same question gives different answers
If the model simply took the single best word every time, the same prompt would produce the same answer on every run. It does not, and our training shows this with a simple exercise: ask for a coffee shop tagline three times and you may get “Your daily grind, perfected,” then “Life happens. Coffee helps,” then “Where the city wakes up.” Nothing changed between runs.
The reason is sampling. Instead of always taking the top-scoring word, the model rolls a weighted die against the probability distribution. For “The weather today is ___”, “sunny” at 63 percent wins most rolls but not all of them. This randomness is a feature. It is what makes output feel natural and varied rather than robotic, and it is also why no two runs are guaranteed to match.
Temperature is the dial on that roll. A low setting, around 0.5, reshapes the odds so the leader runs away; “sunny” jumps from 63 to about 91 percent, which suits facts, code, and data extraction. A high setting, around 2.0, flattens the odds so long shots get a real chance, which suits brainstorming and naming. Two further points from the training are worth knowing. Even at temperature zero the output can still vary, because near-ties at the top of the distribution can flip on tiny numerical wobble; one study found a single prompt splitting into dozens of different endings across a thousand runs. And the newest reasoning models increasingly lock the dial and manage their own randomness, so this is becoming something teams understand rather than tune.
For a software buyer, sampling explains why “run it once and see” is not a test. An AI feature that produced a correct answer in the demo may produce a different answer for your customers tomorrow. Serious vendors test AI behavior across many runs, define acceptance criteria that tolerate variation, and choose settings deliberately for each task instead of leaving defaults everywhere.
Hallucination, in one paragraph
Everything above combines into the risk that gets the most headlines. Nowhere in the predict-roll-append loop is a step that asks “is this true?”, so the model optimizes for what sounds like a likely continuation, not for what is correct. When plausibility and truth part ways, the result is a hallucination: output that is fluent, confident, and false, delivered in exactly the same tone as a correct answer. Our training covers the mechanics and the documented business damage, from the 2023 Google demo error that coincided with roughly 100 billion dollars leaving Alphabet’s market value in a day, to the Air Canada chatbot whose invented refund policy a tribunal enforced in 2024. The fix, called grounding, deserves its own article. Our training gives buyers one line to keep: the model is just as sure when it is wrong.
For a software buyer, the practical question is not whether a model will ever hallucinate, because it will. It is what stands between a confident wrong answer and your customer. Ask a vendor how they check outputs against trusted sources, which cases require human review, and how they monitor for errors after launch.
Why we train fundamentals, not tools
AI tools change every quarter; the machinery underneath changes slowly. Every concept in this article, from token-based billing to lost-in-the-middle, applies regardless of which vendor’s model or assistant a team uses this month. That is why our internal AI Roadmap starts with how models work before it covers any product, and why practitioners run the live sessions. The prompt engineering sessions behind this article were led by an AI engineer with nearly ten years in the field. His experiments show that better context alone can improve output quality by 20 to 30 percent on the same model, which is the practical payoff of understanding the fundamentals.
For a buyer, these concepts double as due-diligence questions. What drives the token cost of the feature you are proposing? How do you decide what goes into the context window, and where? How do you test a system whose output varies between runs? What checks catch a confident wrong answer before a customer sees it? A partner whose engineers have been trained on the fundamentals will have concrete answers, because they have seen the machinery behind each question.
If you are weighing an AI feature for your own product, these are the questions we work through at the start of every scoping conversation: what the feature will cost to run per request, what belongs in the context window, and how to test behavior that varies between runs. Our AI Product Development team builds AI-powered products with these fundamentals designed in from the first estimate, so the proof of concept you see is the behavior your customers get. Talk to us about your AI product.
Source
- Axon Active internal AI Roadmap training, Session 2 “Grounding LLMs”: Chapter 1 (How LLMs work), Chapter 2 (Tokens), Chapter 3 (Context window), Chapter 4 (Sampling and temperature), Chapter 5 (Hallucination).
- Axon Active internal training sessions “[SS1] Prompt Engineering in 2026” (June 2026, trainer session summaries).
- Liu et al., “Lost in the Middle: How Language Models Use Long Contexts,” 2023. https://arxiv.org/abs/2307.03172
Frequently asked questions about LLM basics
What is a token, and why does it matter for cost?
Before a model does anything, it chops your text into chunks called tokens, drawn from a fixed vocabulary of roughly 100,000 entries in older models and about 200,000 in newer ones. Tokens are the unit of billing, so the token count of a feature drives its running cost.
Does an AI assistant remember what my team told it?
No. The context window is working memory, not storage. When a chat product seems to remember a conversation, the application around the model re-sends the entire history with every new message, and every re-sent token is billed again.
Why does the same question give different answers on different days?
Because of sampling. Instead of always taking the top-scoring word, the model rolls a weighted die against the probability distribution. Even at temperature zero the output can still vary, because near-ties at the top of the distribution can flip on tiny numerical wobble.
Why does an AI feature cost more in some languages?
The token vocabulary was built mostly from English text. In our training example, “Strawberry is my favorite fruit” is about 8 tokens in English and about 26 in Vietnamese, so a task that runs at scale in a non-English language can cost roughly three times the English version.
What is a hallucination?
Output that is fluent, confident, and false, delivered in the same tone as a correct answer. Nothing in the model’s predict-roll-append loop checks whether a statement is true, and the model is just as sure when it is wrong.
