A user asks an AI agent to assign a search issue to the responsible team. A moment later, the agent replies:
Assigned to Team Birch.
What would make that statement trustworthy?
The model might have generated the sentence without requesting an assignment. It might have proposed an assignment that the application rejected. Or the assignment might have succeeded while a dropped connection prevented its confirmation from arriving.
The same sentence can describe very different situations.
To understand what happened, we need to follow three things separately: what the model generated, what the program executed, and what changed in the external system. This first article in INVARIANT’s AI series follows one request through those boundaries, including the mathematics behind the prediction.
All records, tool responses, and failure traces below are constructed teaching examples. The probability calculator performs real arithmetic on invented scores; this article does not report a model benchmark.
The invariant we will keep
An invariant is a property that must remain true in every state our system is allowed to reach. A request can fail, a connection can disappear, and the model can propose the wrong action; the invariant must still hold.
For this article, the property is:
The application must never mark an assignment as confirmed unless it has verified completion evidence for that same operation.
“Verified” means a trusted issue-service response establishes that the expected operation committed, and its operation ID, issue ID, and assigned team match our request. A model-generated claim, an acknowledgment that work was queued, or a receipt for another operation does not qualify.
Why this rule? A user may stop investigating after seeing “assigned.” If we allow generated language to set the completion status, we can tell the user their work is finished when nothing happened. We will let the model propose actions, but trusted application code will own the confirmation gate.
This property guarantees neither that every task finishes nor that the chosen team solves the user’s underlying problem. It is a narrower promise: our official completion status never gets ahead of our evidence. We will follow that promise from prediction through execution, then try to break it.
Start with a task we can check
Our issue tracker contains this record:
| Issue | Report | Assigned team |
|---|---|---|
| I-104 | Search keeps timing out when I submit a query. | Unassigned |
Its authoritative ownership directory says:
| Service | Responsible team |
|---|---|
| Search | Team Birch |
| Payments | Team Cedar |
The request is: find the responsible team and assign I-104. For this example, ownership remains fixed during the operation. Success means the issue store confirms I-104 was assigned to Team Birch by our operation. A plausible sentence or an accepted request is insufficient evidence.
Why introduce a language model? Reports arrive in varied language: “search hangs,” “results never load,” or “queries keep timing out.” An LLM can help interpret those descriptions and propose a service or a follow-up question. Whether it does so reliably is something to evaluate.
Once we have a verified service identifier, looking up its owner is ordinary program logic. We do not need a model to replace a reliable directory lookup. The useful design question is where flexible interpretation helps and where an explicit rule already solves the problem.
Text becomes tokens, then vectors
A text-generating LLM does not receive words as human concepts. Its tokenizer converts text into a sequence of integer IDs. Tokens may correspond to whole words, word fragments, punctuation, or byte-level pieces. The boundaries depend on the tokenizer; a word is not necessarily one token. Hugging Face’s tokenizer guide shows concrete examples.
An embedding matrix maps each token ID to a learned vector—a list of numbers. Position information also enters the computation, so the sequence’s order matters.
Follow the highlight through the six stages.
%%{init: {"flowchart": {"curve": "basis", "rankSpacing": 26, "padding": 16}, "themeVariables": {"fontFamily": "Arial, Helvetica, sans-serif", "fontSize": "16px"}}}%%
flowchart TB
accTitle: From text to the next token
accDescr: Request and context are tokenized into IDs and mapped to embeddings with position information. Transformer layers produce a representation, an output projection scores vocabulary tokens, softmax produces probabilities, and decoding selects a token. Append the selected token and continue until a stopping condition.
context("<b>01 · Request and context</b><br/>Input + generated prefix")
vectors("<b>02 · Token IDs → vectors</b><br/>Embeddings + positions")
layers("<b>03 · Transformer layers</b><br/>Attention + feed-forward")
scores("<b>04 · Next-token scores</b><br/>Output projection → logits")
probabilities("<b>05 · Token probabilities</b><br/>Softmax normalizes scores")
selection(["<b>06 · Select the next token</b><br/>Greedy selection or sampling"])
context --> vectors --> layers --> scores --> probabilities --> selection
classDef input fill:#101016,stroke:#bba3f7,color:#eceaf6,stroke-width:1.5px;
classDef model fill:#69449b,stroke:#bba3f7,color:#faf8fe,stroke-width:1.5px;
classDef output fill:#bba3f7,stroke:#bba3f7,color:#101016,stroke-width:1.5px;
class context,vectors input;
class layers,scores model;
class probabilities,selection output;
linkStyle default stroke:#bba3f7,stroke-width:2px;
The returning dot represents appending the selected token to the generated prefix. Continue until a stopping condition. Weights stay fixed during ordinary inference; animation timing is illustrative.
A token ID is an index, not a measure of meaning: token 400 is not “twice as meaningful” as token 200. Likewise, a vector’s individual coordinates generally do not come with tidy labels such as “search” or “ownership.” Meaningful behavior emerges from learned transformations across many dimensions.
There is already an implementation contract here: the tokenizer’s ID mapping must match the model’s expected vocabulary. A different mapping can supply the wrong embeddings even when every array has the expected shape.
Weights stay fixed; context changes
The model’s learned parameters, often collectively called its weights, shape the transformations applied to those vectors. During ordinary inference, those parameters stay fixed. The inputs and intermediate values change.
| Mechanism | What changes? |
|---|---|
| Add a directory entry to the prompt | The available context |
| Append a tool result | The context for the next model call |
| Fine-tune a pretrained model | Trainable parameters, which may include only adapters |
| Distill a teacher into a student | The student’s trainable parameters, using teacher-provided learning signals |
Fine-tuning and distillation can overlap: a pretrained student can be fine-tuned on examples produced by a teacher. Distillation is not simply copying the teacher’s weights. See the original distillation paper and LoRA paper for two distinct training techniques.
Supplying Search → Team Birch lets the model use a new fact without retraining. If the next request omits that fact, we cannot assume the model retained it. An application can preserve information by storing it and including or retrieving it later; that is different from a weight update. Hugging Face’s model overview distinguishes pretrained models from subsequent training.
Pivot: change only the directory entry to Search → Team Atlas. We want the proposed owner to change accordingly. That is desired behavior to test, not an invariant guaranteed by putting the directory into the prompt.
Attention mixes information from context
For a typical causal transformer, attention lets a position combine information from permitted positions in the sequence. Learned projections form queries, keys, and values. In one attention head, the simplified calculation is:
\[\begin{aligned} A &= \operatorname{softmax}\left(\frac{QK^T}{\sqrt{d_k}}+M\right) \\ H &= AV \end{aligned}\]The query–key dot products produce compatibility scores; scaling controls their magnitude. The mask $M$ excludes forbidden positions. Softmax gives mixing coefficients $A$, which weight the value vectors to produce the head output $H$. In causal self-attention, the mask prevents a position from reading future positions. This mechanism comes from the Transformer paper.
Attention coefficients depend on the input. They are different from the learned projection weights, which stay fixed during ordinary inference. Attention also is not the whole network: feed-forward layers, normalization, and residual connections participate in producing the final representation. Dive into Deep Learning develops the scoring and masking calculation.
For our example, the model can combine the issue description with supplied ownership information. That does not turn attention into a verified database lookup. Even a large attention coefficient is not evidence that the model interpreted a fact correctly.
From a representation to the next token
Let $h_t$ be the final hidden vector at the position used to predict the next token. An output projection produces a score vector:
\[z=W_{\text{out}}h_t\]Here $W_{\text{out}}$ is a learned matrix. This simplified expression omits a possible bias term. Each component $z_i$, called a logit, scores one candidate token. The model’s output softmax is separate from the softmax inside attention: this one distributes probability across the vocabulary.
\[p_i=\frac{e^{z_i}}{\sum_j e^{z_j}}\]The probabilities are nonnegative and sum to one, up to numerical precision. They describe next-token predictions conditioned on the available context and generated prefix. A complete sequence is modeled through successive conditional predictions, as explained in Dive into Deep Learning’s language-model chapter.
Take an invented vocabulary containing exactly three tokens, A, B, and C. To see where scores can come from, suppose the hidden vector is (1, 2) and the output matrix has rows (0, 1), (1, 0), and (2, −1). The three dot products are:
1
2
3
A: 0 × 1 + 1 × 2 = 2
B: 1 × 1 + 0 × 2 = 1
C: 2 × 1 + −1 × 2 = 0
Those matrix entries and the hidden vector are invented, not learned in this example. Applying softmax gives:
| Token | Score | Exponentiated score | Probability |
|---|---|---|---|
| A | 2 | 7.389 | 66.52% |
| B | 1 | 2.718 | 24.47% |
| C | 0 | 1.000 | 9.00% |
For A, we divide 7.389 by approximately 11.107. Rounded percentages need not sum to exactly 100%. This is arithmetic over three invented tokens, not a model’s measured confidence about Team Birch.
At scores 2, 1, 0 and temperature 1, A is most probable. This does not establish whether A is correct.
The calculator subtracts the largest scaled score before exponentiating. This preserves the distribution while avoiding unnecessarily large exponentials. You can also download and run the Python version; it requires only Python’s standard library.
Predict before adjusting: if we add 5 to every logit, will the probabilities change? No. The common factor cancels from numerator and denominator. The gaps between scores matter.
A decoder then selects a token. Greedy decoding selects the largest probability; sampling draws from a distribution. Temperature $T>0$ rescales logits as $z_i/T$: a lower value concentrates probability on higher scores, while a higher value flattens it. Real decoding systems can apply additional constraints and filters. These are selection mechanisms, not truth checks. Hugging Face’s generation guide explains the alternatives.
The token’s probability is not the probability that the completed answer is true. Nor does selecting the most likely token at every step guarantee the best complete answer. A generation can be fluent, structurally valid, and wrong about the owner.
This is the first test of our invariant: even a very high probability for the tokens in “assigned successfully” supplies no completion evidence. The probability calculation cannot open the confirmation gate.
After selection, the chosen token joins the generated prefix and the process continues until a stopping condition, such as an end token or an output limit. A tool request can span many generated tokens; it is not a single indivisible prediction.
A proposed tool call becomes a program operation
An LLM is a model. An LLM application is the surrounding software. Here, we use agent for an application in which the model helps choose successive actions based on available observations, rather than following only a fixed sequence. Definitions vary; Anthropic’s distinction between workflows and agents is useful for this article.
The application supplies tool descriptions and receives a proposed call through the model interface. The exact wire format varies. Generating a call does not itself execute the function.
Our illustrative sequence is:
| Step | Model or program behavior | Evidence available afterward |
|---|---|---|
| Interpret | Model proposes that I-104 concerns Search | A candidate interpretation |
| Look up | Program executes an allowed ownership lookup | Directory result: Team Birch |
| Propose | Model requests assignment to Team Birch | A proposed mutation |
| Check | Executor checks scope, permissions, arguments, and owner | Permission to attempt this mutation |
| Execute | Issue service attempts the write | Completion, rejection, pending status, or an uncertain outcome |
| Report | Application reports what the evidence supports | A user-visible outcome |
If the program appends the lookup result to the next model input, the model can use it as context. Its weights need not change. This is the connection between the model mathematics and the agent loop.
The interpretation still needs scrutiny. “Search fails after payment” might be a Payments issue, a Search issue, or insufficient information. Checking that Team Birch exists does not establish that Search was the correct diagnosis.
Put the invariant at the boundary that can enforce it
The model’s response crosses into trusted application code. This is where we enforce the completion invariant, using the expected operation recorded by the application and a receipt from its trusted service adapter:
1
2
3
4
5
6
7
8
def may_report_confirmed(expected, receipt):
return (
receipt is not None
and receipt.state == "committed"
and receipt.operation_id == expected.operation_id
and receipt.issue_id == expected.issue_id
and receipt.team_id == expected.team_id
)
These fields must come from a verified service response, not JSON invented by the model. The predicate checks the relationship between evidence and request; it does not authenticate the response by itself. The full runnable teaching example includes the record definitions and checks for missing, queued, rejected, and mismatched receipts. It uses constructed receipts, not a real issue service.
The status renderer emits its fixed “assignment confirmed” message only when this gate passes. On other paths it reports the established state—pending, rejected, or unknown. Merely appending a success badge beside contradictory free-form model prose would not make the whole user-facing report trustworthy; the model must not independently author the authoritative completion message.
| Property detail | Our completion invariant |
|---|---|
| Enforcement | The only path to the official confirmed status goes through the evidence check. |
| Assumption | The adapter authenticates and validates service responses; the service’s committed receipt means what its contract says. |
| Observable violation | The application reports confirmed with no matching committed-operation receipt. |
| What it does not establish | Correct interpretation of the user’s issue, eventual completion, or the assignment remaining unchanged forever. |
We can reason about preservation: initially the operation is unconfirmed; model output cannot change that status; sending a request cannot change it; only matching verified completion evidence can. A timeout without such evidence leaves it unconfirmed. A late timeout also must not erase evidence we already verified. Tests exercise these boundaries, while the restricted transition path explains why the rule holds within our stated assumptions.
Authorization is a separate invariant:
Every assignment performed through this executor must pass its authorization check before the write.
The executor must enforce this before writing, using an authenticated identity from trusted application state. This assumes every write in scope passes through that check and that permission checks and writes have appropriate consistency. An instruction saying “only make authorized assignments” expresses a policy; it does not implement the gate. An authorized action can still target the wrong team, and an authorized request can still fail to complete.
Structural and semantic checks have separate jobs. A JSON parser can accept {"team":"Team Cedar"}. A schema can confirm that team is a string. A directory check can confirm that Team Cedar exists. All three can pass while the proposed owner is wrong for Search.
This is why we should name the property each check establishes. “Validated” without a named property hides the remaining uncertainty.
A failed response does not always mean a failed action
Now change one condition: the connection drops after the issue store commits the assignment but before the application receives confirmation.
INVARIANT / AI SYSTEMS
The write succeeded. The response was lost.
Follow one assignment across three boundaries.
assign_issue(…)
Arguments + permission
I-104 → Team Birch
Application outcomeUNKNOWN
Database stateCHANGED
No confirmation without evidence.A lost response does not undo a committed write.
The database changed. The application sees a timeout. Reporting “nothing happened” would be unjustified; reporting confirmed success would also exceed the evidence it currently has.
The invariant is preserved by keeping the outcome unknown. That is not the task’s desired outcome, but it is an honest state. Reporting confirmed without the missing evidence would violate the invariant even if we later discovered that the write had succeeded.
| Observation | What the application can conclude |
|---|---|
| No assignment tool ran | This interaction made no assignment through that tool. |
| Explicit rejection before a write | This attempt did not perform the assignment. |
| Request accepted for processing | Work is pending; completion is not established. |
| Completion tied to the requested operation | The assignment is confirmed under the service’s response contract. |
| Timeout after sending | The outcome is unknown until reconciled. |
Blind retries can duplicate effects. Even if assigning the same team twice leaves the same final field value, notifications or audit events might repeat. APIs can use an operation identifier with server-side deduplication, and clients can reconcile against operation status. The guarantee depends on how the service implements and retains that identity. AWS’s discussion of idempotent APIs explains why ambiguous outcomes require this care.
Reading the current team is useful but may not prove our operation caused that state; another writer could have assigned it. The evidence we require should match the claim we intend to make.
This is the confirmation gate introduced above: code emits the fixed confirmation only for a matching, verified committed receipt. Free-form model narration cannot decide whether an operation committed.
| Change one condition | May we report confirmed? | Why? |
|---|---|---|
| The model says “done”; no tool ran | No | Generated text supplies no receipt. |
| A matching response says “accepted” | No | Acceptance is not a commit. |
| A committed receipt names another operation | No | Evidence must belong to this operation. |
| The write commits, but its response is lost | Not yet | The application does not have the evidence. |
| Reconciliation supplies a matching committed receipt | Yes | The evidence now supports confirmation. |
Notice that the rule stays the same across all five cases. The model’s wording and the network behavior change; the requirement for matching completion evidence does not.
What “going wrong” actually means
Often, nothing is wrong with the matrix multiplication. The model computes a valid distribution, but its generated interpretation or action does not satisfy the task.
| Boundary | Possible failure | Check or response |
|---|---|---|
| Context | Ownership data is absent or stale | Retrieve authoritative information; track its revision. |
| Interpretation | The affected service is ambiguous | Seek evidence or ask for clarification. |
| Generated arguments | A team identifier is fabricated | Validate identifiers and the required relationship. |
| Execution | An unauthorized write is proposed | Reject it at the executor. |
| Confirmation | The connection drops after a commit | Preserve uncertainty and reconcile. |
| Reporting | The model says “done” without evidence | Derive completion status from verified execution state. |
These controls do not prove the entire system correct. They give specific properties enforceable boundaries, and give experiments specific failures to measure.
There are also two outcomes worth separating: task completion and accurate reporting. An agent can accurately report a permission failure without completing the requested assignment. It can also complete an assignment and describe the result incorrectly. Evaluation should notice both.
Try the boundaries yourself
Consider these changes to the same example before opening the answers:
The owner changes in the prompt. Must the model's weights change?
No. The model can use the supplied fact during inference with fixed weights. Correct use of the new fact remains behavior to test.
The output is valid JSON and Team Cedar exists. Is the assignment correct?
Not for Search under our directory. Syntax and existence checks do not establish the owner relationship. The service interpretation itself may also need verification.
The tool times out. Is it safe to assume nothing happened and repeat the call?
No. The write may have committed. Use the service’s reconciliation and deduplication contract rather than assuming the absence of a response means the absence of an effect.
We started with “Assigned to Team Birch.” We can now ask what supports each part of that sentence: the interpretation of the issue, the ownership lookup, the authorized write, and the completion evidence. Our central invariant belongs to that last boundary: no official confirmation without matching verified completion evidence.
The next investigation will take a smaller boundary apart: the output parsed—did it answer correctly? For this first step, the distinction to keep is that a prediction proposes what might come next; the program must establish what it permits and what actually happened.