Software engineers get used to a fairly comforting model of the world: give the same system the same input, run the same code, and you should get the same output. There are plenty of exceptions, but the idea is reliable enough that it shapes how we debug. When behaviour changes, we go looking for the thing that changed.
That instinct works well for ordinary software. A deployment might have introduced a bug, a configuration value might have shifted, a dependency might be behaving differently, or some piece of state might no longer be what we expected. There is usually a chain of cause and effect that we can reconstruct.
Then you start building products around large language models and the ground moves slightly. The same apparent input can produce a different output even when nothing is obviously broken, and that changes the kind of evidence we need before we can say with confidence that a system has regressed.
“It used to work”
A familiar production conversation goes something like this:
“This gave a better answer last week.”
The natural response is to start searching for a cause. Did the application change? Was there a prompt update? Did retrieval return something different? Did the model or deployment change underneath us?
Those are sensible questions, but there is a more basic problem that often appears first: sometimes we cannot even establish the thing we are trying to explain. The previous result may have come from one demo, the exact prompt may never have been recorded, or the workflow may simply produce a range of acceptable outputs from one run to the next.
In that situation, “I remember this being better before” is not evidence of a regression. It is an observation. That distinction matters because once we treat an observation as a proven change in system behaviour, the investigation quickly turns into a collection of plausible stories.
A remembered output is not a baseline
Imagine a workflow scores its responses from 1 to 5. You run it once and get:
4
Then you run it again and get:
3
That second result might look worse, but on its own it tells us very little. If we run the same workflow ten times and see:
4 3 4 4 3 4 3 4 4 3
the score of 3 looks much less alarming. It may simply be part of the system's normal behaviour.
This is one of the first uncomfortable lessons when moving from deterministic software into AI products: variation is part of the thing you are engineering. A single output is therefore a poor baseline. In many cases, the useful baseline is a distribution of behaviour across repeated, controlled runs.
The alien planet
Traditional application debugging often resembles something like this:
input
↓
deterministic application logic
↓
output
An AI product usually has more hidden state between the request and the result:
input
↓
application state
↓
retrieval
↓
prompt construction
↓
model configuration
↓
model inference
↓
guardrails
↓
post-processing
↓
output
Each stage can affect what eventually reaches the user, which means even the phrase “same input” becomes ambiguous. We may have sent the same user text while changing the rendered prompt, the retrieved documents, the model deployment, the source data or the inference parameters.
If we cannot reconstruct those details, debugging becomes an exercise in speculation. We can still come up with explanations, but we cannot tell which ones are actually supported by evidence.
Make the invisible visible
The first requirement is not necessarily a more sophisticated evaluation framework. It is traceability. For an interesting execution, we should be able to reconstruct enough of the run to understand what happened.
That might include the application version, environment, workflow version, prompt template and rendered prompt, retrieved documents and scores, model and deployment identifier, model parameters, source-data versions, guardrail results, post-processing, latency, token usage and final output.
Once that information exists, the question changes. Instead of asking:
“Why did it answer differently?”
we can ask:
“What differed between these two runs?”
The first question invites theories. The second gives us something concrete to compare.
Observability tells you what happened
Suppose two outputs differ. A trace might show that the application commit, prompt template and model name are identical, but retrieval returned different context. That gives us somewhere specific to investigate.
In another case, retrieval and the final prompt might be identical while the model deployment identifier has changed. Or perhaps everything we can observe is the same, which is useful in its own way because it allows us to rule out several hypotheses and take inherent model variability more seriously.
This is what good observability buys us: it turns an opaque model invocation into an execution we can inspect. But there is still a second question that observability cannot answer on its own.
Did the product actually get worse?
A trace can explain one execution. It cannot tell us whether the behaviour of the product has materially changed.
Evaluation is the behavioural test layer
Suppose version A produces these scores over ten controlled runs:
3 4 4 3 4 4 3 4 3 4
and version B produces:
3 3 4 3 3 4 3 3 3 4
Now we have something more meaningful to investigate. Not because one run returned a 3, but because the overall behaviour may have shifted.
For deterministic code, we are comfortable with assertions such as:
given X
expect Y
Generative systems often need something broader:
given this workload
is the distribution of behaviour still acceptable?
What counts as acceptable depends on the product. A support assistant might care about correctness, groundedness and policy compliance. A marketing workflow might care about tone, readability, brand requirements and factual accuracy. A retrieval system might care about recall, ranking and answerability.
Some of those dimensions can be tested deterministically. Others need heuristics, model-based evaluation or human judgement. The important shift is that quality becomes something we define and measure deliberately rather than something we infer from whichever response happened to appear in the latest demo.
Repeated runs matter
One of the easiest mistakes is to compare two versions using one execution each:
Version A → score 4
Version B → score 3
On that evidence, version A appears better. But run both versions ten times:
Version A
4 3 4 3 4 3 4 3 4 3
Version B
3 4 4 4 3 4 4 3 4 4
and the story changes. For probabilistic systems, the unit of comparison may need to be a sample rather than an individual response.
That does not mean every test requires hundreds of executions. It means we need some understanding of the system's ordinary variation before we interpret a different result as a regression. Without that, we risk treating noise as signal.
Prototype evidence does not survive production expectations
This problem becomes especially obvious when a successful prototype starts turning into a real product. During prototyping, a perfectly reasonable test might be: “Try it a few times. Does this seem useful?”
That is often enough to answer the prototype question: can this idea work at all? But once people start relying on the system, the questions become much harder. Has quality changed? Is this model actually better? Did yesterday's deployment introduce a regression? Is the new prompt an improvement, or did we just happen to get a good result?
Those are production questions, and they require production-grade evidence. The mistake is not prototyping quickly. The mistake is carrying prototype-grade evidence into production-grade expectations.
Capture the variables that can explain behaviour
AI infrastructure adds another wrinkle because a familiar label can hide important changes underneath it. A request may still say:
GPT-4.1 mini
while the deployment, provider configuration or serving infrastructure has changed.
That does not mean every behavioural difference should be blamed on the provider. It simply means that model identity should be treated as observable system state rather than an assumption. The same principle applies to prompts, retrieval, source data, configuration, code and evaluation logic.
If a variable could plausibly explain behaviour, capture enough information to investigate it later.
Measure what matters to the product
There is no universal evaluation suite, and trying to measure everything usually creates more noise than insight. The useful metrics come from the behaviour the product is supposed to deliver.
For one system, groundedness and policy compliance might matter most. For another, tone and readability may be central. A retrieval-heavy product may care far more about recall and ranking than stylistic quality.
The goal is not to accumulate metrics. It is to make the behaviours we care about measurable enough that changes become evidence rather than anecdotes.
The debugging question changes
In deterministic software, we often begin with:
What broke?
For AI products, a better starting point is:
What changed, and what evidence do we have that behaviour moved outside its expected range?
That framing keeps several possibilities open. A deterministic part of the system may have changed, an external dependency may be different, the data or context may have shifted, or the model's behaviour may genuinely have moved. It is also possible that nothing meaningful changed and we simply observed normal variance.
There is one more possibility that is easy to overlook: the original baseline may never have been reliable enough to support the comparison in the first place. That is uncomfortable, but it is common.
A trace and an eval answer different questions
The distinction is simple:
Observability asks: what happened during this run?
Evaluation asks: is the system behaving acceptably across runs?
We need both. Without observability, a regression is difficult to investigate. Without evaluation, it is difficult to establish that a regression exists at all.
Together, they turn “it seems worse” into something much more useful: quality on a particular workflow has moved outside its normal range, and the affected runs also show a change in retrieved context. At that point we have stopped arguing about impressions and started dealing with an engineering problem.
Software engineering still works here
The lesson is not that traditional software engineering stops being useful once models enter the system. Quite the opposite. We still need versioning, tests, controlled environments, observability, reproducibility, monitoring, release discipline and careful incident investigation.
What changes is the object we are engineering. Correctness alone is no longer enough; we also need to reason about variability, quality and evidence.
The question moves from:
“Does this code produce the expected output?”
toward:
“Is this system's behaviour measurable, explainable and within acceptable bounds?”
For software engineers entering AI product development, that can feel like landing on a slightly alien planet. Most of the engineering instincts still work, but the ground behaves differently.
When the system becomes probabilistic, engineering discipline matters more, not less.