If You Cannot Reproduce Yesterday's Answer, You Do Not Have A Product

A user reports that your AI feature said something wrong on Tuesday.

Can you see what it actually said? What went in? What documents it was looking at? Which model answered? If any of those is a shrug, you are not debugging — you are guessing, and the fix you ship is a guess too.

Conventional application logging does not cover this, because the interesting failures are not exceptions. Nothing threw. The system returned a confident, well-formed, wrong answer with a 200 status code.

The field everyone forgets

Log the exact model version string the provider returned, on every single call.

Not the alias you requested. Providers serve aliases that point at different underlying builds over time, and behaviour changes underneath a stable name. When quality shifts one Tuesday with no deploy on your side, the model version in your logs is the only artefact that distinguishes "a provider changed something" from "our retrieval broke" — and without it that investigation has no floor.

Record what you asked for and what you got. When they differ, you have learned something.

What else belongs in the record

The full rendered prompt, not the template. Templates hide the bug. You want the string that was actually sent, after every variable, every truncation, and every piece of injected context.

The retrieved context, with document identifiers and versions. Retrieval failures dominate real-world RAG problems, and the usual cause is boring: the model answered perfectly from the wrong chunk. You cannot see that without knowing which chunks it had.

All parameters — temperature, max tokens, top-p, seed if you use one, and any tool or function definitions in scope. These get edited during incidents and forgotten.

Every tool call and its result. For anything agentic this is the actual story: what it decided to do, with what arguments, and what came back. A summary of the final answer tells you nothing about a five-step chain that went sideways at step two.

The complete output, including any refusal or safety intervention, plus the finish reason. "Stopped because it hit the token limit" and "stopped because it was done" look identical downstream and mean entirely different things.

Latency broken down — retrieval, model, tools separately. Aggregate latency tells you something is slow. Component latency tells you what.

Token counts and cost per call. Cost regressions are silent. A prompt change that adds context to every request will not fail any test; it will just quietly multiply your bill.

A trace identifier threaded through the whole chain, so a user complaint maps to a specific sequence of calls without archaeology.

Privacy is a design constraint, not an afterthought

You are now storing user input verbatim, which raises real obligations.

Decide retention deliberately and keep it short unless something justifies longer. Redact or tokenise sensitive fields on the way in rather than promising to clean the store later. Restrict who can read prompt logs, because they contain whatever your users typed. And write down what your provider does with the same data — retention, training, sub-processors — because your users' expectations and your vendor's default settings are not automatically aligned.

The uncomfortable framing that gets this prioritised: a prompt log is a database of things your users said when they thought they were talking to a machine.

Sampling, when volume makes full capture impractical

Log everything while volume is low; the data is more valuable than the storage costs.

At scale, keep full fidelity on anything anomalous — errors, refusals, unusually long or short outputs, high latency, low confidence, thumbs-down feedback — and sample the ordinary traffic. The mistake is uniform sampling, which throws away exactly the rare events you needed.

Recording is not alerting

Logs answer questions after the fact. They do not tell you something is wrong, and for AI features the failures are quiet enough that nobody notices for weeks.

A small number of alerts covers most of it. Alert on refusal rate moving, because a jump usually means a prompt or policy change did something you did not intend. Alert on output length distribution shifting, which catches truncation and runaway generation before users report either. Alert on cost per request, which is the fastest signal that a prompt change added context everywhere. And alert on the model version string changing when you did not deploy — that one should page someone.

Notice what is missing from that list: accuracy. You cannot alert on correctness in production because you do not have ground truth there. That is what the evaluation set is for, run on a schedule against known cases. Monitoring tells you the shape of traffic changed; evals tell you whether the answers got worse.

The check that tells you whether it works

Pick a real interaction from last week. Try to reconstruct precisely what happened: the input, the context, the model, the parameters, the tools, the output, the cost.

If you can do that in a couple of minutes, your instrumentation is adequate. If you find yourself inferring, you have a gap — and you will discover exactly where it is at the least convenient moment, which is what everyone before you also discovered.