Observability for Generative AI Products Beyond Token Counts

A generative AI product can be available, fast, and wrong.

Traditional service telemetry still matters. Requests fail, dependencies slow down, queues saturate, and deployments introduce regressions. But a healthy API does not prove that retrieval found useful evidence, an agent selected the right tool, or the final response helped the user complete a task.

The observability model has to connect both kinds of truth: what the system did and whether the product outcome remained acceptable.

Begin with decisions, not dashboards

The first question is not “What can we log?” It is “What decisions must this telemetry support?”

A useful system should help a team:

• detect user-facing degradation;
• identify which stage caused it;
• compare a change against a stable baseline;
• control cost and capacity;
• investigate safety or privacy incidents;
• reproduce a representative failure; and
• decide whether to continue, roll back, or limit a release.

Every signal should serve at least one operational decision. This keeps the telemetry model smaller, more explainable, and less likely to collect sensitive content without a clear reason.

It also separates observability from analytics. Product analytics may explain adoption and behavior over weeks. Operational observability should make a current failure diagnosable while there is still time to act.

Keep the service-level foundation

Generative behavior does not replace ordinary reliability engineering.

Start with traffic, errors, latency, and saturation across the whole request path. The Google SRE guidance on monitoring distributed systems frames these as the four golden signals and recommends simple, high-signal alerting tied to real service failures.

For an AI product, the familiar foundation should include:

• request volume and concurrency;
• end-to-end latency distributions;
• time to first useful output for streamed experiences;
• timeout, cancellation, and retry rates;
• dependency availability;
• queue depth and worker saturation; and
• fallback or degraded-mode activation.

Use distributions rather than averages for latency. A healthy median can hide a long tail caused by a specific model, tool, region, or retrieval path.

These signals answer whether the service is operating. They do not answer whether it is operating well.

Trace the product journey, not only the model call

An AI request is usually a distributed workflow.

It may include authentication, policy checks, retrieval, reranking, prompt assembly, one or more model calls, tool execution, validation, persistence, and response streaming. Recording only the model request creates a detailed view of one dependency and a blind spot around the product.

Model the trace around meaningful operations:

1. Receive the user task.
2. Resolve configuration and product policy.
3. Retrieve or assemble context.
4. Invoke a model.
5. Execute tools or downstream actions.
6. Validate or transform the result.
7. Deliver the user-visible response.

Each span should have a clear beginning, end, owner, and failure meaning. Avoid tracing small in-process functions simply because instrumentation is easy.

The OpenTelemetry semantic conventions provide common names for spans, metrics, events, and attributes so telemetry remains comparable across services and languages. Their generative AI work includes operation identity, model information, token usage, and response properties. Use those conventions where they fit, but keep an explicit version because the GenAI conventions are still evolving.

Make every result attributable

Debugging a probabilistic system requires knowing which system produced the result.

Attach stable, bounded identifiers for:

• application and release version;
• workflow or agent version;
• model and provider configuration;
• prompt or instruction version;
• retrieval corpus and index version;
• tool and schema version;
• policy and guardrail version;
• experiment or feature-flag cohort; and
• evaluator version when a score is attached.

These identifiers should describe reviewed artifacts, not contain the artifacts themselves. A prompt version is safe to aggregate; a full system prompt may not be.

Attribution makes comparisons possible. Without it, a quality change after a deployment becomes a vague correlation. With it, the team can isolate whether the change followed a prompt revision, model rollout, retrieval refresh, or tool contract update.

Separate metrics from evidence

Metrics should be cheap to aggregate and safe to retain.

Good operational metrics use bounded dimensions:

• requests by operation and status;
• latency by stage and version;
• input and output token distributions;
• tool success, refusal, and timeout rates;
• retrieval result counts and no-result rates;
• validation and policy-intervention rates;
• retry and fallback rates; and
• estimated cost by product capability.

Do not put user identifiers, prompts, URLs with embedded data, or arbitrary tool arguments into metric labels. High-cardinality dimensions make the system expensive and difficult to query; sensitive dimensions create a larger problem.

Traces and structured events carry the evidence behind an individual request. Logs remain useful for discrete application events that do not belong to a timed operation. The three signals should share trace and request correlation identifiers so an aggregate anomaly can lead to a representative execution.

Treat content capture as privileged

Prompts and responses can be extremely useful during investigation. They can also contain personal data, confidential material, credentials, retrieved documents, or tool results.

The OpenTelemetry GenAI attribute guidance explicitly warns that input and output messages are likely to contain sensitive information and recommends opt-in collection with filtering or truncation support. That should be the starting posture: content capture is privileged, not default.

A responsible policy defines:

• which environments may capture content;
• which fields are always removed;
• how secrets and personal data are detected;
• sampling conditions;
• encryption and access controls;
• retention limits;
• audit trails for access; and
• deletion and incident procedures.

In many cases, identifiers and structural metadata are enough. Record the prompt version, not the prompt. Record retrieved document identifiers and ranking scores, not the full documents. Record tool names and outcomes, not unrestricted arguments.

When sampled content is necessary, isolate it from general observability access and make the collection decision visible in the trace.

Observe quality as its own layer

Latency and token counts are not quality metrics.

Quality signals should reflect the product contract. Depending on the capability, that may include:

• groundedness in approved evidence;
• correct tool selection and argument construction;
• schema or format validity;
• task completion;
• appropriate refusal;
• citation support;
• adherence to product policy; and
• user correction, abandonment, or escalation.

Some signals are deterministic and can run inline. A schema can be validated before a result is returned. A citation can be checked against the retrieved evidence. A forbidden action can be blocked.

Other qualities require an asynchronous evaluator or human review. Do not force every judgment into the critical path. Sample representative traffic, evaluate it with a versioned rubric, and link the result back to the production trace without exposing the original content broadly.

Online proxies also need calibration. A thumbs-up is useful but incomplete. A user may accept a fluent answer without noticing an unsupported claim, or reject a correct answer because it arrived too late. Compare proxy signals with reviewed outcomes before treating them as release gates.

Connect production telemetry to evaluations

Offline evaluation and production observability should use the same vocabulary.

If the evaluation suite measures groundedness by workflow version and the production system records neither, the two systems cannot explain each other. Align identifiers, outcome categories, and failure taxonomies so a production incident can become a regression test and an evaluation change can be verified after release.

A practical loop looks like this:

1. A production signal identifies a meaningful failure class.
2. A representative example is reviewed and sanitized.
3. The example becomes a durable evaluation case.
4. A proposed change runs against the evaluation suite.
5. The release is limited to a measurable cohort.
6. Production quality and reliability are compared with the baseline.
7. The team promotes, revises, or rolls back the change.

This turns observability into part of the improvement system rather than a passive archive.

Alert on user impact

Not every unusual model response deserves a page.

Alerts should represent conditions that require timely human action: a sustained rise in failed user journeys, severe latency, runaway retries, critical policy failures, or a loss of an essential dependency without a viable fallback.

Keep diagnostic signals available for investigation, but do not page on token variance, evaluator noise, or a single unusual trace. The SRE principle applies here too: alerts should be simple, understandable, and tied to a clear failure.

Quality alerts often need a window and a minimum sample size. A low-volume evaluator can move sharply because of one result. Define uncertainty and data freshness alongside the threshold so the alert does not imply more confidence than the measurement supports.

Prepare for incidents before they happen

The NIST Generative AI Profile emphasizes continuous monitoring, explicit incident ownership, rehearsed response plans, and learning from retrospectives. Those practices become concrete when the telemetry model can answer:

• Which users and capabilities were affected?
• Which versions and dependencies were involved?
• What sensitive information may have been processed?
• Can the failure be reproduced safely?
• Is there a known fallback or kill switch?
• What evidence supports recovery?

Define these queries before an incident. Confirm that the relevant data exists, the access path works, and the team knows who can make the rollback or containment decision.

Build the smallest complete loop

An effective first version does not require collecting everything.

Start with:

1. End-to-end traces across the few stages that dominate latency and failure.
2. Golden-signal metrics plus token and cost distributions.
3. Stable versions for prompts, models, retrieval, tools, and policy.
4. Privacy-safe correlation between production traces and sampled evaluations.
5. One dashboard for service health and one for product-quality trends.
6. A small number of alerts tied to documented response actions.
7. A release comparison that can answer whether the change helped.

Expand the system when a real operational decision lacks evidence.

Observability should make uncertainty manageable

Generative AI products introduce variable outputs, evolving dependencies, and quality failures that ordinary uptime cannot see. The answer is not unlimited logging.

A useful observability system preserves the reliability foundation, traces the complete product journey, attributes every result to versioned artifacts, separates aggregate metrics from sensitive evidence, and closes the loop between production behavior and evaluation.

That does not remove uncertainty. It makes uncertainty visible enough for a team to operate the product responsibly.