PostHog AI observability tools: A 2026 guide

# PostHog AI observability tools: A practical 2026 guide AI applications are moving from simple prompt experiments to production systems that combine language models, retrieval, tools, agents, databases, and user-facing workflows. That change creates a monitoring problem. A conventional application metric such as request latency or HTTP error rate is useful, but it does not explain whether an AI response was grounded, whether a tool call returned the right data, whether a prompt change damaged quality, or whether rising usage made the application uneconomical. PostHog is often considered a product analytics platform, but its event-based model can also support AI observability. With the right event design, teams can connect model calls to users, sessions, product outcomes, experiments, and costs. The result is not merely a log of model requests. It is a way to understand how AI behavior affects the product. This guide explains what PostHog AI observability tools can mean in practice, how to design an instrumentation strategy, which metrics matter, how to investigate failures, and where adjacent AI tools fit into the workflow. It also explains the limitations of using a product analytics platform as the only observability system. Features, integrations, and pricing can change, so verify current details in the official PostHog documentation and product pages before implementation. ## What is AI observability? AI observability is the practice of collecting and interpreting the signals needed to understand an AI system in production. Those signals usually cover four dimensions: | Dimension | Questions it should answer | Example signals | |---|---|---| | Reliability | Is the system available and technically healthy? | Error rate, timeout rate, retry count, provider failures | | Performance | Is the experience fast enough? | Time to first token, total latency, queue time, tool duration | | Quality | Are outputs useful, safe, and correct for the task? | User feedback, evaluator score, citation checks, refusal patterns | | Economics | Is the system financially sustainable? | Input tokens, output tokens, model cost, cache use, cost per outcome | For AI products, observability must also follow the full execution path. A single user request may invoke a router, retrieve documents, call a model, execute one or more tools, ask the model to revise its answer, and then stream a response to the user. Monitoring only the final HTTP request hides the parts most likely to fail. A useful trace therefore records a parent operation and its child operations. The parent might be an assistant conversation. Children might include retrieval, prompt construction, model generation, moderation, and tool execution. PostHog can represent those operations as structured events and properties. The precise implementation depends on your application architecture and the PostHog SDK or ingestion method you choose. ## Where PostHog fits in an AI observability stack PostHog is strongest when technical AI signals need to be connected to product behavior. It can help answer questions such as: - Which model version is associated with successful task completion? - Do users abandon conversations after slow first responses? - Which prompt variant produces more useful feedback? - Are expensive tool calls concentrated in a particular workflow? - Which customer segment encounters retrieval failures? - Did a release improve quality while increasing latency or cost? It is less suitable as the only place for high-volume raw prompt and completion storage, deep distributed tracing, infrastructure profiling, or incident management. A mature stack may send low-level logs and traces to a specialized observability backend while sending carefully selected product events and evaluation results to PostHog. ### PostHog compared with adjacent AI tools The directory tools available in this environment are not replacements for PostHog observability. They can support documentation, prototyping, content production, or application development around an AI system. The comparison below keeps those roles separate. | Tool | Best role in an AI workflow | Observability role | Pricing tier | |---|---|---|---| | PostHog | Product analytics, feature flags, experiments, and event analysis | Primary product-facing analysis layer when instrumented correctly | Check the official site for current pricing | | [Notion](/en/tools/notion-ai) | Documentation, runbooks, evaluation notes, and prompt governance | Stores human-readable operating knowledge, not runtime telemetry | Freemium; check the official site for current pricing | | [Gemini](/en/tools/gemini) | General-purpose research, drafting, and analysis | Can assist with reviewing sampled traces, but is not a telemetry store | Freemium; check the official site for current pricing | | [Replicate](/en/tools/replicate) | Running and integrating model workloads | Can be instrumented as a model provider or inference step | Freemium; check the official site for current pricing | | [Loom](/en/tools/loom) | Recording product walkthroughs and incident explanations | Useful for qualitative team communication, not automated monitoring | Freemium; check the official site for current pricing | | [Gamma](/en/tools/gamma) | Creating internal presentations about system findings | Helps communicate trends after analysis | Freemium; check the official site for current pricing | The practical lesson is simple: use PostHog for structured behavioral evidence, use documentation tools for policies and decisions, and use model or application platforms for execution. Avoid calling a content or development tool an observability platform merely because it can process AI-generated text. ## The data model: events, traces, and properties A reliable implementation begins with a stable event taxonomy. Names should describe meaningful operations rather than implementation details that change every sprint. For example, `ai_generation_completed` is generally more durable than `openai_sdk_wrapper_v3_finished`. A useful minimum model contains these event families: | Event family | Purpose | Recommended properties | |---|---|---| | `ai_request_started` | Marks the beginning of a user-visible AI operation | Trace ID, session ID, feature, model, prompt version, timestamp | | `ai_retrieval_completed` | Records search or retrieval behavior | Query type, document count, latency, retrieval version, empty-result flag | | `ai_tool_call_completed` | Records an external action or function call | Tool name, success flag, duration, validation result, retry count | | `ai_generation_completed` | Records model output and performance | Model, provider, latency, tokens, finish reason, safety outcome | | `ai_feedback_submitted` | Captures explicit user judgment | Rating, feedback category, trace ID, surface, user segment | | `ai_outcome_recorded` | Links AI behavior to a business or product result | Outcome name, success flag, time to outcome, feature, experiment variant | Use a common `trace_id` to connect all children of one user request. Use a `span_id` or operation ID when you need to distinguish individual model, retrieval, or tool steps. A `session_id` connects related turns, while a user or account identifier supports cohort analysis. Keep these identifiers consistent across browser, server, worker, and queue boundaries. ### Properties that deserve special attention #### Model and version Record the provider, model family, model version where available, routing decision, and fallback state. A label such as `production` is not enough to reproduce an incident. Model aliases can point to changing implementations, so store the identifier returned by the provider when practical. #### Prompt and retrieval versions Do not store only the prompt text. Store a prompt template version, system-policy version, retrieval configuration version, and parser version. Version identifiers make comparison possible without exposing sensitive text in every event. #### Timing Capture start time and end time for each span. For streaming applications, separate time to first token from total completion time. For retrieval, record search latency separately from reranking and context assembly. #### Usage and cost Store input and output token counts when available. If provider pricing is not returned by the API, calculate a cost estimate using a versioned pricing table maintained in your application. Mark estimates clearly and avoid presenting them as provider-invoiced totals. #### Outcome Technical completion is not the same as success. A model may return HTTP 200 and still fail the user’s task. Record a downstream outcome such as export completed, answer accepted, workflow submitted, or human correction required. ## Privacy and data governance AI telemetry often contains user prompts, documents, names, identifiers, and model outputs. Treat observability data as production data. Before sending it to PostHog, define what is necessary for diagnosis and what should remain in a controlled store. A practical policy usually includes the following: 1. Send metadata by default: IDs, versions, durations, statuses, counts, and scores. 2. Send raw prompts or completions only when there is a clear debugging need and an approved retention policy. 3. Redact secrets, access tokens, payment data, health information, and unnecessary personal information before ingestion. 4. Hash or pseudonymize identifiers when analysts do not need the original value. 5. Restrict access to raw traces more tightly than access to aggregate dashboards. 6. Define retention periods for raw content, derived metrics, and evaluation results. 7. Document whether customer data can be used in debugging, evaluation, or model-improvement workflows. Do not assume that masking a user name is enough. Prompt text can reveal identity through context, and retrieved documents may contain confidential information. A redaction layer should run before telemetry leaves the service that handles the data. ## Metrics that matter for AI applications ### Reliability metrics Track request success and failure by provider, model, feature, and version. Include timeouts, rate limits, malformed responses, schema validation failures, tool errors, and retrieval failures. A single success rate can hide a severe failure in a small but important customer workflow, so segment by operation. Useful reliability views include: - Generation completion rate. - Structured-output validation rate. - Tool execution success rate. - Empty retrieval rate. - Fallback frequency. - Retry frequency and retry success. - Safety or policy-block rate. - Percentage of traces with missing child spans. ### Latency metrics AI latency is perceived differently from ordinary API latency. A streamed answer may feel responsive if the first token arrives quickly even when the total completion takes longer. Measure both. Track: - Queue time before processing. - Retrieval duration. - Time to first token. - Time between streamed chunks when relevant. - Total generation duration. - Tool-call duration. - End-to-end user-visible duration. - Tail latency, such as a high percentile, rather than only the average. Break latency down by model, prompt version, context size, tool path, and fallback path. Large contexts may improve answer quality while creating slow or expensive requests; the dashboard should make that tradeoff visible. ### Quality metrics Quality needs a combination of explicit, implicit, and automated signals. No single metric is sufficient. Explicit signals include thumbs-up or thumbs-down, a rating, a correction button, or a reason selected by the user. Implicit signals include whether the user copied an answer, accepted a generated draft, completed the next workflow step, or immediately regenerated the response. These signals are useful but ambiguous. A copy action can mean the response was useful or that the user copied incorrect text for later editing. Automated signals may include citation presence, schema validity, groundedness checks, prohibited-content checks, or a rubric-based evaluator. Treat automated scores as measurements with error, not as ground truth. Store evaluator version and rubric version so scores remain interpretable after the evaluator changes. ### Cost metrics Cost is a product concern, not only an infrastructure concern. Record estimated model cost per request, per successful outcome, per active account, and per feature. Distinguish input and output usage, because a prompt with a large retrieved context can be expensive even when the final answer is short. Useful questions include: - Which feature consumes the most model budget? - Which traces are expensive but do not produce a successful outcome? - Does a cheaper model perform adequately for simple requests? - Are retries caused by parsing failures increasing total cost? - Does a prompt or retrieval change increase context size? Never use cost alone as a quality target. A low-cost system that fails tasks creates support work and damages trust. ## Step-by-step: instrument a basic AI request The following process is provider-neutral. Adapt the event calls to the PostHog SDK used by your backend. Keep telemetry server-side for model and tool operations whenever possible. ### Step 1: Define the trace boundary Choose the user-visible operation that should be investigated as one unit. For a support assistant, this may be one submitted question and answer. Generate a trace ID at the beginning of the request and pass it to every downstream function. ### Step 2: Capture request metadata Record feature name, application version, environment, user or account identifier, session ID, prompt version, and model-routing decision. Avoid recording the full prompt until privacy review is complete. ### Step 3: Create child spans Wrap retrieval, moderation, tool calls, and generation separately. Each span should have a start timestamp, end timestamp, status, and error classification. A child span should be emitted even when it fails, because failure locations are essential for diagnosis. ### Step 4: Record usage and output status After the provider responds, record token counts, finish reason, schema validation, safety result, and fallback state. If the provider returns no usage data, store a null value rather than inventing precision. ### Step 5: Record the product outcome later The user’s result may happen after the model response. Emit a separate outcome event when the user accepts, submits, exports, or corrects the result. Link it with the trace ID. ### Step 6: Test failure paths Force a provider timeout, invalid structured output, empty retrieval result, failed tool call, and user cancellation in a non-production environment. Verify that the events preserve the same trace ID and distinguish each failure. A simplified event sequence looks like this: ```text ai_request_started ai_retrieval_completed ai_tool_call_completed ai_generation_completed ai_feedback_submitted ai_outcome_recorded ``` The sequence is illustrative rather than a required PostHog schema. The important properties are consistent identifiers, clear status fields, and a link between technical execution and product outcome. ## Step-by-step: build a PostHog AI observability dashboard ### Step 1: Start with a small set of questions Create separate views for reliability, performance, quality, and economics. Do not begin with every available property. A dashboard that answers a few operational questions is easier to maintain than a wall of charts. ### Step 2: Add volume and failure trends Chart request volume and failure rate over time. Break down failures by operation, provider, model, application version, and environment. Ensure the time zone and event delay are understood before interpreting a sudden change. ### Step 3: Add latency breakdowns Use event properties or span durations to compare retrieval, tool, generation, and total latency. Add filters for feature and model. Watch for changes that affect only tail latency; average latency can look stable while a subset of users waits much longer. ### Step 4: Add quality signals Display feedback rate, positive and negative feedback, correction rate, evaluator scores, and downstream completion. Segment by prompt version and retrieval version. Keep sample size visible, because a high score from a small number of evaluations should not outweigh a broader signal. ### Step 5: Add cost and outcome views Chart estimated cost per request and cost per successful outcome. Compare model routes and workflow types. A model that costs more per call may still be efficient if it prevents retries or produces more completed tasks. ### Step 6: Add release and experiment annotations Mark deployments, prompt changes, model changes, retrieval changes, and feature-flag changes. Without annotations, analysts may see a metric shift but miss the operational event that caused it. ### Step 7: Define alerts carefully Alert on actionable conditions: sustained provider failures, schema-validation collapse, unusually high tool errors, missing telemetry, or a meaningful latency regression. Avoid alerting on every small movement in subjective quality scores. Alerts should include the affected feature, version, and trace links needed for investigation. ## Debugging an AI incident When users report that an assistant is giving poor answers, start from the user-visible symptom and walk down the trace. A practical investigation sequence is: 1. Confirm the affected workflow and time window. 2. Compare request volume, failures, latency, quality signals, and cost with a healthy period. 3. Filter by deployment, prompt version, model route, retrieval version, and account segment. 4. Inspect representative traces, including successful and unsuccessful examples. 5. Identify whether the problem began in retrieval, tool execution, prompt construction, model generation, parsing, or presentation. 6. Check whether telemetry is incomplete. Missing spans can make a healthy component appear responsible. 7. Reproduce the failure with a redacted input and the same versions. 8. Record the root cause, mitigation, and follow-up evaluation. ### Common failure patterns | Symptom | Likely investigation path | Useful evidence | |---|---|---| | Answers became less relevant | Retrieval, context assembly, prompt version, model route | Retrieval version, empty-result rate, evaluator score | | Responses became slow | Queue, retrieval, tool calls, context size, provider latency | Span durations, first-token time, fallback rate | | Structured output fails | Schema, parser, prompt change, model capability | Validation status, raw error class, prompt version | | Costs rise suddenly | Routing, retries, context growth, output length | Token counts, retry count, model route, cost estimate | | Users stop completing tasks | Quality, UI presentation, latency, workflow errors | Outcome events, feedback, abandonment, release annotation | | Only some accounts fail | Data permissions, tenant-specific retrieval, account configuration | Account segment, retrieval status, tool response status | Avoid debugging from one dramatic trace. One example can reveal a failure mode, but it cannot establish prevalence. Combine trace inspection with segmented aggregate data. ## Evaluation workflows Observability tells you what happened; evaluation helps determine whether the result was good. Build an evaluation workflow that can run against a stable test set and sampled production traces. ### Create an evaluation dataset Use representative, privacy-reviewed examples covering common requests, difficult requests, empty knowledge-base cases, ambiguous instructions, tool failures, and policy-sensitive cases. Store a stable case ID and expected behavior rather than exposing raw customer content broadly. ### Define a rubric A rubric might score factual grounding, task completion, instruction following, format validity, tone, and safety. Make each criterion concrete. For example, a groundedness criterion should say what counts as unsupported content and how partial evidence is handled. ### Version everything Version the test set, rubric, evaluator, prompt, retrieval configuration, and model route. A score is not meaningful if the system that produced it cannot be identified. ### Compare before and after When changing a prompt or model, compare the same cases under the old and new configuration. Then examine production outcomes after release. Offline improvement does not guarantee better user experience if latency, cost, or integration failures increase. ### Use human review strategically Human review is expensive but valuable for ambiguous or high-impact cases. Sample cases based on uncertainty, disagreement between evaluators, negative user feedback, and large changes in scores. Record reviewer agreement and instructions where the decision matters. ## Comparing observability use cases | Use case | Key events | Important dimensions | PostHog fit | |---|---|---|---| | Chat assistant | Request, generation, feedback, outcome | Model, prompt, session, account | Strong for product behavior and cohort analysis | | Retrieval-augmented generation | Retrieval, generation, citation check | Corpus, retrieval version, document count | Strong when retrieval metrics are instrumented | | Agent workflow | Planning, tool calls, approvals, outcome | Tool, step, retry, human intervention | Useful for funnel analysis; needs careful trace design | | AI content generation | Request, draft, edit, publish | Content type, model, editor actions | Strong for adoption and completion metrics | | Customer support automation | Classification, response, escalation | Queue, category, resolution, agent handoff | Strong for workflow outcomes and segmentation | | Batch inference | Job, item, model result, error | Batch ID, item status, duration, cost | Useful for job-level reporting; pair with job logs | | Safety monitoring | Policy check, block, appeal, review | Policy, category, reviewer decision | Useful for trends; sensitive data controls are essential | ## Choosing supporting tools from the directory The directory tools can complement an observability program when their roles are clearly defined. [Notion](/en/tools/notion-ai) can hold the AI system runbook, event dictionary, prompt change log, evaluation rubric, and incident review notes. Keep links from documentation to dashboard names and version identifiers so an analyst can move from a decision to evidence. [Gemini](/en/tools/gemini) may help a team summarize a carefully sampled and redacted set of traces or draft an evaluation rubric. It should not be treated as an automatic source of truth. A reviewer still needs to verify conclusions against the underlying events. [Replicate](/en/tools/replicate) can be relevant when your application invokes hosted models or experiments with different inference workloads. Instrument provider calls with model, version, latency, usage, error, and outcome properties. Provider availability does not eliminate the need for application-level monitoring. For application prototyping, [Lovable](/en/tools/lovable) and [Bolt.new](/en/tools/boltnew) can help create interfaces or workflow prototypes. Before using a generated prototype with real users, add explicit server-side telemetry, privacy controls, error handling, and test coverage. Generated application code should not be assumed to have production-grade observability. For visual and media workflows, [Runway](/en/tools/runway), [Luma AI](/en/tools/luma-ai), [Ideogram](/en/tools/ideogram), and [Canva](/en/tools/canva) may be part of a content pipeline. Track asset-generation requests, approval, revision, export, and publication as product events. The fact that a media tool produces an asset does not by itself reveal whether the asset was useful or whether the workflow was profitable. ## Anti-patterns to avoid ### Logging everything forever Raw prompts and outputs are expensive, sensitive, and difficult to govern. Store the minimum content needed for a defined debugging or evaluation purpose, and use retention controls. ### Treating HTTP success as AI success A successful response can contain an incorrect answer, invalid schema, irrelevant retrieval, or failed business action. Link technical status to downstream outcomes. ### Using one score as quality A thumbs-up rate, evaluator score, or completion rate can move for reasons unrelated to answer quality. Use multiple signals and inspect segments. ### Mixing prompt versions If prompt changes are not versioned, before-and-after comparisons become speculative. Version templates, policies, parsers, retrieval settings, and model routes. ### Ignoring missing telemetry A dashboard can look healthy because failed requests never emitted completion events. Emit start and failure events, monitor event completeness, and include abandoned traces in analysis. ### Calculating cost with hidden assumptions Cost estimates depend on tokenization, provider pricing, caching, retries, and routing. Store the pricing-table version and label estimates clearly. ### Sending client-only events for server operations Browser events may miss provider failures, tool errors, and server retries. Emit authoritative operational events from the server and use client events for user interaction. ### Optimizing latency at any cost Reducing context or model size may lower latency while damaging answer quality. Evaluate the complete tradeoff across quality, cost, and completion outcomes. ## A practical implementation checklist ### Before instrumentation - Define the user-visible AI workflows. - Identify data owners and privacy requirements. - Choose trace, session, user, and account identifiers. - Define event names and required properties. - Decide what raw content, if any, may be retained. - Define the outcome for each important workflow. ### During implementation - Emit start, success, and failure states. - Create child spans for retrieval, tools, moderation, and generation. - Record model, prompt, retrieval, parser, and application versions. - Capture first-token and total latency for streaming responses. - Record usage data and mark estimates. - Test provider failures, malformed responses, and user cancellation. ### Before launch - Validate redaction with realistic examples. - Check that identifiers remain consistent across services. - Compare dashboard counts with application logs. - Confirm that failures cannot silently disappear. - Add release annotations and ownership information. - Define alert thresholds and escalation paths. ### After launch - Review representative traces regularly. - Track quality and product outcomes, not only reliability. - Sample negative feedback for human analysis. - Revisit retention and access policies. - Audit event schemas when models or workflows change. - Document incidents and update evaluations after fixes. ## Frequently asked questions ### Is PostHog an AI observability platform? PostHog can support AI observability when AI operations are modeled as structured events and linked to users, sessions, versions, outcomes, and experiments. Its strongest use case is connecting AI behavior with product analytics. It should not automatically be assumed to replace specialized tracing, log management, infrastructure monitoring, or model evaluation systems. ### What should I log for an LLM request? At minimum, log a trace ID, operation name, model and provider, prompt version, application version, start and completion status, latency, token usage when available, error class, and downstream outcome. Add retrieval and tool spans for applications that use them. Store raw prompt and completion content only under an approved privacy and retention policy. ### Should prompts and model outputs be sent to PostHog? Only when there is a defined reason and appropriate controls. Metadata is usually safer and sufficient for aggregate analysis. If raw content is needed for debugging, redact secrets and personal data, restrict access, and set a limited retention period. Remember that context and retrieved documents may contain sensitive information even when the visible prompt does not. ### How do I measure AI quality? Combine explicit feedback, implicit behavior, automated checks, human review, and downstream task outcomes. Version the rubric and evaluator. Analyze quality by feature, model, prompt version, retrieval configuration, and user segment rather than relying on one overall score. ### How can I measure AI cost accurately? Capture provider usage data where available and calculate estimates with a versioned pricing table. Include retries, fallbacks, tool calls, retrieval-related processing, and batch work when relevant. Report both cost per request and cost per successful outcome, and label estimates separately from invoiced provider totals. ### Can PostHog monitor AI agents? It can help analyze agent workflows if each meaningful step is instrumented with a shared trace ID and separate operation or span identifiers. Record planning, tool calls, retries, approvals, failures, and final outcomes. Agent traces can become high-cardinality and expensive, so choose event detail and retention deliberately. ### Which adjacent tools should I use with PostHog? Use documentation tools such as [Notion](/en/tools/notion-ai) for runbooks and governance, model execution services such as [Replicate](/en/tools/replicate) for relevant inference workflows, and application builders such as [Lovable](/en/tools/lovable) or [Bolt.new](/en/tools/boltnew) for prototypes that still require proper telemetry before production. Use each tool according to its actual role rather than treating it as an observability substitute. ### What is the first dashboard I should build? Start with one workflow and show request volume, failure rate, time to first token, total latency, feedback, downstream completion, estimated cost, model route, and prompt version. Add trace links for representative failures. Expand only after the first dashboard helps the team make a concrete operational or product decision. ## Conclusion PostHog AI observability tools are best understood as an instrumentation and analysis approach built on PostHog’s event model, rather than as a single magic feature. The value comes from connecting model execution to the user and business context around it. Start with stable trace identifiers, a small event taxonomy, strong privacy controls, and explicit product outcomes. Add retrieval, tool, generation, evaluation, latency, and cost details as the workflow requires. Then use dashboards and sampled traces together: aggregate data shows patterns, while traces explain causes. In 2026, reliable AI products will be distinguished less by whether they can call a model and more by whether their teams can understand, evaluate, and improve what happens after the call. A well-designed PostHog implementation gives product and engineering teams a shared view of that behavior, provided it is paired with disciplined versioning, testing, governance, and the right supporting systems.