Back to content
    AI and Software DevelopmentDeveloperTechnical security guide

    What to Log in AI Agents: Observability, Tool Calls, Errors, and Cost Guide

    Learn how to track AI agent tool executions, MCP calls, approval decisions, token usage, errors, and costs with privacy-first observability.

    Published: August 23, 2026Updated: August 23, 2026InoviqLab
    AI agent observability trace visualization showing end-to-end user request, model calls, tool executions, approval decisions, and cost metrics.
    Audience
    Developer
    Content type
    Technical security guide
    Evergreen guide. Publication and update dates are tracked in article metadata.
    AI AgentObservabilityOpenTelemetryTool CallMCPToken UsageAI CostAudit LogTracing

    Short answer

    Deploying autonomous AI agents, LLM function calling, and RAG pipelines in production introduces operational challenges: non-deterministic outputs, API latency spikes, unexpected token cost inflation, and hidden prompt failures.

    Traditional application logging (capturing HTTP 500 errors or database exceptions) is insufficient for AI systems. You must implement **AI Observability** to capture prompt inputs, completion outputs, token usage counts, latency breakdowns, and vector retrieval scores.

    AI Agent Observability Architecture:

    System Application Logs (Winston / Pino) + LLM Execution Tracing (LangSmith / Helicone / OpenTelemetry) + Token & Cost Allocation Dashboards + Prompt Versioning & Evaluation Metrics =

    Full AI Observability

    Key Observability Metrics for AI Agents:

    Observability MetricWhat It MeasuresWhy It Matters
    Prompt & Completion TokensTotal token consumption per requestControls monthly OpenAI / Anthropic API costs
    End-to-End LatencyTime to first token (TTFT) and full response timeIdentifies user experience bottlenecks
    Tool / Function Calling TracesExternal API calls executed by the agentDetects agent looping or execution failures
    Vector Retrieval Score (RAG)Similarity score of retrieved context chunksPrevents LLM hallucinations due to weak context
    Error & Fallback RatesAPI rate limit breaches or JSON parsing failuresEnsures reliability and automated fallback routing

    1. Implementing OpenTelemetry and Tracing for AI

    Use open observability standards like OpenTelemetry or dedicated AI observability platforms (LangSmith, Helicone, Phoenix) to trace agent tool calls:

    • Assign unique `trace_id` to every user session.
    • Log exact prompt inputs, system instructions, and raw LLM completions.
    • Mask sensitive PII (emails, API keys) before logging payloads.

    AI Observability Checklist

    • [ ] Track token consumption per user, session, and prompt template
    • [ ] Log LLM execution latency and tool-calling execution steps
    • [ ] Sanitize PII and sensitive user input from raw prompt logs
    • [ ] Set up automated cost alerts when daily token budgets exceed thresholds

    Sources

    • OpenTelemetry Documentation — Tracing and Metrics Specifications
    • NIST SP 800-92 — Guide to Computer Security Log Management

    Share

    LinkedInX