Agent Observability (OTel Tracing)¶

Structured execution traces for every agent run, stored in PostgreSQL and visualized via a custom Django admin UI. The same spans also flow to Sentry for ops-style cross-service tracing — see Observability Stack for how this doc fits into the broader telemetry architecture.

Architecture¶

The system uses OpenTelemetry (OTel) Python SDK to instrument the agent's reasoning loop. Spans flow through a single TracerProvider to two consumers: the in-app PostgreSQL exporter (powering the admin drill-down UI) and Sentry's SentrySpanProcessor (for cross-service tracing). The OTel trace_id is the universal correlation key — it appears as trace_id on every log line and as Sentry's trace field on every event.

Agent reasoning loop (agent_core.py)
  |
  |  OTel spans created at each step
  v
BatchSpanProcessor (background thread)
  |
  v
PostgresSpanExporter
  |
  v
OTelTrace / OTelSpan tables (PostgreSQL)
  |
  v
Agent Trace Viewer (Django admin UI)

Span Hierarchy¶

Each agent run produces a single trace with this span tree. Names and sentry.op values follow the OpenTelemetry GenAI semantic conventions so Sentry's AI Agent UI recognizes them out of the box.

POST /api/chat/stream                       # ASGI request (OpenTelemetryMiddleware) - never persisted
  |-- invoke_agent campuscore_agent         # 1 per chat turn (op = gen_ai.invoke_agent)
  |     |-- iteration_1
  |     |     |-- chat <model>              # LLM call (op = gen_ai.chat)
  |     |     |     |-- POST                # httpx → anthropic/openai (auto)
  |     |     |-- execute_tool <name>       # tool call (op = gen_ai.execute_tool)
  |     |     |     |-- SELECT documentchunk # psycopg → pgvector (auto)
  |     |-- iteration_2
  |     |     |-- chat <model>
  |     |     |-- token_management          # internal helper
  |     |-- iteration_N ...

The ASGI request span starts before enter_agent_run(), so it is never tagged and never persisted - every persisted invoke_agent row carries a parent_span_id with no matching row, and the turn page reparents such orphans under the agent root at read time (build_waterfall in apis/admin_agent_trace_rendering.py). The middleware is constructed with exclude_spans=["send"] so an SSE response's per-frame send events never become spans, and the exporter refuses any ... http send span by name as a backstop.

The root span always ends. A run interrupted mid-stream (user stop, client disconnect) or failed before the loop ends its root span from AgentCore.run's finally, stamping campuscore.agent.status as cancelled or error - an unended span is never exported, and its trace row would otherwise sit permanently empty. Rootless rows from process kills are swept by prune_traces after a day.

Legacy names (agent_run, llm_call, tool_call.<name>) persist in the database for traces written before the GenAI rename. The admin's classify_span and root-span lookup handle both naming styles; no migration is required for historical data.

Key Files¶

File	Purpose
`services/observability/__init__.py`	Package exports: `init_tracing`, `shutdown_tracing`, `get_tracer`
`services/observability/tracer.py`	`TracerProvider` setup; registers `AgentRunTaggerProcessor`, `PostgresSpanExporter`, and (when DSN set) `SentrySpanProcessor` + `SentryPropagator`
`services/observability/attributes.py`	OTel GenAI semantic attribute constants (`gen_ai.`) + CampusCore extensions (`campuscore.`); legacy `agent.` / `llm.` / `tool.*` aliases retained for back-compat
`services/observability/agent_run_tagger.py`	Contextvar + `on_start` span processor that marks every span created inside `AgentCore.run` — used by the exporter to filter agent-only traces
`services/observability/instrumentation.py`	OTel auto-instrumentors for psycopg, httpx, and stdlib logging; pytest-gated to avoid feedback loops
`services/observability/exporter.py`	`PostgresSpanExporter` — filters to agent-tagged spans, groups by trace_id, bulk-creates DB rows
`campus_core/asgi.py`	Wraps the ASGI app with `OpenTelemetryMiddleware` so the request span is created in the async context (NOT via `DjangoInstrumentor`)
`campus_core/observability/sentry_setup.py`	`sentry_sdk.init(instrumenter="otel", enable_logs=True, ...)` + FERPA scrubber wiring
`models.py` (OTelTrace, OTelSpan)	PostgreSQL storage for traces and spans
`apis/admin_agent_trace_apis.py`	Django views: the triage screen, the turn page, the span inspector, the live chunk endpoint
`apis/admin_agent_trace_rendering.py`	Span interpretation: `build_waterfall`/`TurnWaterfall`, `span_kind`, `parse_tool_span`/`parse_llm_span`, and `structure_attributes` for the generic attributes table
`templates/admin/agent_traces.html`	The triage shell
`templates/admin/agent_trace_turn.html`	The turn page (waterfall, budget bar, replay strip, inspector pane)
`templates/admin/partials/admin_at_*`	HTMX partials: the turn list, the span inspector, and the live chunk content
`static/css/admin_tools.css`	Shared tokens: span-kind and status colours, pills, tiles, waterfall classes

Models¶

OTelTrace¶

One row per agent run. Denormalized summary fields for efficient listing/filtering: - id (UUID) — OTel trace_id - conversation (FK, nullable) — link to conversation - user_message, ai_message (FK, nullable) — linked messages - query, status, model - started_at, completed_at, and total_duration_ms, which Postgres computes as the gap between them - total_input_tokens, total_output_tokens - iteration_count, tool_call_count, tools_used (JSONField)

OTelSpan¶

One row per span in the trace. Standard OTel shape: - id (CharField) — OTel span_id hex - trace (FK to OTelTrace) - parent_span_id — for tree reconstruction - name — e.g., invoke_agent campuscore_agent, iteration_1, execute_tool search - attributes (JSONField), events (JSONField) - started_at, ended_at, and duration_ms, which Postgres computes as the gap between them

Instrumentation Points¶

Manual spans created by AgentCore.run() (GenAI conventions):

Location	Span Name	`sentry.op`	Key Attributes
`agent_core.run()`	`invoke_agent <name>`	`gen_ai.invoke_agent`	`gen_ai.prompt`, `gen_ai.conversation.id`, `gen_ai.request.model`, `campuscore.agent.status`, token totals
Reasoning iteration	`iteration_{n}`	—	`campuscore.iteration.number`, tool call count
LLM call	`chat <model>`	`gen_ai.chat`	`gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, `gen_ai.response.finish_reasons`
Tool execution	`execute_tool <name>`	`gen_ai.execute_tool`	`gen_ai.tool.name`, `gen_ai.tool.input`, `gen_ai.tool.output`, `campuscore.tool.success`, `campuscore.tool.output_truncated`, `campuscore.tool.output_original_chars`
Token management	`token_management`	—	`campuscore.token_management.strategy`, message counts

Tool output storage¶

gen_ai.tool.output is written through bounded_tool_output (services/observability/tool_output.py), the single home of the storage contract; a stored structured payload always parses as JSON. Search hits that live in the database (entries carrying a chunk_id) store their metadata plus a 300-character content excerpt with the original content length, and the viewers resolve the full chunk live through the admin_at_chunk_content endpoint. Ephemeral payloads store raw up to a 32,000-character ceiling; above it they are trimmed longest-strings-first (never below a 200-character floor, keys untouched) so entry identity survives. campuscore.tool.output_truncated is written on every tool span with output - True or False - and marks a row written under this contract; campuscore.tool.output_original_chars records the pre-trim serialized length when a cut happened. Rows written before the contract were prefix-sliced at 4,000 characters; the readers salvage their arrays item-by-item and apply the 4,000-character length heuristic only when the truncation attribute is absent.

Auto-instrumented spans created during an agent run (inherit parent context from the surrounding manual span):

Source	Span Name	Where it appears
`PsycopgInstrumentor`	`SELECT main_app_documentchunk` (etc.)	Children of `execute_tool` (search) and `chat` (token counting)
`HTTPXClientInstrumentor`	`POST`, `GET`	Children of `chat` (anthropic/openai API calls) and `execute_tool` (e.g., Cohere reranker)
`OpenTelemetryMiddleware`	`POST /api/chat/stream`	Root request span (parent of `invoke_agent`); untagged, so never persisted

The auto-instrumented spans are kept inside agent traces because they're the actual network and DB calls the agent made — exactly what's useful for offline analysis ("why was iteration 2 slow?" → child span shows the LLM API took 8 seconds).

Admin UI¶

Vocabulary rule. The loop is the run: one turn runs the ReAct loop once, and each pass through it is an iteration. UI copy titles the collection "Agent Loop" and counts iterations ("Iteration N", "Max iterations"); it never counts loops. Every wire name already speaks iteration (iteration_{n} spans, campuscore.iteration.* attributes, the max_iterations status, the agent.iteration_count metric, AGENT_MAX_ITERATIONS), so UI copy and wire vocabulary agree.

Two operator-only (superuser) surfaces read the persisted traces.

The Agent Traces tool at /cc_admin/agent-traces/ is a two-screen observability console: - Triage (the landing screen): time-range select, aggregate tiles (turns, error rate, p50/p95 latency, tool calls), a turns-over-time chart by status, preset chips (errors, max iterations, cancelled, slower than p95, tool failures), status/model/tool/conversation filters, full-text search, and a paginated turn table with an honest total. - Turn page at /cc_admin/agent-traces/<trace_id>/: header stats with a time-budget bar (model / tools / everything else), a waterfall over every persisted span - orphans reattached under the root, infrastructure spans collapsed behind one counted row - an iteration replay strip titled "Agent Loop", and a span inspector opened per row (deep-linkable via ?span=<id>). - Span inspector: typed per-kind bodies - tool spans show arguments, the parsed result block, and a note when the model saw a capped rendering; LLM spans show model, message count, token usage, input messages, and output; infrastructure spans get the grouped attributes table.

Colour is two non-overlapping channels, defined in static/css/admin_tools.css: span-kind (reasoning violet, model cyan, tool pink, infrastructure slate) on bars and dots, and reserved status colours on icon+word pills only. Red appears only on failure states.

The Conversation Viewer at /cc_admin/conversation-viewer/ shows chats as the user saw them; clicking an AI message opens its Trace Details panel, rendered from the turn's OTelTrace row plus its tool spans. The panel shows the query, status, duration, and token totals, then each tool call with its arguments and retrieved documents. Tool-call parsing is shared between both tools via apis/admin_agent_trace_rendering.py (parse_tool_span), and both render the result block through one partial (admin/partials/admin_tool_result_content.html): a truncation banner stating "kept X of Y characters", entry headers that fall back to the source_url when a hit has no title, and a "Show full content" control on excerpted hits that loads the live chunk content - or an honest missing message when the chunk was re-ingested away.

Cross-links run both ways: the turn page links "View Conversation" (?conversation=<uuid> auto-selects the chat), and the Trace Details panel links "Open in Agent Traces" straight to that turn's page. The legacy ?trace=<uuid> form on the triage shell redirects to the turn page, so old links keep working.

Initialization¶

init_tracing() is called from MainAppConfig.ready() in apps.py. The BatchSpanProcessor runs on a background thread, flushing spans every 5 seconds.

Async Generator Spans¶

The agent's run() method is an async generator, so context managers (with start_as_current_span()) can't be used across yields. Instead, spans are manually managed:

span = tracer.start_span("invoke_agent campuscore_agent")
ctx = trace.set_span_in_context(span)
# ... yield events ...
span.end()

Child spans receive the parent context explicitly via the context= parameter.

OTel as the single source of truth (Sentry bridge)¶

OTel is the only system that creates spans for CampusCore. Sentry consumes them via SentrySpanProcessor, so the OTel trace_id is what populates Sentry's trace field — the two never diverge.

ASGI request → OpenTelemetryMiddleware (asgi.py)
              │  opens root request span
              ▼
              Django middleware chain → view → ChatService.generate_response
                                              │
                                              ▼
                                              AgentCore.run
                                              │  starts "invoke_agent" child span
                                              │  starts "chat" / "execute_tool" grand-children
                                              ▼
                                              all spans end → BatchSpanProcessors:
                                                ├─ PostgresSpanExporter → OTelTrace/OTelSpan rows
                                                └─ SentrySpanProcessor → Sentry transactions/spans
                                                   (uses the same trace_id)

Key wiring:

campus_core/asgi.py wraps the inner ASGI app with OpenTelemetryMiddleware from opentelemetry-instrumentation-asgi. The Django ASGI handler then runs inside that span, so trace.get_current_span() works correctly in every downstream coroutine — including async middleware, streaming generators, and the agent's async loop. We deliberately avoid DjangoInstrumentor() because its sync-style process_request/process_response middleware loses OTel context across the sync_to_async boundary under uvicorn/ASGI.
campus_core/observability/sentry_setup.py initializes Sentry with instrumenter="otel", telling it not to auto-create transactions. DjangoIntegration is retained but with all tracing knobs disabled (middleware_spans=False, signals_spans=False, cache_spans=False) — kept only for the error-capture path (request body, user attribution).
apps/main_app/services/observability/tracer.py attaches both PostgresSpanExporter (for the admin UI) and SentrySpanProcessor (for Sentry) to the same TracerProvider. Also installs SentryPropagator as the global OTel text-map so outbound HTTP requests carry sentry-trace + baggage headers, joining downstream services into the same trace.
apps/main_app/services/observability/instrumentation.py runs PsycopgInstrumentor, HTTPXClientInstrumentor, and LoggingInstrumentor in AppConfig.ready(). It's gated off during pytest (set OTEL_INSTRUMENT_IN_TESTS=1 to override) because PsycopgInstrumentor wraps every Django ORM cursor — including the ones PostgresSpanExporter uses — creating a write-amplification feedback loop.

Log correlation¶

Every JSON log line carries trace_id, span_id, request_id, and user_id. The values come from campus_core/observability/logging_filters.py::RequestIdFilter, which reads the current OTel span via trace.get_current_span(). Because OpenTelemetryMiddleware opens the span at the outermost ASGI layer, every log emitted during a request sees the same trace_id — and that value matches Sentry's trace field for the same request.

CloudWatch Logs Insights query example:

fields @timestamp, level, logger, message, trace_id, request_id, user_id
| filter trace_id = "abc123..."
| sort @timestamp asc

Why no DjangoInstrumentor¶

opentelemetry-instrumentation-django adds itself as a sync-style Django middleware (process_request/process_response). Django wraps sync middleware via asgiref.sync.sync_to_async when serving an async view, which runs the middleware in a thread with a copied contextvars context. Mutations to the OTel current-span contextvar inside that thread don't propagate back to the outer async context, so downstream async code (including streaming generators) sees no active span. We diagnosed this empirically: trace= was empty on every "request completed" log even though DjangoInstrumentor reported is_instrumented_by_opentelemetry == True. The fix was switching to OpenTelemetryMiddleware (a true ASGI middleware) wrapping the ASGI app itself.

Filtering: only agent spans reach the admin¶

With OTel auto-instrumentation on, every Django request produces a tree of spans (ASGI root, DB queries, outbound HTTP, etc.). Persisting all of them into OTelTrace/OTelSpan would drown the agent admin with non-agent traffic — admin views, static assets, healthchecks, the conversation list endpoint, etc.

To keep the admin focused, we tag spans created inside an agent run and the exporter filters on the tag:

File	Role
`agent_run_tagger.py`	Defines a contextvar `_in_agent_run` and a `SpanProcessor` (`AgentRunTaggerProcessor`) that sets `campuscore.in_agent_run = True` on every span started while the contextvar is True. Registered as the first processor on the `TracerProvider` so the marker is in place before any downstream processor reads the span.
`agent_core.py`	Calls `enter_agent_run()` at the top of `AgentCore.run` (just before opening the `invoke_agent` span) and `exit_agent_run(token)` in `finally`. Every span started in this window — including child psycopg/httpx spans inside tool calls — inherits the marker via OTel's contextvar-driven context propagation.
`exporter.py`	First action in `persist_spans` is `[s for s in spans if s.attributes.get(AGENT_RUN_MARKER_ATTR)]`. Untagged spans return immediately without touching the DB.

Net effect: the admin shows exactly the spans that happened during an agent loop. The ASGI request span, DB queries for view dispatch, healthcheck spans, etc. are dropped at the exporter. Sentry is unaffected — SentrySpanProcessor runs as a sibling processor and sees every span regardless of the marker.

Test-suite gating¶

PsycopgInstrumentor wraps every Django ORM cursor — including the cursors PostgresSpanExporter uses to write spans. In tests, this creates a write-amplification feedback loop that materially slows the suite. enable_auto_instrumentation() in instrumentation.py detects pytest (via PYTEST_CURRENT_TEST env var or pytest in sys.modules) and no-ops by default. Override with OTEL_INSTRUMENT_IN_TESTS=1 if a specific test needs the auto-instrumentation surface active.

Golden-signal metrics¶

Beyond traces, the chat/agent/retrieval code emits counters and histograms via campus_core.observability.metrics.MetricsService (CloudWatch-bound — see Observability Stack). Alarms target these names directly in infrastructure/app/monitoring.tf.

Surface	Metric	Where emitted
Chat	`chat.request.count`, `chat.request.error_count`, `chat.request.duration_ms`	`chat/chat_service.py::generate_response`
Chat	`chat.steps_per_response`, `chat.tools_used_per_response`	`chat/chat_service.py::generate_response` (on success)
Agent	`agent.run.count`, `agent.run.error_count`, `agent.run.status.<completed\\|error\\|max_iterations>`	`_emit_agent_metrics` in `agent_core.py` (called from the completion path and the error handler; runs ending in the outer `finally` emit no metrics)
Agent	`agent.iteration_count`, `agent.tool_call_count`, `agent.tokens.input`, `agent.tokens.output`	same
Agent	`agent.no_tool_calls_count`	same — fires when an agent run completed without ever calling a tool (expected for general-assistance turns; the rate is the trend signal for routing changes)
Tool	`tool.<name>.success_count`, `tool.<name>.failure_count`, `tool.<name>.exception_count`, `tool.<name>.duration_ms`	`execute_tool_traced` in `tool_execution.py`
Retrieval	`retrieval.duration_ms`, `retrieval.request_count`, `retrieval.result_count`	`retrieval_service.py::retrieve` (`result_count` samples successful searches only)
Retrieval	`retrieval.empty_result_count`, `retrieval.error_count`	same - `empty_result_count` counts genuinely empty successes only; a hard error increments `error_count` and neither of the result metrics, so an outage cannot masquerade as a relevance problem
LLM	`llm.request.count`, `llm.request.error_count`, `llm.request.duration_ms`, each also emitted as a per-provider family (`llm.request.count.<provider>`, ...)	`llm_infra/metering.py::_emit_call_metrics`, fired from the same seam that writes every `ModelUsageEvent` row (all LiteLLM callbacks plus the explicit rerank paths), in web and worker alike

psycopg v3 + OTel¶

CampusCore uses psycopg[binary]>=3.2. The matching OTel instrumentor is opentelemetry-instrumentation-psycopg (v3 only — the v2 instrumentor is a separate package, opentelemetry-instrumentation-psycopg2). Together this means every DB query made by Django's ORM or by hand-rolled connection.cursor() blocks becomes an OTel span when auto-instrumentation is active.

If you ever need to revert to psycopg2 (you shouldn't), also swap the instrumentor name in instrumentation.py::_instrument_psycopg or those spans silently disappear from the trace tree.