Usage Analytics

Settings > Analytics gives institution admins four categorized sections behind a sub-tab bar: adoption & engagement, usage & feedback, knowledge base, and model usage. Model usage is an honest placeholder until model metrics are rolled up. The other three sections are computed from data the app already writes. The design premise is "write nothing, precompute everything": production request paths carry zero analytics instrumentation, and the dashboard reads only narrow per-day rollup tables.

Architecture

chat / traces / feedback / files / ingestion   (written for their own reasons)
        |
        v  nightly (EventBridge 02:45 UTC, worker task)
rollup_analytics  ->  services/analytics_rollup.rollup_window()
        |               reads .using("replica"|"default"), writes default
        v
Analytics* fact tables (models/analytics.py, one row set per UTC day)
        |
        v  request time (workspace_admin only)
GET /api/workspace/analytics/overview  +  /export (CSV per panel)
        |
        v
web/src/components/settings/AnalyticsSettings.tsx (lazy chunk with Recharts)

• apps/main_app/services/analytics_rollup.py is the one front door: rollup_window(days=..., only_date=..., dry_run=...) -> RollupReport.
• Each day is delete-then-rewrite inside one transaction, so a re-run over identical inputs produces identical rows and late writes inside the window self-correct.
• The default window self-heals: it runs from the newest AnalyticsAssistantDaily.date (the tables are their own ledger) to yesterday, clamped to [3, 30] days, so a missed night backfills on the next run and a fresh deployment backfills a bounded month.
• The command notifies the workflow Slack channel on unhandled failure and the EventBridge schedule retries twice - a silent gap becomes unrecoverable once prune_traces (03:15 UTC) ages out the raw traces.
• With USE_READ_REPLICA on, source reads go to the replica alias explicitly (campus_core.db_routers.replica_available()); writes always go to the primary, and citation FK writes use integer ids only because a replica-bound instance assigned to a primary-bound row raises ValueError.

Fact tables

All in apps/main_app/models/analytics.py, all keyed on UTC date:

Table	Grain	Key columns
AnalyticsUsageDaily	day x role (role_slug="" = all users)	active_users, new_users, conversations, messages
AnalyticsUsageHourly	day x UTC hour	messages (feeds the heatmap)
AnalyticsAssistantDaily	day	turn outcomes, latency p50/p95, answered/no-citation, feedback
AnalyticsKnowledgeDaily	day	files added/failed, ingestion run/item counts
AnalyticsCitationDaily	day x source file	citations (CASCADE: deleting a file erases its history)

Metric semantics

• Active user: sent at least one message that UTC day (Message.role="user" joined through conversation.user). Conversations whose user was deleted (SET_NULL) are excluded.
• Questions per person: derived in the client as messages / active_users per day from the all-users row - no extra rollup column. Days with no active users render as a gap in the line, never a division by zero. The headline card is the same ratio over the whole range, plus today when the range ends yesterday, so its numerator matches the "Questions asked" card beside it.
• New user: their first-ever human message falls on that day. If a user later deletes their early conversations, a recompute can shift the attribution - accepted drift, same class as the snapshot caveats below.
• Returning user: derived in the client as active_users - new_users, so the adoption bars split each day's users into first-timers and people who had used the assistant before. new_users is counted inside the active set at rollup time, so the difference can never go negative. It inherits the new-user drift above: deleting early conversations can move a person back into the new bucket on a later recompute.
• There is deliberately no charted turn total. Turn counts come from traces and question counts from messages, so the two are close but not the same number; the outcome chart says it counts assistant runs rather than implying they equal questions asked.
• Role attribution: a user counts under every active role they hold at rollup time. Role-at-time-of-use is not reconstructible because UserRole rows mutate in place. The role_slug="" row counts each user once, so role rows deliberately do not sum to it, and the UI labels role numbers "active user-days".
• Turns: OTelTrace rows by started_at (the exporter persists agent runs only). Latency percentiles are nearest-rank over the day's total_duration_ms values.
• Answered turns / citations: read from Message.agent_metadata, parsed through PersistedTurnTrace (services/chat/turn_trace.py), which is persisted transactionally on the answer row. A row the model rejects aborts that day's rollup before it writes and names the row in the failure notice; migration 0145 is what makes such a row a bug rather than an expected shape. A turn citing only web links (no local file) is not a no-citation turn; no_citation_turns counts answers whose sources list is empty. Citation rows count only campus-scoped files and non-private-folder files: conversation attachments and private personal-folder files never surface on the dashboard. Audit agent_turn rows are deliberately not the source - the async audit queue is lossy across deploys.
• Knowledge files: access_scope in (folder, campus) only; conversation-scoped chat attachments are excluded. Ingestion runs bucket on run.created; failed = status failed or abandoned. The cumulative-files series counts files added since rollups began - files predating the first rollup are invisible to it, and the chart is labeled accordingly. The knowledge chart plots files_added, files_failed, and ingestion_runs_failed grouped, not stacked: files_failed is a subset of that day's additions, while a failed import produced no file at all and would otherwise be invisible (the crawl-failure case).
• Turn outcomes are labeled by what the user experienced. turns_max_iterations is charted as an answer, not a failure, because that path still delivers a fallback response and answered_turns counts it; labeling it unfinished would contradict the uncited-share chart that divides by answered_turns. no_citation_turns means the answer pointed at no source at all, which includes the assistant asking a clarifying question, so the chart is titled "answers citing no campus document" and claims nothing about why.
• CSV-only columns: ingestion_runs, items_completed, and items_failed are exported but no longer charted; they describe pipeline mechanics rather than anything an administrator acts on.
• Written feedback exists but nothing here reads it. Rating an answer badly in chat opens a "what went wrong?" dialog whose answer is stored on Message.feedback_text (web/src/components/chat/FeedbackReasonDialog.tsx writes it through POST /api/messages/feedback). The rollups count ratings by type only, so the dashboard shows how many answers were rated badly and never why. Two things are worth knowing before building that view. A reason picked from the offered list is stored as its own label text on its own line, so counting reasons means exact-matching the strings in FEEDBACK_REASONS - which is why they have a single definition. And the text is prose a student typed, so it is redacted by key on its way to Sentry (campus_core/observability/sentry_scrubber.py) and must not be surfaced in aggregate reporting without deciding who may read it.
• Snapshot caveats: feedback (mutable one-row-per-message), ingestion-run counters (keep updating after the run's day), and role attribution all freeze with the day's last recompute. Inside the trailing window they self-correct; after it they are history.
• Timezones: daily facts bucket on UTC days (documented limitation - there is no campus-timezone setting). The hour-of-week heatmap is the exception: the API ships raw UTC (date, hour) cells and the SPA converts them to the viewer's local weekday/hour.

The client-facing surface

The tab is workspace-admin only and deliberately carries no feature flag - usage visibility is part of the accountability surface, same reasoning as the Audit Log. It is aggregate-only by design: nothing on it goes finer than role, and person-level activity stays in the Audit Log. Viewing writes an analytics_viewed audit row; exporting writes analytics_exported (critical tier). The overview endpoint accepts a range in one of two forms and reads only rollup tables plus a live "today" strip (three single-day indexed queries against Message/OTelTrace - the one deliberate exception to rollups-only). A preset is days in {7, 30, 90, 365}, a Literal, so anything else is a 422. A custom window is start and end as inclusive YYYY-MM-DD UTC dates: both or neither, start no later than end, and end no later than yesterday, because only complete days are rolled up. A valid pair takes precedence over days, which on a GET cannot be told apart from its own default, so precedence is the only implementable rule. days is still validated first, so ?days=13&start=..&end=.. is a 422 even though the bounds would have won - a case the SPA cannot produce, since it re-serializes the range it parsed.

Both forms resolve through one ResolvedRange, which reads the clock exactly once - a request crossing UTC midnight must not resolve end against one day and then compare it against another.

A free start date carries no implicit bound, so two ceilings state one each. MAX_RANGE_DAYS (731) is the widest window a request may ask for: two calendar years, which is 731 inclusive days whenever the window contains a leap day. MAX_HEATMAP_DAYS (90) bounds the hourly series separately, because at up to 24 cells per day it is the densest series by a factor of two while the client folds every cell into one of 168 weekday-hour buckets. With fully populated series a request at the cap serializes about 535 KB, of which the hourly cells are a fifth; without the heatmap bound the same request would be roughly 1.4 MB, and the extra megabyte would be discarded on arrival. The response reports the window it resolved (start, end, hourly_start) and the heatmap title names its own narrower window when there is one. today is present only when the range ends yesterday: a historical window must not be polluted with today's activity, and withholding it server-side means the client cannot add numbers it was never given. Audit rows for analytics_viewed and analytics_exported record the resolved bounds rather than a day count, which no longer identifies a window.

CSV exports stream with the audit module's formula-injection escaping and a 100k row cap; the citations panel exports per-file totals over the range. An export is named after the window it holds (analytics-usage-2026-06-01-to-2026-06-30.csv) rather than the moment it was taken. The SPA component is a lazy chunk so Recharts stays out of the main bundle. Each chart, table, and heatmap sits in its own card, two per row, so no plot spans the full content width - full-width bar charts rendered absurdly wide bars. Titles carry their own meaning and there are no explanatory subtitles: a title that needs one is the wrong title, so every qualifier an administrator needs ("total", "average", "cumulative since analytics began", "your local time") is in the title itself. Sections are a registry inside AnalyticsSettings.tsx: each entry carries the slug, tab label, CSV export target, and panel renderer. The tabs are real links, and the active section rides the ?section= URL param, so a section can be shared as a link and survives refresh.

The range picker and headline strip stay global across sections, and the range rides the URL too - days= for a preset, start=/end= for a custom window. One helper (analyticsSearch) builds every link and navigation on the page, because a writer that set only its own param would drop the other's. Custom bounds come from a hand-rolled month calendar (DateRangeCalendar.tsx, no date-picker dependency): the first click sets the start, the second sets the end and commits, and days after the newest complete one cannot be clicked, so a reversed or not-yet-complete window is unreachable rather than rejected. It opens as an overlay anchored under the pills, never as a row inside the header - anything that takes space in that header changes its height, which moves the tabs, cards, and charts below every time the picker opens or the mode changes. The calendar opens on the window in effect, whether that came from a preset or a custom range, and the trigger pill carries the chosen dates so the range is readable without opening it. The pill group wraps at narrow widths, because a cross-year label makes five pills wider than a phone. A URL asking for an unusable range - one bound, a malformed date, a reversed pair - shows the reason and loads nothing, because rendering the default range under a URL that claims something else is exactly the failure a picker is meant to remove. The span cap deliberately lives only on the server; the page surfaces its message rather than keeping a second copy that can drift. A range predating the rollup tables shows the honest empty state - rollups begin when analytics was deployed, and going deeper is an operator action (rollup_analytics --days N). The model usage section is a designed coming-soon state, not a blank page - model metrics are not collected yet, and the placeholder says so rather than faking capability.

Operations

• Backfill: python manage.py rollup_analytics --days N (an explicit --days overrides the self-healing window's 30-day clamp, so --days 365 is a valid deep backfill), or --date YYYY-MM-DD for one complete past day. --dry-run reports without writing.
• Concurrent runs are safe: each day's rewrite takes a Postgres advisory lock, so a manual backfill overlapping the nightly schedule waits instead of colliding.
• Retention: rollup rows are kept indefinitely by design - they are the durable aggregate record and stay tiny (the largest table, citations, is bounded by knowledge-base size x days and shrinks when files are deleted via CASCADE).
• First deploy: tables start empty; the first nightly run backfills up to 30 days.
• prune_traces retains 90 days of raw traces against a 30-day max recompute window - wide margin, but do not schedule the prune tighter without rethinking the rollup window.

Adding a new metric

1. Add the column to the right fact table (or a new narrow table keyed on date) and migrate.
2. Compute it inside _compute_day in services/analytics_rollup.py from existing source tables - never add hot-path instrumentation for analytics.
3. Extend the overview output model in apis/analytics_api.py, run scripts/sync_openapi_types.sh, and render it in AnalyticsSettings.tsx. A whole new category, such as real model usage, is one entry in the component's SECTIONS registry plus the panel component that entry renders.
4. Pin the hand-computed value in tests/test_analytics_rollup.py and the API shape in tests/test_analytics_api.py.

Model usage and cost

The Model usage section is the one panel not derived from the tables above. It reads AnalyticsModelUsageDaily, a rollup of the ModelUsageEvent cost ledger, and shows cost by day, feature, and model as a clearly-labeled estimate priced from the internal registry. The full design - the pricing registry and the single metered call path - lives in model-cost-tracking.md.