Skip to content

Model usage and cost tracking

Every model call the app makes is metered once, priced from one typed registry, and surfaced to clients on the analytics dashboard as a clearly-labeled estimate. This is the system that replaced the old estimate that reported $7 for a scrape run that really cost $80. The dashboard is meant to be the client-facing source of truth for model cost; validating it against real provider billing is a separate, future CampusCore-admin QA step (a deviation of more than a couple of points is a bug), not part of this system.

Why the old number was wrong

Two independent faults multiplied. The pricing table undercharged Gemini by 3.3-5x, and only the ingestion pipeline recorded usage at all - the chat and agent loop, compaction, embeddings usage, and provider-side retries were never counted. The corrected registry alone reprices that run's real tokens at $80.00 against the $80.04 the provider billed.

The pieces

  • The registry (services/llm_infra/model_registry.py) is the single source of truth for both pricing and the model allowlist. Each entry is SKU-level (input, cached input, output, reasoning, image input), versioned by effective date, and carries the provider-docs source of its rates. A model with no entry cannot be called: assert_usable raises UnpricedModelError, so an unpriced model can never silently cost $0. Rates are USD per 1,000,000 tokens.

  • The call wrappers (services/llm_infra/client.py) - acompletion, aresponses, aembedding, embedding - are the only functions the app calls. Each asserts the model is priced, asserts required capability params (embeddings dimensions), and injects the active attribution into the request metadata. litellm.drop_params is off, so a param a provider does not support raises at development and test time instead of being dropped.

  • Attribution (services/llm_infra/attribution.py) is a contextvar an attributed(feature, ...) scope sets at each entry surface (a chat run, an ingestion step). It carries feature, run_id, item_id, user_id, and conversation_id, and the wrappers copy it onto the request so the meter can file each call.

  • The meter (services/llm_infra/metering.py) is one global LiteLLM CustomLogger registered in settings.py. Every call - chat, Responses API, embeddings, sync or async, success or failure - becomes one ModelUsageEvent row: real provider usage priced through the registry, attributed from the metadata. A metering failure never fails the call.

  • The stores (models/model_usage.py): ModelUsageEvent is the append-only universal ledger (cost in micro-USD so sub-cent calls keep precision); AnalyticsModelUsageDaily is the nightly rollup the dashboard reads. The Pipeline Manager and Scraping Manager cost surfaces read the same raw ledger, aggregated on demand through services/document_ingestion/cost_reporting.py - ingestion keeps no cost store of its own.

  • The dashboard is the Model usage section of Settings > Analytics (web/src/components/settings/AnalyticsSettings.tsx), fed by the model_usage section of the analytics overview API. It shows cost by day, by feature, and by model, every value labeled an estimate with its pricing version.

Adding or changing a model's price

Append to REGISTRY in model_registry.py with rates read from the provider's pricing docs (recorded in source). Change a price by appending a new PriceVersion with a later effective date, never by editing an old one, so historical ModelUsageEvent rows stay reproducible under the version they were priced with. A model must be in the registry before any call site can use it.

Validating against provider billing

The dashboard figures are estimates priced from the registry, and the intent is for them to track real provider billing closely - a deviation of more than a couple of percent means a bug (a wrong rate, an unmetered call path, a missed token field). Confirming that is a future CampusCore-admin QA step done against the provider consoles or their billing APIs, not a client-facing feature; it is deliberately out of this system so the dashboard stays the single client-facing cost surface.