Skip to content

Model usage and cost tracking

Every model call the app makes is metered once, priced from one typed registry, and surfaced to clients on the analytics dashboard as a clearly-labeled estimate. This is the system that replaced the old estimate that reported $7 for a scrape run that really cost $80. The dashboard is meant to be the client-facing source of truth for model cost; validating it against real provider billing is a separate, future CampusCore-admin QA step (a deviation of more than a couple of points is a bug), not part of this system.

Why the old number was wrong

Two independent faults multiplied. The pricing table undercharged Gemini by 3.3-5x, and only the ingestion pipeline recorded usage at all - the chat and agent loop, compaction, and embeddings usage were never counted. The corrected registry alone reprices that run's real tokens at $80.00 against the $80.04 the provider billed.

The pieces

  • The registry (services/llm_infra/model_registry.py) is the single source of truth for both pricing and the model allowlist. Its entries live in the database, seeded from the code's SHIPPED_ENTRIES, as the next section describes. Each entry is SKU-level (input, cached input, output, reasoning, image input), versioned by effective date, and carries the provider-docs source of its rates. A model with no entry cannot be called: assert_usable raises UnpricedModelError, so an unpriced model can never silently cost $0. Rates are USD per 1,000,000 tokens.

  • The call wrappers (services/llm_infra/client.py) - acompletion, aresponses, aembedding, embedding, arerank - are the only functions the app calls. Each first refreshes the registry snapshot when another process changed the rows, then asserts the model is priced, asserts required capability params (embeddings dimensions), and injects the active attribution into the request metadata. Each also copies the request's reasoning effort into the metadata as cc_reasoning_effort, because LiteLLM rewrites the effort parameter for each provider before the meter sees it. Each also sets a request timeout from REQUEST_TIMEOUT_SECONDS by the model's capability unless the caller passes its own, so no call waits on a hung provider for LiteLLM's default. The bound is per attempt. A wrapper call makes one attempt: acompletion, aembedding and embedding send max_retries=0 unless the caller passes its own, and aresponses goes through LiteLLM's httpx handler, which makes one attempt already. A caller that wants retries asks for them; the embeddings client's document methods pass max_retries=2 for batch embedding, while embed_query, which a search or a folder save waits on, makes one attempt. settings.py sets litellm.DEFAULT_MAX_RETRIES = 0, because LiteLLM's embedding handler reads it as max_retries or litellm.DEFAULT_MAX_RETRIES and would otherwise turn a caller's 0 back into 2. litellm.drop_params is off, so a param a provider does not support raises at development and test time instead of being dropped. Every Anthropic chat call asks for prompt caching with one top-level cache_control field, unless the caller passes its own, because Anthropic caches nothing unless the request asks. OpenAI and Gemini cache on their own, and the wrappers never send them the field. A cache read is recorded as tokens_cached and priced at the entry's cached_input rate. A cache write is counted as ordinary input and priced at the input rate, a quarter under Anthropic's write rate, because the price shape has no cache-write rate.

  • Attribution (services/llm_infra/attribution.py) is a contextvar an attributed(feature, ...) scope sets at each entry surface (a chat run, an ingestion step). It carries feature, run_id, item_id, user_id, and conversation_id, and the wrappers copy it onto the request so the meter can file each call.

  • The meter (services/llm_infra/metering.py) is one global LiteLLM CustomLogger registered in settings.py. Every call - chat, Responses API, embeddings, sync or async, success or failure - becomes one ModelUsageEvent row: real provider usage priced through the registry, attributed from the metadata. The row's reasoning_effort is the effort the request sent, or empty when it sent none, so cost can be split by effort. A row is one LiteLLM call, however many connections the provider SDK made inside it, so SDK retries a caller asked for do not add rows. A metering failure never fails the call. The row is written by a LiteLLM hook that runs after the call returns, so a short-lived script must exit through run_metered (services/llm_infra/metered_run.py) in place of asyncio.run. asyncio.run cancels the pending hooks when the script's coroutine returns, and LiteLLM's own exit flush runs after Python has shut down the thread pool the write needs, so the last calls' rows are lost. run_metered waits for LiteLLM's pending tasks and then its logging worker's queue, under one ten-second deadline, and logs a warning if rows are still pending when the deadline passes. The long-lived web and worker processes need none of this, because their loop never closes with a hook pending.

  • The stores (models/model_usage.py): ModelUsageEvent is the append-only universal ledger (cost in micro-USD so sub-cent calls keep precision); AnalyticsModelUsageDaily is the nightly rollup the dashboard reads. The Pipeline Manager and Scraping Manager cost surfaces read the same raw ledger, aggregated on demand through services/document_ingestion/cost_reporting.py - ingestion keeps no cost store of its own.

  • The dashboard is the Model usage section of Settings > Analytics (web/src/components/settings/AnalyticsSettings.tsx), fed by the model_usage section of the analytics overview API. It shows cost by day, by feature, and by model, every value labeled an estimate with its pricing version.

The registry lives in the database

The registry's entries are rows in three tables: RegisteredModel, with its ModelAlias and dated ModelPrice rows (models/model_registry.py). SHIPPED_ENTRIES in model_registry.py is the seed, and a row added later in the Models tool sits beside the shipped ones. ModelRegistryService (services/llm_infra/model_registry_service.py) is the only writer, and it validates every write through the ModelEntry shape first.

Each process serves a snapshot of the rows merged over the shipped entries. A write bumps a shared stamp in the cache once its transaction commits, and the next model call in every process sees the moved stamp and reloads the rows. A shipped entry the database lacks still resolves, and a row the ModelEntry shape refuses is skipped with a warning, so an older task keeps serving during a rolling deploy. If the cache or the database is unreachable, the process keeps serving its warm snapshot, or the shipped entries when it has none, and leaves the failing store alone for sixty seconds with one warning a minute. The writing process drops its own snapshot on commit as well, so it reads the new rows even when the stamp bump fails. Cloud deployments share the stamp through Valkey, but the local stack's LocMem cache is per process, so the local worker sees a model added in the tool only after its next restart.

Each chat entry carries a ChatRequestSpec: its input window, its endpoint (chat completions or Responses), the reasoning efforts it accepts and their default, whether it takes a temperature, and whether it may call tools in parallel. LLMWithTools shapes every request from that spec, and every context budget reads the window through context_window_for.

The boot sequence runs sync_model_registry right after ensure_superuser. The sync writes every shipped entry over the row with its id, whatever that row's origin, and keeps an operator's default effort while the entry still accepts it. A shipped alias a tool-added row held goes back to the shipped entry, with a ! line and a warning, and never stops the boot. A shipped row the code no longer has is deleted, unless a setting or an agent model choice still names it: then it stays, and the sync prints what holds it. Rows added in the tool are never deleted by the sync, and --dry-run prints what a run would change without writing it.

Choosing the agent's model

The chat agent runs at the deployment's active model, unless the person has an override. Both are rows: AgentModelDefault holds the one active model and its reasoning effort, and AgentModelOverride holds one person's model and effort (models/agent_model_choice.py). AgentModelChoiceService (services/agent/agent_model_choice.py) is the only writer of both tables, and it reads them again on every turn, with no cache.

Every write is checked. The model must be a chat entry in the registry, and the deployment must hold that provider's API key. The providers are OpenAI, Gemini, Cohere and Anthropic, and the wrappers attach each one's key from the typed settings. OPENAI_API_KEY and GEMINI_API_KEY are required, while COHERE_API_KEY and ANTHROPIC_API_KEY are optional, so a deployment without the Anthropic key cannot select a Claude model. The effort must be one the entry accepts, and it is empty exactly when the entry accepts none.

The first boot sync sets the active model to DEFAULT_AGENT_MODEL at that entry's default effort, and a later sync never changes it. A turn with no active-model row fails with NoAgentModelDefault instead of guessing a model. A model a choice holds cannot be deleted, and a row added in the tool cannot stop being a chat model while a choice holds it. When an entry stops accepting a choice's effort, the boot sync or the registry writer moves the choice to the entry's default effort and logs which choice it moved. A log line names an override's person by user id, never by email.

The Models tool

The Models tool at /cc_admin/models/ is where a superuser changes the agent's model and the registry, and it opens from the Models tile in the Admin Center. Its views live in apis/admin_models_apis.py, and its templates are templates/admin/models.html and the admin_models_* partials. Every view reloads the snapshot when the stamp moved, and every change writes a critical-tier audit row with an ADMIN action: AGENT_MODEL_DEFAULT_UPDATED, AGENT_MODEL_OVERRIDE_SET, AGENT_MODEL_OVERRIDE_REMOVED, MODEL_ENTRY_CREATED, MODEL_ENTRY_UPDATED, MODEL_ENTRY_DELETED or MODEL_DEFAULT_EFFORT_UPDATED. The tool writes only through AgentModelChoiceService and ModelRegistryService, so it refuses exactly what they refuse, and each refusal shows beside the field it is about.

The Active model pane sets the active model and its effort, and lists the overrides. The model list is grouped by provider, and each line shows the price and the window. A model whose provider key this deployment lacks is greyed, with the key it needs. The effort chips are the ones the chosen entry accepts, and a change applies from the next turn, never to a turn already running. An override is added by the person's exact email or username, and removing one takes a second click to confirm.

The Registry pane lists every entry with a badge for where it came from: - shipped: the code ships it, and a deploy keeps it in step, so only its default effort can change here. - added here: added in this tool, and it can be edited or deleted while nothing uses it. - no longer shipped: the code dropped it while a setting or a choice still names it, and a banner says what holds it; the next boot removes it once nothing does.

A dot beside the capability says whether this deployment has the provider's key. The add form has three groups: identity, how the agent calls it (shown only for a chat model), and the price with its effective date and source. A price may not take effect after today, and editing an entry added here never rewrites a recorded price version. The edit form dates its price today, so a changed rate becomes a new version, and a rate posted for a date that already has prices is refused unless it matches them.

Adding or changing a model's price

To ship a model with the code, append a ModelEntry to SHIPPED_ENTRIES in model_registry.py with rates read from the provider's pricing docs (recorded in source), and the next boot writes it. Change a price by appending a new PriceVersion with a later effective date, never by editing an old one, so historical ModelUsageEvent rows stay reproducible under the version they were priced with. A model must be in the registry before any call site can use it.

Validating against provider billing

The dashboard figures are estimates priced from the registry, and the intent is for them to track real provider billing closely - a deviation of more than a couple of percent means a bug (a wrong rate, an unmetered call path, a missed token field). Confirming that is a future CampusCore-admin QA step done against the provider consoles or their billing APIs, not a client-facing feature; it is deliberately out of this system so the dashboard stays the single client-facing cost surface.