Vector Index Observability¶

Internal architecture doc for the pgvector / HNSW health dashboard and the maintenance pipeline behind it.

Operator-facing onboarding lives in Slack Setup and the index-maintenance sections of GitHub Environment Variables. Both feed into here.

The problem this solves¶

Three production failures in the previous quarter all traced back to "we had no view into the state of the index":

HNSW + WHERE-clause interaction silently zeroed out folder-filtered searches. Diagnosed only after ad-hoc psql forensics, fixed by setting hnsw.ef_search = 200 + hnsw.iterative_scan = 'strict_order'.
Folder discoverability score sat at 0.543 against a 0.46 threshold for the VSU Employees folder. One bad query away from breaking; no monitoring on the gap.
HNSW graph degradation under incremental churn. As scrape pipelines insert and delete chunks, the graph fills with stale neighbors and recall degrades — but VACUUM only cleans the heap, not the graph. REINDEX is the only fix, and we had no signal for when to run it.

Each of these would have been a 10-second click in a dashboard. Instead they were 90-minute psql sessions. This module is the dashboard.

Architecture¶

                              ┌────────────────────────────────────┐
                              │  /admin/observability/vector/      │
                              │                                    │
                              │  ┌──────────┐ ┌──────────┐         │
                              │  │ Overview │ │ Routing  │         │
                              │  └──────────┘ └──────────┘         │
                              │  ┌──────────┐ ┌──────────────────┐ │
                              │  │ Metrics  │ │  Maintenance     │ │
                              │  └──────────┘ │  [Run check]     │ │
                              │               │  [Rebuild HNSW]  │ │
                              │               └──────────────────┘ │
                              └────────────────────────────────────┘
                                              │
                                              ▼
       ┌──────────────────────────────────────────────────────────────────┐
       │  apps/main_app/services/vector_stores/                            │
       │                                                                   │
       │  vector_index_stats.py   — Overview tab data (inventory + health) │
       │  recall_eval.py          — approx-vs-exact recall, corpus-sampled │
       │  maintenance/            — metric evaluators + run_check / run_   │
       │                            rebuild / reset_stuck_rebuild          │
       ├──────────────────────────────────────────────────────────────────┤
       │  apps/main_app/services/observability/                            │
       │                                                                   │
       │  folder_discoverability  — heatmap / cannibalization / routing    │
       │  query_metrics.py        — OTel-driven traffic / latency rollups  │
       │  document_probe.py       — per-row probe panel (admin change_form)│
       └──────────────────────────────────────────────────────────────────┘
                                              │
                                              ▼
                  ┌──────────────────────────────────────────────┐
                  │  IndexMaintenanceLog (DB-backed audit trail) │
                  │  kind=check | rebuild                         │
                  │  trigger=manual | <metric_name>               │
                  │  findings={metrics, composition, ...}         │
                  └──────────────────────────────────────────────┘
                                              │
                       ┌──────────────────────┴──────────────────────┐
                       ▼                                             ▼
        ┌────────────────────────────────┐         ┌─────────────────────────────────┐
        │  Slack workflow_runs channel   │         │  EventBridge daily schedule     │
        │  via campus_core.shared_utils  │         │  modules/scheduled_ecs_task/    │
        │  .slack_utils                  │         │  → auto_rebuild_if_justified    │
        └────────────────────────────────┘         └─────────────────────────────────┘

Everything sits inside the Django app — no separate metrics backend, no Prometheus, no external collector. The dashboard reads from existing tables (main_app_documentchunk, main_app_sourcefile, main_app_knowledgefolder, main_app_oteltrace/main_app_otelspan) plus the standard Postgres observability surface (pg_relation_size, pg_index, pg_stat_user_tables, pg_statio_user_indexes, pg_stat_activity). Net new tables: just IndexMaintenanceLog.

Key files¶

services/vector_stores/vector_index_stats.py — Pydantic-typed aggregations for the Overview tab. Read-only PG queries; nothing writes. The "By source type" buckets derive from IngestSourceType; unknown there means rows genuinely missing the source_type stamp (pre-backfill history, or rows whose provenance the backfill_chunk_source_type command could not infer).
services/observability/folder_discoverability.py — N folders × M probes cosine matrix. Catches the "VSU Employees scored 0.543 against a 0.46 threshold" failure mode by surfacing it visually.
services/vector_stores/recall_eval.py — approximate-vs-exact recall over vectors sampled out of the corpus. The brute-force scan is the mathematical ground truth, and every stored embedding is a valid query, so no curated query set exists to go stale or to be institution-specific.
services/vector_stores/maintenance/ — the METRICS registry (each metric's identity, threshold, tier, and reader, declared once in metrics.py), run_check (check.py), run_rebuild and reset_stuck_rebuild (rebuild.py), the composition snapshot (composition.py), and the Maintenance tab payload (dashboard.py), re-exported through the package facade. The single source of truth for "what does failing mean."
models/vector_index.py IndexMaintenanceLog — one row per check or rebuild event. The findings JSONField is the audit trail.
management/commands/check_index_health.py, rebuild_hnsw_index.py, auto_rebuild_if_justified.py — CLI entrypoints. Same code paths the dashboard buttons call.

The metrics¶

Five metrics, each declared once in the METRICS registry in maintenance/metrics.py — name, card label, tooltip, threshold text, cost tier, and a reader that measures the signal. Every metric renders as a MetricStatus with level ∈ {ok, warn, fail, unknown}. A failing metric is what auto_rebuild_if_justified (and a human eyeballing the dashboard) uses to decide whether to REINDEX.

Every one of them is institution-agnostic: four read Postgres's own statistics, and the fifth uses the corpus as its own query set. Nothing here needs a curated eval set, which is why a fresh deployment gets a working dashboard on day one.

Metric	Data source	Threshold	What "fail" means
recall_drop	Mean recall@10 over 50 vectors sampled from the index itself. For each, how many of the graph's ten results are as close as exact search's true tenth-nearest.	< 0.90	The graph is missing neighbors a sequential scan finds. The most direct signal, because it measures the outcome rather than a proxy for it.
churn_high	`n_tup_ins + n_tup_upd + n_tup_del` from `pg_stat_user_tables` minus the baseline at last rebuild.	≥ 20% of rows OR ≥ 10K absolute	Lots of writes since last rebuild. Doesn't itself mean degradation — pure inserts grow but don't rot — but combined with other signals confirms accumulation.
bloat_high	`n_dead_tup / n_live_tup` from `pg_stat_all_tables`.	≥ 15% dead	Dead heap tuples → dead HNSW graph nodes pgvector still walks during search → degraded recall + latency. VACUUM cleans the heap; only REINDEX rebuilds the graph.
cache_low	`idx_blks_hit / (idx_blks_hit + idx_blks_read)` from `pg_statio_user_indexes`.	< 90% hit	HNSW graph paging from disk. p95 latency is about to climb. May mean `shared_buffers` is undersized or another workload is evicting our pages.
latency_p95	p95 of `execute_tool search` span durations over 24h vs a baseline p95.	≥ 1.5× baseline	Search is meaningfully slower than its baseline. Often co-occurs with `cache_low` or `bloat_high`.

How recall@10 is measured¶

The index is asked for the ten nearest neighbors of a vector already stored in it, and an exact scan is asked the same question. Two details make the number mean what it claims:

Both passes pin their own query plan. Left alone, Postgres prefers a sequential scan over the HNSW index on a small corpus — measured on a 3,175-row table, the unforced plan is Limit → Sort → Seq Scan. A sequential scan is exact, so both passes would run the same query and recall would read a meaningless 1.00 on precisely the young deployments this exists to watch. The approximate pass therefore forces the index scan and verifies with EXPLAIN that it got one; if it didn't, the metric reads unknown with the plan node it saw rather than a green number. Its hnsw.ef_search and hnsw.iterative_scan come from the same apply_hnsw_search_tuning production search uses, so the measurement describes the index as this deployment actually queries it.

Neighbors are compared by distance, not by identity. Real corpora repeat text — scraped boilerplate, a page ingested twice — so many chunks sit at byte-identical distances from a query and the two passes break those ties differently. Counting chunk IDs read 0.88 on an index that a REINDEX could not improve, with every "missed" neighbor at the same distance to seven decimal places. Counting by distance reads 1.00 on that same index.

Two properties are worth knowing before reading the number:

It is an estimate from 50 samples, so it carries about ±0.02 of sampling noise. An index sitting exactly on the threshold can move between warn and fail day to day.
With hnsw.iterative_scan = 'strict_order' the search keeps expanding until it can return ten results in strict distance order, which holds recall at 1.00 across a wide range of ef_search values. That makes recall_drop a confirmation signal rather than an early warning: it moves when the graph is genuinely damaged, and bloat_high / churn_high are what move first.

Reading combinations¶

The dashboard intentionally surfaces all five side-by-side because each one alone is ambiguous. Real diagnostic value comes from combinations:

bloat_high failing AND recall_drop failing → genuine degradation. REINDEX.
bloat_high failing alone, recall fine → recent churn but no observable impact yet. Optional REINDEX; consider waiting.
churn_high failing alone → growth or churn, but no observable impact. Look at the composition diff on the Maintenance tab to see whether it was inserts (grew) or updates/deletes (rotted).
cache_low failing → memory/eviction issue, not graph state. Don't REINDEX; check shared_buffers and concurrent workloads.

A search-quality problem that isn't the index's fault — folder filtering rejecting matches the index would return — is not on this dashboard. That signal is tracked in #98, which builds it from the analytics rollups rather than by sniffing result-summary prose.

The auto_rebuild_if_justified command picks the first failing metric and uses its name as the trigger value on the resulting rebuild log row. That choice is intentionally simple — operator-side judgment for the more nuanced combinations is what the dashboard exists for.

The maintenance log¶

Every check or rebuild writes one row to IndexMaintenanceLog. Schema:

Column	Values	Why
`kind`	`check` \| `rebuild`	Distinguishes "we measured" from "we mutated."
`trigger`	`manual` \| `recall_drop` \| `churn_high` \| `bloat_high` \| `cache_low` \| `latency_p95`	"Why does this event exist?" `manual` = button click or CLI invocation. A metric name = the daily job fired this rebuild because that metric failed.
`status`	`in_progress` \| `success` \| `failed` \| `skipped`	`in_progress` rows exceeding ~30 min after a deploy crash become "stuck" — see the Reset section below.
`total_rows`	int	Snapshot at event time.
`churn_rows_since_last_rebuild`	int	The churn reading's own number, carried as structured data alongside its display string.
`index_size_bytes`	bigint	`pg_relation_size` snapshot.
`recall_at_10_before` / `_after`	float\|null	Always null today: nothing writes them. Now that the estimate is probe-free, having `run_rebuild` fill them would give every rebuild a measured before/after.
`findings`	JSONB	Full metric list + composition snapshot + before/after dead-tuple stats. The audit trail.
`error_message`, `notes`	text	Free-form. `notes` is append-only; `error_message` captures exceptions verbatim.

Composition snapshots¶

findings.composition_before (rebuild) and findings.composition (check) carry a snapshot of:

{
  "total_rows": 30987,
  "by_folder": {"vsu-employees": 1373, "another-test-folder": 115, ...},
  "by_source_type": {"web_scrape": 28000, "knowledge_folder": 1373, ...},
  "mean_chunk_length": 2464.18,
  "captured_at": "2026-05-17T01:11:42.123456+00:00"
}

The Maintenance tab diffs the latest snapshot against the previous rebuild's snapshot to show "+1,247 rows in vsu-employees, +0 elsewhere." That's the signal for "a bulk import shifted the distribution since we last rebuilt."

Datetime serialization¶

Heap stats come from psycopg as native datetime objects. Django's JSONField uses stdlib json.dumps which doesn't serialize datetimes, so we ISO-stringify before storing — see _json_safe_dead_tup() in maintenance/rebuild.py. Easy to forget when adding new findings keys; this is also why composition uses model_dump(mode="json").

Commands¶

Three management commands, three different audiences:

`check_index_health`¶

python manage.py check_index_health [--json]

Operator-facing. Runs all five evaluators (including recall, the expensive one), writes a kind=check log row, prints a human-readable summary, exits: - 0 — all metrics ok or warn - 1 — at least one metric failed (rebuild justified) - 2 — the check itself crashed (bug or DB unavailable)

The --json flag is for cron / EventBridge: emits the findings dict as a single line on stdout. Exit codes stay the same; the JSON is supplementary.

Also fires Slack notifications via notify_workflow_event for the start + completion. The completion message header reflects the worst level seen (✓ all ok / ⚠ warnings / ✗ rebuild justified).

`rebuild_hnsw_index`¶

python manage.py rebuild_hnsw_index [--trigger TRIGGER] [--dry-run]

Operator-facing mutating command. Pipeline: 1. Snapshot composition + dead-tuple stats + size + pg_stat counter. 2. Run VACUUM ANALYZE main_app_documentchunk — cleans dead heap tuples before REINDEX so the new graph isn't born with dead nodes baked in. 3. Run REINDEX INDEX CONCURRENTLY main_app_docidx_embed_hnsw_cosine_v2 — doesn't block reads or writes, holds a ShareUpdateExclusiveLock on the table only (DDL blocks). 4. Re-snapshot the same stats. Persist the delta to the rebuild log row.

--dry-run skips the actual SQL but still writes the rebuild log row with status=skipped. Useful for verifying the dashboard's "rebuild button" Terraform/IAM/Slack glue without a real REINDEX.

--trigger controls the IndexMaintenanceLog.trigger value. Operators always pass manual (the default); the auto-rebuild command passes the failing metric name.

`auto_rebuild_if_justified`¶

python manage.py auto_rebuild_if_justified [--dry-run]

Cron-target. Always invoked by EventBridge, not humans. Pipeline: 1. Run the full check. 2. If findings.failing_metric_names is empty, exit 0 (no rebuild, no further Slack noise beyond the check's own ✓ all ok message). 3. If non-empty, call run_rebuild(trigger=<first failing metric>). The rebuild posts its own start + complete Slack messages.

Exit codes: - 0 — check clean OR rebuild succeeded - 1 — rebuild was justified but raised - 2 — the check itself raised

The exit-code split is so the EventBridge alarm distinguishes "found a problem and recovered" from "the schedule itself broke."

EventBridge schedule¶

infrastructure/app/schedules.tf  → module "auto_rebuild_schedule"
  created unless ENABLE_INDEX_MAINTENANCE_SCHEDULE=false
                                            │
                                            ▼
infrastructure/modules/scheduled_ecs_task/
  • aws_iam_role (assumed by scheduler.amazonaws.com)
  • aws_iam_role_policy (ecs:RunTask + iam:PassRole)
  • aws_scheduler_schedule (cron expression + container override)
                                            │
                                            ▼
ECS RunTask on the campuscore-worker task definition with
command override: ["python", "manage.py", "auto_rebuild_if_justified"]

The schedule is on by default, so a deployment that sets no index-maintenance variables still gets daily health checks and rebuilds. An environment opts out by setting ENABLE_INDEX_MAINTENANCE_SCHEDULE=false; the schedule disappears on the next deploy.

The worker task definition is the target, not the web one: the worker's entrypoint honors containerOverrides.command through an exec "$@" shim, while the web entrypoint hardcodes the Gunicorn bootstrap and ignores its arguments, so an override there would silently start a web server instead.

Default cron is cron(0 6 * * ? *) (06:00 UTC daily), overridable via INDEX_MAINTENANCE_SCHEDULE_CRON. The schedule pins the current task-definition revision so deploys roll forward atomically.

One schedule, not two¶

The original design had two schedules — a standalone daily check + a separate auto-rebuild. We collapsed to one because auto_rebuild_if_justified already runs the check internally with the same Slack lifecycle. Having both fire simultaneously would just duplicate the brute-force recall pass at 06:00 UTC every morning for no behavioral benefit.

If you want a check without a possible rebuild for a specific environment, set ENABLE_INDEX_MAINTENANCE_SCHEDULE=false and run python manage.py check_index_health from a shell on demand.

Reusing the module¶

modules/scheduled_ecs_task/ is generic — anything that wants to be "Django management command, daily, ECS Fargate, same image, Slack-tied" plugs in with ~15 lines:

module "my_new_schedule" {
  source = "../modules/scheduled_ecs_task"
  count  = var.enable_my_thing ? 1 : 0

  name                = "campuscore-${local.env_sanitized}-my-thing"
  schedule_expression = var.my_thing_cron
  cluster_arn         = local.base.ecs_cluster_id
  task_definition_arn = aws_ecs_task_definition.worker.arn
  execution_role_arn  = local.base.ecs_task_execution_role_arn
  task_role_arn       = local.base.ecs_task_role_arn
  subnet_ids          = local.base.subnet_ids
  security_group_ids  = [local.base.ecs_tasks_sg_id]
  container_name      = "campuscore-worker"
  command             = ["python", "manage.py", "my_thing"]
}

Per-table autovacuum tuning¶

Migration 0057 sets tighter autovacuum parameters on the document-chunk table (named main_app_documentindex when 0057 ran; renamed to main_app_documentchunk in migration 0060 — the settings persist across the rename):

ALTER TABLE main_app_documentchunk SET (
  autovacuum_vacuum_scale_factor   = 0.05,   -- VACUUM at 5% dead (vs PG default 20%)
  autovacuum_vacuum_threshold      = 500,
  autovacuum_analyze_scale_factor  = 0.05,
  autovacuum_analyze_threshold     = 500
);

Why: the PG default scale_factor=0.2 means autovacuum doesn't fire until 20% of rows are dead. For HNSW that's catastrophic — recall starts degrading visibly past 10–15% dead because dead heap tuples = dead graph nodes until the next REINDEX. The tighter 5% trigger keeps the heap mostly clean between rebuilds.

If a DBA looking at pg_settings wonders why our settings differ from the cluster default: this is intentional and per-table only. Other tables use PG defaults.

The Reset Stuck Rebuild affordance¶

If the Django process dies mid-REINDEX (deploy, OOM, ECS task killed) the in-progress log row is never updated and the dashboard button stays disabled. The Maintenance tab shows a rose banner + "Reset stuck rebuild" button when:

An IndexMaintenanceLog row exists with kind=rebuild and status=in_progress, AND
pg_stat_activity has no active REINDEX backend on main_app_docidx_embed_hnsw_cosine_v2.

The combination means "the log thinks a rebuild is running, but Postgres says otherwise — almost certainly orphaned." Clicking the button calls maintenance.reset_stuck_rebuild() which flips the row to status=failed with a diagnostic note.

If pg_stat_activity does show a live REINDEX, the reset is refused with a RuntimeError carrying the active PID(s). That's the safety check that prevents marking a real in-flight rebuild as failed and racing it with a fresh one. To genuinely cancel a stuck rebuild, the operator must SELECT pg_cancel_backend(<pid>) from psql first; the reset then accepts.

Operational playbook¶

"Rebuild HNSW is disabled and I don't know why"¶

Check the Maintenance tab. Two failure modes light up the rose banner:

Rose banner: "Rebuild log row stuck, no active REINDEX" → click Reset stuck rebuild. Button re-enables.
No banner, but button still disabled → check the Overview tab's Index health card for indisvalid: ✗ INVALID. A previous REINDEX CONCURRENTLY left a broken index entry. Don't blindly retry — investigate via SELECT * FROM pg_indexes WHERE indexname = 'main_app_docidx_embed_hnsw_cosine_v2' and consider dropping the broken index manually before rebuilding.

"Recall metric is at exactly the floor"¶

If recall_drop reads 0.90 ± 0.02 you're inside the estimate's own sampling noise, and it will move across the threshold on its own. The metric card's detail line carries the sample count and the worst single sample, which is where to start. Three things to check, in order:

Is the index valid? The Overview tab's indisvalid field. A failed REINDEX CONCURRENTLY leaves an invalid index that the planner refuses, and the recall metric will read unknown with index_not_used rather than a number.
Are dead tuples the cause? Check bloat_high. Dead heap tuples are dead graph nodes until the next REINDEX, and that is the case a rebuild actually fixes.
Did a REINDEX move it? Run one from the Maintenance tab and re-check.

Don't reach for HNSW_EF_SEARCH here. Under hnsw.iterative_scan = 'strict_order' — the deployment default, and what the approximate pass runs with — recall does not respond to it: measured on a 3,175-row index, recall reads 1.00 at ef_search of 100, 40, 10, 4 and 1 alike. Raising it and seeing no change proves nothing.

If recall stays low after a REINDEX, the index's build parameters (m, ef_construction) no longer suit the corpus size. That's a CREATE INDEX with new parameters, not a maintenance REINDEX.

"Slack stopped posting after a deploy"¶

The most common pre-deploy state vs post-deploy state mismatch:

Diagnostic log line Slack token resolution failed: settings.SLACK_BOT_TOKEN attribute PRESENT, settings value length 0, os.environ length 0 → the deploy didn't pass the token through. Check the deploy-app job's "Export Terraform variables" step output.
Same line but settings length N, os.environ N → token is fine. The bug is elsewhere — check NOTIFICATION_CHANNELS['workflow_runs'] is set.

Full playbook in Slack Setup → Troubleshooting.

"How do I add a new probe?"¶

Probes feed the Folder Routing tab and nothing else. No index-health metric reads them — recall is measured against exact search over vectors sampled from the corpus, so adding or removing a probe cannot change any metric on this page.

Edit prompts/observability_probes.py. Add a Probe(text=..., expected_folder_slug=...) at the end of the PROBES list. The probe-set hash is content-addressed (used as a cache key), so the edit busts cached folder-discoverability matrices on the next page load.

The set is still VSU-flavored, which is a real limitation of the routing dashboard on any other institution. Genericizing it belongs with the retrieval evals, where query text is the point.

What we deliberately don't do¶

Use a separate embedding_history table to track drift. The original plan included one. Walking through the actual code paths showed it doesn't earn its keep: scrape re-processing deletes and re-inserts (no in-place vector update) and the admin re-embed path uses a deterministic input — there's no realistic scenario today where a vector "drifts in place" we'd detect by comparing hashes. We can revisit if we ever introduce a non-deterministic embedding pipeline.
Wrap the Slack SDK. campus_core/shared_utils/slack_utils.py exposes get_slack_client() (returns WebClient | None) and format_slack_message() (composes Block Kit blocks). Call sites use the SDK directly. No layer of "post()" helpers; the SDK's API is the API.
Run two EventBridge schedules. See "One schedule, not two" above.
Ask an institution for an eval set before we can tell whether its index is healthy. No vector database does; they trigger maintenance from structural statistics — tombstone ratios, segment fragmentation, memory residency — and so do we. The one metric that could have needed a query set doesn't, because the corpus supplies its own. Query-text evals remain valuable for retrieval quality, which is a different question and lives with the retrieval evals.
Leave the schedule off until thresholds are calibrated. It used to default off, which in practice meant new deployments had no monitoring at all until someone remembered to set a GitHub variable — and nobody did. An uncalibrated threshold that fires a non-blocking REINDEX CONCURRENTLY too eagerly costs far less than an index nobody is watching. The thresholds in maintenance/metrics.py are still heuristic and want recalibrating once enough IndexMaintenanceLog history exists to calibrate against.