Web Scraping & Crawl Modes

How the crawler turns a scraper configuration into SourceFile rows the ingest pipeline processes, and what each Scraping Manager mode actually does. The code is the source of truth; this doc orients.

The WebPage queue model

The crawl is a database-backed URL queue. WebPage rows (apps/main_app/models/scraping.py) carry a crawl_status, the WebPageCrawlStatus enumeration - PENDING (default), CRAWLING, CRAWLED, CRAWL_FAILED, SKIPPED, constrained at the row by webpage_crawl_status_known - and the main loop in Crawler.crawl (apps/main_app/services/scraping/crawler.py) repeatedly pulls a batch of PENDING rows, fetches each URL, saves the raw content, and extracts links. The scrape_webpages management command is the thin CLI trigger over it: run_lifecycle.run_scrape wraps one crawl in the JobTracker/Slack run lifecycle. Discovered links and seed URLs are inserted with bulk_create(ignore_conflicts=True), so a URL that already has a row keeps whatever status it had - insertion never re-queues anything. The table is shared across scraper configurations; nothing about a row says which config discovered it, but every row carries a generated host column (the URL's authority segment, extracted by Postgres) that config scoping filters on - see below.

Two reset queries are the only way already-visited rows re-enter the queue:

• The force-fetch reset moves CRAWLED, CRAWL_FAILED, SKIPPED, and CRAWLING rows back to PENDING (the CRAWLING ones are orphans a crashed run left mid-flight).
• The retry-failed reset moves only CRAWL_FAILED and CRAWLING rows back.

When a run has a path filter, both resets and the queue query are bounded by the same PathScope (see below), so a path-scoped run cannot rewrite queue state outside its path. Every run is additionally bounded by its config's ConfigScope (next section), whatever the mode.

Per URL, page_processor.process_single_url decides how much work to do. Without force_fetch, a page that already has a SourceFile is not even fetched: an extracted file marks the row CRAWLED and returns, an uploaded/failed file is re-enqueued for processing without a download. With force_fetch, the page is always fetched; ContentPersistence.save_raw_content then compares content hashes, and a hash match on an already-extracted file returns the existing record so the page processor skips the SQS enqueue. That hash path is what makes re-crawls cheap: re-fetch everything, re-process only what changed.

The modes

Modes (apps/main_app/services/scrape_modes.py) are presets over four flags - force_fetch, skip_hash_check, retry_failed, reindex_extracted - plus which management command the mode runs and which extra input it requires. The mode's key and its display label live on ScrapeRun.Mode, a TextChoices enum the registry is keyed by; a database constraint keeps ScrapeRun.scrape_mode inside it, and the trigger endpoint parses the submitted form value into the enum once, rejecting anything else inline. The row stores only that key: command_argv reads the flags back out of the registry entry when it builds the command line. The trigger keeps only the inputs the mode requires, so a stale URL or path typed under another mode is dropped rather than silently narrowing the run. The CLI has no such filter, so scrape_webpages refuses --url together with --path outright: a single-page run has no path scope, and the crawler would run the page and never consult the path.

Mode	Flags	What it does
Resume Crawl	all off	Drain only rows still PENDING. No resets, no re-fetching.
Crawl Changed+New	force_fetch	Reset and re-fetch every known page; re-process changed ones; crawl newly discovered URLs.
Full Crawl	force_fetch + skip_hash_check	Re-fetch and re-process everything, hash check bypassed.
Retry Failed	retry_failed	Reset and re-crawl CRAWL_FAILED (plus orphaned CRAWLING) rows; the drain then also takes still-queued URLs, and links found on retried pages are followed.
Single Page	force_fetch, requires URL	Fetch one URL now; hash-skip if unchanged; link extraction is disabled. An out-of-scope URL is refused inline (a CrawlScopeError, surfaced as a CommandError at the CLI) - the durable fix is editing the config's allowed domains.
Path Scope	force_fetch, requires path	Crawl Changed+New bounded to a path prefix - see below.
Reprocess Failed	retry_failed, runs process_content	Re-enqueue files that crawled fine but failed processing. No crawling.
Reindex Existing	reindex_extracted, runs process_content	Re-enqueue already-extracted files for a fresh indexing pass. No crawling.

The last two modes run process_content (apps/main_app/management/commands/process_content.py), which never fetches; it re-enqueues existing SourceFile rows to the document worker. It links its JobTracker to the ScrapeRun by id (same as the crawler), posts start/complete/fail messages to the workflow Slack channel, and if anything fails between the pre-enqueue status flip and a successful enqueue, it restores the file's prior status and last_scrape_run_id (conditionally, so a worker that already advanced the row is never overwritten) - a re-run of the same mode picks the file up again.

Every run is scoped to its config

ConfigScope (apps/main_app/services/scraping/config_scope.py) bounds every reset, the queue drain, and both process_content filters to the rows the selected config's own is_allowed_domain rules admit - with two or more configs, a run for one config never rewrites or fetches another's rows. It has exactly one predicate: is_allowed_domain itself. The ORM side reads the distinct host values present in WebPage, keeps the ones the predicate accepts, and filters host__in on that list - exact by construction and one btree-indexable lookup, with no second rendering of the config's regex rules in SQL. The drain rebuilds the filter each batch, so hosts discovered mid-crawl join the scope as the crawl finds them; SourceFile reaches the scope through a WebPage id subquery.

WebPage.host is a Postgres stored generated column extracting the URL's authority (netloc including userinfo/port, matching what urlparse hands is_allowed_domain). A URL with no parseable authority stores NULL and falls outside every scope - the same verdict Python reaches on its empty netloc.

The scope also gates row creation: starting URLs outside the config's domains fail the run with a CrawlScopeError (a scoped drain could never pick them up), and redirect-chain rows and redirect destinations are only recorded when in scope - an off-scope destination marks the original row SKIPPED with the reason instead. A WebPage is a redirect exactly when it has a redirects_to target: WebPage.is_redirect reads that field, a database constraint rejects a blank target, and both queue resets clear the target so the next crawl re-determines it from the live response. The per-page Retry button in the Scraping Manager refuses (with an explanation) when no configuration covers the page's URL, rather than queueing a row nothing will ever drain. When a run has both a path filter and a config scope, they intersect - a URL must satisfy both, so PathScope's registrable-domain family never widens a config whose rules are narrower.

Path scoping

PathScope (apps/main_app/services/scraping/path_scope.py) is the single owner of path-filter matching. Constructed from the operator's path input and the config's base_url, it builds one anchored regex - the registrable domain of base_url plus any subdomain, then the path prefix bound at a segment boundary - and exposes it two ways: q(url_field) for ORM filters and matches(url) for in-process checks. /news matches /news, /news/x, and /news?page=2, never /newsroom or /foo/news/bar, and never a URL outside the config's domain family (the registrable domain of base_url and its subdomains - the same family allowed_domains admits by default). A path of / scopes to the whole domain family; an empty path raises, and a seed URL the scope itself would reject (for example a pasted full URL) fails the run loudly instead of completing at zero pages.

A Path Scope run uses the scope in four places: the seed URL, both resets, the PENDING queue query, and link insertion - discovered links outside the scope are not recorded at all, so the run leaves no stranded PENDING rows for a later Resume Crawl to trip over. process_content applies the same scope to SourceFile.metadata["url"] via q("metadata__url"); only its command line reaches this - the trigger endpoint drops a path for modes that don't require one. JSONField note: the lookup must be __regex (compiles to text extraction plus ~); __contains on a JSONField key is JSONB containment, which for string values is equality and matches no path fragment.

Run lifecycle and telemetry

A run is a ScrapeRun row driven by its launched process (an isolated ECS Fargate task in cloud, a subprocess locally - apps/main_app/services/scraping/launcher.py). The row records which launcher started it in launcher and that launcher's own identifier in launch_handle; ScrapeRun.handle decodes the pair into a LocalProcess or an EcsTask, and that is what the launcher's is_alive, describe_stop and terminate take. Completion is reconciled from the run's JobTracker by run_monitor.reconcile_run (apps/main_app/services/scraping/run_monitor.py); nothing in the lifecycle reads WebPage counts. A run that reached COMPLETED or FAILED carries its completed_at, and a FAILED one carries a non-empty error_message, both by database constraint. The tracker follows the same rule: JobTracker.status is the JobTrackerStatus enumeration, and a FAILED tracker carries a non-empty failed_reason by constraint, which the monitor copies onto the run verbatim. Each processed URL emits one ScrapeRunEvent (new / unchanged / changed / failed / skipped) through scrape_telemetry.record_event, which also maintains the run's counters. A followed redirect adds a second event: redirected on the origin URL, whose destination was crawled in the same run and has its own event. The event's transition records how it compares to the URL's previous crawl outcome - regression when it failed after succeeding, recovery when it succeeded after failing, empty when neither - and the Regressed and Recovered chips in the run's events pane filter on it. Only success and failure outcomes form that baseline, so a redirected or skipped event never stands in for the URL's last real crawl result.

Deep docs for what happens after the crawl: document-pipeline.md and document-ingestion-state-machine.md.