OptionalapiOptionaldocumentsThe run's documents, shared the same way: a URL one source fetched is not fetched again.
The engine for one job — a new one per job, best built on one shared
HostPacer (EngineOptions.pacer), so a host two jobs reach is paced once.
OptionalforceRe-extract even when nothing changed.
OptionalignoreFetch from a server whose robots.txt disallows it. Two Länder publish their Drucksachen openly and disallow every client; this is the operator's decision to make. The pipeline checks every document URL against its host's robots.txt before fetching it, whichever source produced the URL, and the override is never silent: the report warns once per host.
The jobs to run, no label twice; outcomes come back in this order.
OptionalmetadataDownload no documents. A new record gets metadata only and abstains on qa; a
stored one is re-extracted from the documents it already archived, so a
metadata correction lands and nothing it held is lost.
OptionalnowInjected clock — the only place the pipeline reads time (retrieved_at).
OptionalonOptionalonA job has finished, failed or been skipped.
OptionalonOptionalonA job is about to start; the callbacks name it by its label.
OptionalperceiverOptionalrobotsThe run's robots.txt policy, shared by every source of one syncSources run so
a host's file is read once for all of them. Built for this source when absent;
one that is given must have been built with the same ignoreRobots.
OptionalsignalStop early: checked before each ref, so the ref in hand is finished, the
catalog is saved and the report says interrupted. ka sync aborts it on the
first Ctrl-C or SIGTERM.
OptionalspaceThe disk-space guard (spaceGuard from lib-store). Given, a run whose documents
to fetch would not fit — estimated from what the source already archived — is
refused with StoreError after discovery and before the first download, and a
run stops between two refs, like an aborted one, once a volume drops below the
floor (SyncReport.lowSpace).
OptionalstopStart no job once one has failed: the rest are skipped (reason: "after-failure"). Default false.
The credential for a source that takes one (
Source.apiKeyEnv), if any.