How long one batch of a sync runs before its index postings and catalog rows are
written (batchCatalog). A run killed outright (SIGKILL, a power cut) loses the rows
and postings of at most one batch — the records are on disk, and the next sync over
the window, or ka reindex, catalogues them again. One batch for the whole run lost
every row of a long sync to a Ctrl-C.
It was 25 refs. Each batch rewrites every shard its records touch, which for long
papers is all 256 — the whole index, 202 MB in a corpus of 7,500 — so at a fast source
it rewrote the index every few seconds (issue #30). By time, the cost stays a share of
the run.
How long one batch of a sync runs before its index postings and catalog rows are written (
batchCatalog). A run killed outright (SIGKILL, a power cut) loses the rows and postings of at most one batch — the records are on disk, and the next sync over the window, orka reindex, catalogues them again. One batch for the whole run lost every row of a long sync to a Ctrl-C.It was 25 refs. Each batch rewrites every shard its records touch, which for long papers is all 256 — the whole index, 202 MB in a corpus of 7,500 — so at a fast source it rewrote the index every few seconds (issue #30). By time, the cost stays a share of the run.