Optionaloptions: { onUnreadable?: (id: string, error: StoreError) => void }
OptionalonUnreadable?: (id: string, error: StoreError) => voidCalled for a record that cannot be read, which is then left out of the index. Without it the first such record aborts the rebuild, as it always did.
Drop and rebuild the whole index from the records on disk.
Batched, because doing it a record at a time is what made indexing expensive:
indexRecordreads, parses, serialises and atomically writes one file per shard the record's tokens touch, and a real record touches 130–182 of the 256. Two hundred records took ten seconds, 96% of it shard I/O. Grouping every record's postings first turns that into one write per shard for the whole corpus, and one catalog write instead of one per record.Holds the corpus lock (
CorpusLockedErrorwhile a sync writes).