backfill

Feeds every document of every live segment through observer.

This is how derived data is built over data that is already written, which is the whole point of the project: by the time anyone knows which model or which index they want, rewriting the documents is the thing they cannot afford. Nothing here rewrites anything — the segments are immutable and are read, in order, exactly as a compaction would read them.

The observer is asked about each segment through SegmentObserver.beginSegment and may return null to skip one it already covers, so a repeated call costs only the segments that are new. SegmentObserver.retain is called at the end with the segments that were live throughout.

Runs on the calling thread and pins a version for its duration — so a compaction may proceed underneath it, and the files it is reading will not be deleted while it is inside them. What this does mean is that a segment compacted away during the backfill is still sketched, and the following retain will tell the observer to drop it again. Correct, and cheaper than blocking maintenance for the length of a full scan.

That closing retain reports the segments that are live when the scan finishes, not the ones that were live when it started. The difference is not cosmetic: an observer that deletes derived data for segments outside the set it is given would, on the pinned set, delete the sidecar of a segment a compaction produced during a long scan — a file that is live, that nothing will rewrite, and that the observer has no other way to learn about.

Documents in the memtable are not included: they are not in a segment yet, and there is no per-segment unit to attach them to. Call flush first for a complete pass.