IndexCatalog

The indexes a store has, and the sidecars that carry them.

val indexes = IndexCatalog(directory)
DocumentStore.open(directory, StoreOptions(segmentObserver = indexes)).use { store ->
    indexes.attach(store)
    val handle = indexes.createIndex(store, IndexDefinition.inverted("$.team"))
    store.snapshot().use { snapshot ->
        indexes.read(store, handle, snapshot).use { reader ->
            println(IndexQuery.keysEqualTo(store, reader, IndexTerm.ofString("analytics")))
        }
    }
}

An index is per-segment immutable sidecar files, and that single constraint produces everything this class does. Creating one over existing data writes new files against segments that are already written, so no document is rewritten and createIndex over ten million documents costs a scan rather than a rebuild. A query uses the sidecars that exist and scans where they do not, so an index is usable while it is still building, with no cutover. dropIndex deletes files. And a compaction replaces the sidecars of the segments it consumed with sidecars for the segments it produced, as a consequence of the merge rather than as a step anybody has to remember.

There is no compaction-time merge of posting lists, and that is a decision rather than a gap. Ordinals are positions within a segment, so a compaction renumbers every one of them; merging two input posting lists would mean remapping every ordinal in both. Reading the term back out of the document — which the compaction is already holding, already decoded, on a pass it was making anyway — is strictly less work than that. So compaction awareness here is structural: each output segment opens its own observation, and retain prunes the inputs.

Two objects and the order matters, as it does for the catalog — and here it is three. The observer has to be installed in StoreOptions before the store opens, because a flush can begin the moment it does; attach loads the sidecars and covers whatever they do not; and close must be called, because nothing else releases the mappings and on Windows a mapped file cannot be deleted at all. That is honest and it is not ergonomic. Smoothing it over is rabosh-api's job.

Thread safety. Safe from any thread. The store's maintenance thread drives beginSegment, complete and retain while a reader may be inside read; one segment's accumulation happens on one thread and is not shared. Since phase 15 this catalog has a thread of its own as well — see createIndexInBackground — which drives exactly the same three callbacks and is subject to exactly the same rules.