Zettel — Proton DocumentDB LID Space and Attribute Vector Indexing

Inside every content node (vespa/searchcore/src/vespa/searchcore/proton/), each document type is managed by a DocumentDB partitioned into three sub-databases (ready, notready, removed). Within each sub-database, global 12-byte GlobalId (GID) hashes are mapped through LidGidMap to a dense 32-bit Local Document ID (LID) in the range [1, lid_limit).

Mechanism & Upstream Grounding

1. Dense Array Indexing via LID (vespa/searchlib/src/vespa/searchlib/attribute/):
   • Because LIDs are compact 32-bit integers (uint32_t), every AttributeVector (single-value numeric, array/weightedset multivalue via MultiValueMapping, string enum store via EnumStoreT, or tensor via TensorAttribute) is indexed directly by lid as an $O(1)$ array lookup without hash-table indirection during query evaluation.
2. RCU Generation-Protected Readers (vespa/vespalib/src/vespa/vespalib/util/generationhandler.h):
   • A single writer thread per attribute mutates the active vector and increments the generation counter, while concurrent search threads hold read guards (GenerationHandler::Guard) so lock-free readers never observe freed memory buffers.
3. LID Space Compaction (vespa/searchcore/src/vespa/searchcore/proton/lidspace/):
   • When documents are removed or moved across ready/notready sub-databases, LidSpaceCompactionJob relocates high-numbered LIDs into lower free slots and shrinks committedDocIdLimit once old reader generations drain.

Boundary Conditions & Trade-offs

• Hard Per-SubDB Ceiling: Because uint32_t LIDs reserve high bits for internal flags in posting lists and bitvectors, a single Proton DocumentDB sub-database is bounded to $<2^{31}$ documents.
• Transient Memory Overhead: Uncompacted LID holes waste columnar slots across all scalar attributes until background LID compaction completes (see inv-lid-monotonic-generation-guard and search-ranking).