Quartz 5

Invariant — Proton LID Reuse Requires Monotonic RCU Generation Drain

Properties23
doc_idGDN-CLAIM-0001
sluginv-lid-monotonic-generation-guard
titleInvariant — Proton LID Reuse Requires Monotonic RCU Generation Drain
roleinvariant
lifecycle_stateactive
maturitygrowing
verificationmachine-checked
confidencehigh
evidence_strengthdirect
dispositionadopt
seductive_wrong_takeawayAssuming a removed document's 32-bit LID can be immediately recycled for a newly fed document as soon as LidGidMap unmaps the old GlobalId.
falsificationInspect vespa/searchcore/src/vespa/searchcore/proton/lidspace/ and vespa/vespalib/src/vespa/vespalib/util/generationhandler.h: any immediate LID reuse while active search threads hold an older GenerationHandler::Guard causes concurrent queries to read stale attribute slots or mismatched posting entries.
do_not_import_boundaryDo not import immediate slot-reuse assumptions from mutex-locked B-trees into lock-free RCU AttributeVector or HNSW graph readers in vespa/searchlib and vespa/searchcore.
sourcesvespa/searchcore, vespa/searchlib, vespa/vespalib
indexed-byGDN-NOTE-0001, GDN-NOTE-0003
relatedGDN-NOTE-0003, GDN-NOTE-0012, GDN-EVID-0001
supported-byGDN-NOTE-0012, GDN-EVID-0001
created2026-10-27
updated2026-10-27
producersgardener@wave9
owner_loopgardener
tagsinvariant, proton, lid, rcu, generation-handler, searchcore
modified2026-10-27

1 min read

Invariant — Proton LID Reuse Requires Monotonic RCU Generation Drain

In vespa/searchcore (proton), a freed 32-bit Local Document ID (LID) in ReferenceLidTracker / LidAllocator must never be reassigned to a new document until every concurrent search reader holding a GenerationHandler::Guard from the removal generation or earlier has completed (first_used_generation > removal_generation).

Why It Holds (Chesterton Fence)

Search threads in proton read AttributeVector arrays and HNSW link lists without per-lookup locks (O(1) direct array indexing by LID; see proton-lid-space-and-attribute-vectors). If a removed LID were immediately reused while an in-flight query still held a posting list containing that LID, the query would silently score and return attributes belonging to the newly inserted document.

Seductive Wrong Takeaway (CRIT-001)

  • Misconception: remove(docId) immediately frees the integer LID for the next put(docId).
  • Falsification: LidAllocator::unregister_lid places removed LIDs onto a generation-tagged hold list (_holdLids) drained only after generationHandler.update_oldest_used_generation() advances past the hold generation.
  • Do-Not-Import Boundary: Never bypass GenerationHandler hold queues when adding custom columnar or graph structures in vespa/searchlib.

Graph View

Backlinks