Zettel - Storage plane: persistence SPI, storaged glue, vdslib vocabulary, and visitors

Summary

Vespa’s storage plane is one binary, storaged, playing two roles decided by config: distributor or storage (service layer) node. The persistence module owns only the contract: storage::spi::PersistenceProvider, the seam between the storage layer and whatever actually stores documents. The storageserver module owns only the process glue that turns StorServerConfig into a DistributorProcess or a ServiceLayerProcess; the real storage logic (bucket databases, the visiting runtime) lives in the adjacent storage framework module, and concrete backends implement the SPI elsewhere — proton’s PersistenceEngine extends storage::spi::AbstractPersistenceProvider and storage::spi::BucketExecutor. vdslib is the shared vocabulary module: Java-side Distribution, ClusterState, NodeState, and VisitorStatistics, plus small C++ containers like vdslib::Parameters. Visitors are the bulk-read plane: dynamically loaded plugins that stream a bucket’s serialized documents through plugin logic. This note is the node-local counterpart to messagebus-bucket-distribution-and-ideal-state (“Zettel - MessageBus Routing, BucketId Prefix Splitting, and Ideal State Convergence”), which covers how requests find buckets and how replicas converge; here we cover what runs inside the node that receives them.

Layer map (storage vs proton/searchcore boundary)

• persistence/: interface only. README: “the persistence SPI and a dummy implementation” (dummyimpl plus a conformancetest). The SPI covers bucket-scoped writes (putAsync, updateAsync, removeAsync, removeIfFoundAsync), reads (get, createIterator/iterate/destroyIterator), bucket admin (listBuckets, getBucketInfo, getModifiedBuckets, split, join), and cluster-state push (setClusterState). Write timestamps are generated by the distributor, so ordering lives above the SPI.
• storageserver/: StorageApp::createProcess reads StorServerConfig: if isDistributor it builds a DistributorProcess (holding a DistributorNode plus VisitorDispatcherConfig); otherwise a service layer process (DummyServiceLayerProcess for provider types STORAGE/DUMMY). ServiceLayerProcess holds the ServiceLayerNode, a VisitorFactory::Map of external visitors, and setupProvider()/getProvider() returning the spi::PersistenceProvider.
• searchcore (proton): implements the SPI per node. Boundary: proton owns the per-node searchable structures (indexes, attributes, document store); the storage layer above owns placement, redundancy, and cluster-state reactions. Neither crosses: storage code never touches proton internals; proton never does placement. Nodes of both kinds form content-nodes.
• vdslib/: Java Distribution subscribes to StorDistributionConfig/DistributionConfig and exposes getRootGroup()/getRedundancy(); state/ models cluster and node state; VisitorStatistics counts buckets, documents, and bytes visited vs returned. It is the vocabulary that config-admin-delivered distribution configs and the C++ storage side agree on.

Visitor framework

storage::Visitor self-describes as “a piece of code existing in a shared library linked in, that iterates serialized documents from the persistence layer”. The base class drives per-bucket SPI iteration (BucketIterationState), feeds HitCounter::updateVisitorStatistics(vdslib::VisitorStatistics), and returns data via VisitorMessageHandler. VisitorThread runs each visitor single-threaded and keeps a LibMap of loaded visitor libraries; sibling files provide count, dump, recovery, and reindexing visitors. streamingvisitors contributes streaming::SearchVisitor : public storage::Visitor: it builds a streaming query, matches via VSM, ranks via RankManager/RankProcessor, produces docsums, and returns a documentapi::QueryResultMessage. Per SearchVisitorProtocol.html, the client sends createVisitor with VisitorLibrary: "SearchVisitor" (the default is a dump visitor), and “the distributors in VDS do not have dispatch capability” — the QRS client merges partial results. Search, recovery, reindexing, count, and dump all ride the same SPI iteration contract.

Open questions

• What does VisitorDispatcherConfig actually make the distributor do — split bucket sets across nodes for one visit?
• Who pushes document selection down into SPI Selection: service layer chains or the visitor itself?
• Are vdslib’s Java SearchResult/DocumentSummary legacy, or still on the visitor reply path?
• How do SPI remove entries interact with visits that include removes (tombstone visibility)?
• Does proton’s BucketExecutor registration alter visit/merge concurrency guarantees?