Summary
Vespa config servers keep authoritative control-plane state (tenants, applications, flags, generation counters, locks, barriers) inside a ZooKeeper ensemble that Vespa itself packages, starts, and reconfigures. The zkfacade module wraps Apache Curator behind a small Vespa-specific client API with a pinned ABI; the zookeeper-server module runs the actual ZK server under vespa-home. Concrete examples show the split: small JSON control state lives in ZK nodes, while bulky payloads (file reference data, ZK’s own dataDir and myid) stay on local disk. Related context: config-admin.
Key mechanisms
- Facade: zkfacade/README is “Vespa ZooKeeper interface”. The Curator component reads its client config from var/zookeeper/conf/zookeeper-client.cfg (zkfacade/src/main/java/com/yahoo/vespa/curator/Curator.java:65), defaults to a 120-second session timeout (line 68), and its injected constructor takes VespaZooKeeperServer explicitly so the ZK server is started before clients (line 97).
- Boundary API: VespaCurator (zkfacade/src/main/java/com/yahoo/vespa/curator/api/VespaCurator.java:20) exposes stat / read / write / delete / list, a compare-and-set write taking an expectedVersion (lines 32-40), a re-entrant lock returning AutoCloseable (line 58), and SingletonWorker register / unregister / isActive for leader-style activation (lines 69-77). zkfacade/abi-spec.json pins this surface, including records VespaCuratorMeta(version), as the stable module ABI; a MockCurator exists under zkfacade/src/main/java/com/yahoo/vespa/curator/mock/.
- Server packaging: zookeeper-server/README explains each ZK version lives in its own subdirectory and the runtime version is selected with feature flag ZOOKEEPER_SERVER_VERSION. Configurator generates the server config under vespa-home, including dataDir (Configurator.java:98) and the myidFile (line 157). Reconfigurer starts the server and supports runtime ensemble reconfiguration (Reconfigurer.java:27).
- State in ZK, anchored in configserver: tenants live under /config/v2/tenants/ with sibling roots /config/v2/locks/, /config/v2/barriers/, and /vespa, all created at startup (configserver/src/main/java/com/yahoo/vespa/config/server/tenant/TenantRepository.java:92-96, 292-296); a global supermodel generation counter is the node /config/v2/RPC/superModelGeneration (SuperModelGenerationCounter.java:16). Feature-flag values are UTF-8 JSON nodes under /flags/v1/
, kept warm with a Curator directory cache (configserver-flags/src/main/java/com/yahoo/vespa/configserver/flags/db/FlagsDbImpl.java:25-35, 47-49). - Disk versus ZK split: ConfigServerFlagSource chains OrderedFlagSource(BootstrapFlagSource, ZooKeeperFlagSource); the bootstrap source reads a local flag file once at startup for emergencies and takes precedence over ZK when it defines a value (configserver-flags/src/main/java/com/yahoo/vespa/configserver/flags/ConfigServerFlagSource.java:24-27, db/BootstrapFlagSource.java:22-31). File payloads used by file distribution are not in ZK; the config server’s FileDirectory keeps them on disk under configserverConfig.fileReferencesDir() (configserver/src/main/java/com/yahoo/vespa/config/server/filedistribution/FileDirectory.java:49-61), feeding the protocol in GDN-NOTE-0038.
Open questions
- Are uploaded application packages (sessions) fully ZK-resident or hybrid with disk? SessionRepository and ZooKeeperDeployer in configserver were not read in this pass.
- Contents of zkfacade/leakage-spec.json unread.
- Whether MockCurator replicates CAS version semantics closely enough for tests that depend on expectedVersion ordering.