Quartz 5

Runtime feature flags: vespa/flags module, FlagDbFile on-disk format, the /flags/v1 ZK store, and model-build-time delivery to nodes

Properties18
doc_idGDN-NOTE-0042
slugzettel-flags-runtime-feature-flags
titleRuntime feature flags: vespa/flags module, FlagDbFile on-disk format, the /flags/v1 ZK store, and model-build-time delivery to nodes
rolezettel
lifecycle_statedraft
maturityseedling
verificationunverified
confidencemedium
evidence_strengthdirect
dispositiondefer-until-data
sourcesvespa/flags/src/main/java/com/yahoo/vespa/flags/file/FlagDbFile.java, vespa/flags/src/main/java/com/yahoo/vespa/flags/json/FlagData.java, vespa/flags/src/main/java/com/yahoo/vespa/flags/json/wire/WireFlagData.java, vespa/flags/src/main/java/com/yahoo/vespa/flags/json/wire/WireRule.java, vespa/flags/src/main/java/com/yahoo/vespa/flags/PermanentFlags.java, vespa/configserver-flags/src/main/java/com/yahoo/vespa/configserver/flags/db/FlagsDbImpl.java, vespa/configserver-flags/src/main/java/com/yahoo/vespa/configserver/flags/db/BootstrapFlagSource.java, vespa/configserver-flags/src/main/java/com/yahoo/vespa/configserver/flags/db/ZooKeeperFlagSource.java, vespa/configserver-flags/src/main/java/com/yahoo/vespa/configserver/flags/ConfigServerFlagSource.java, vespa/configserver-flags/src/main/java/com/yahoo/vespa/configserver/flags/http/FlagsHandler.java, vespa/configserver-flags/src/main/java/com/yahoo/vespa/configserver/flags/http/DefinedFlags.java, vespa/configserver/src/main/java/com/yahoo/vespa/config/server/deploy/ModelContextImpl.java, vespa/configserver/src/main/java/com/yahoo/vespa/config/server/application/TenantApplications.java
indexed-byGDN-NOTE-0004
relatedGDN-NOTE-0004
created2026-10-28
producersscout@wave-d-depth
owner_loopgardener
tagszettel, config, flags, feature-flags, zookeeper, json, vespa
modified2026-10-28

3 min read

Summary

vespa/flags is a standalone library that defines runtime feature flags as id + typed value + targeting rules, with JSON as the storage format. On the config server, authoritative values live in ZooKeeper under /flags/v1/ (the layout already documented in GDN-NOTE-0037), and an emergency local file db at var/vespa/flag.db takes precedence over ZK. Crucially, flags are not fetched by running containers from /flags/v1: the config server resolves flag values per host, cluster, and application during model building, and the resolved values are baked into the generated config payloads that nodes receive through their normal config subscriptions (generation semantics in GDN-NOTE-0041). The /flags/v1 HTTP API on the config server is an operator tooling surface, not the node delivery path.

Key mechanisms

  • On-disk db: FlagDbFile implements both FlagRepository and FlagSource; the no-arg constructor resolves the path to Defaults.getDefaults().underVespaHome(“var/vespa/flag.db”) (FlagDbFile.java:44-50). read() deserializes the whole file into a Map of FlagData (FlagDbFile.java:65-69); sync(Map) diffs against current contents by comparing serializeToJson() strings, logs new/updated/removed flags, and rewrites the entire file via FlagData.serializeListToUtf8Json when anything changed (FlagDbFile.java:71-103).
  • JSON format: the wire format is Jackson-mapped WireFlagData with fields id (string), rules (list of WireRule), and attributes (map of dimension name to value, the default fetch vector) (WireFlagData.java:22-25); each WireRule has conditions (AND-ed WireCondition list) and value (raw JSON) (WireRule.java:16-18). FlagData.toWire fills id, rules, and defaultFetchVector (FlagData.java:119-130), and FlagData.deserializeUtf8Json is the reverse (FlagData.java:172). Resolution picks the first rule whose conditions all match defaultFetchVector.with(fetchVector) (FlagData.java:95-100); condition types live in the json package (RelationalCondition, WhitelistCondition, BlacklistCondition, ListCondition, per directory listing).
  • ZK store: FlagsDbImpl creates /flags/v1 at startup and keeps one UTF-8 JSON FlagData node per flag id, read through a Curator DirectoryCache; setValue writes data.serializeToUtf8Json() to /flags/v1/ (FlagsDbImpl.java:25-49, 65-67) - same nodes GDN-NOTE-0037 recorded.
  • Precedence chain: ConfigServerFlagSource chains OrderedFlagSource(BootstrapFlagSource, ZooKeeperFlagSource) (ConfigServerFlagSource.java:24-27). BootstrapFlagSource reads var/vespa/flag.db exactly once at construction, documented as the emergency escape hatch, and should take precedence over ZK whenever it defines a value (BootstrapFlagSource.java:15-35).
  • Operator API: FlagsHandler handles GET/PUT/DELETE on /flags/v1 (FlagsHandler.java:24-43), including a listing at /flags/v1 with “data” and “defined” resources (FlagsHandler.java:56-58); DefinedFlags renders flag definitions sorted by id, using FlagDefinition.getUnboundFlag().id() (DefinedFlags.java:31-38).
  • Delivery to nodes: there is no container-side FlagSource implementation outside the flags and configserver-flags modules (verified by repo-wide search). Instead ModelContextImpl.FeatureFlag binds flags server-side with dimensions application id and vespa version, then narrows per cluster type, cluster id, and hostname (ModelContextImpl.java:170-199), and ModelContextImpl.FeatureFlags exposes typed values such as heapSizePercentage, queryDispatchWarmup, feedConcurrency, and responseSequencerType (lines 200-222). PermanentFlags (config-server-only flags from ConfigserverConfig, PermanentFlags.java:30) are bound the same way, e.g. PermanentFlags.INCOMPATIBLE_VERSIONS in TenantApplications (TenantApplications.java:102). container-jdisc components therefore see flags only as ordinary config.

Open questions

  • Propagation latency: a FlagsDbImpl write changes ZK state, but the trigger that causes model rebuilding (and hence new flag-derived config for nodes) was not traced from FlagsDbImpl to SessionRepository/deploy.
  • Which component wires ConfigServerFlagSource and FlagsHandler into the config server’s component graph: only pom.xml/CMakeLists dependencies of configserver on configserver-flags were confirmed.
  • Whether anything besides BootstrapFlagSource ever writes var/vespa/flag.db (no writer found in-tree).

Graph View