Cloudflare DNS Builds Cut Peak Memory 85% and Reduce Worst Delays 12-Fold
- Cargo
- Machine Learning
- Shipped
- Shipper
- Alex Fattouche
- Readers
- 542
Targeted DNS builds cut peak memory 85% and reduced the largest measured delay from 14.6 minutes to 1.21 minutes.

Cloudflare’s DNS pipeline now rebuilds only the records and shared metadata a change can affect. That shift matters most for very large zones: some contain more than 20 million records, and a full rebuild can consume too much memory to finish. In a 24-hour comparison, peak working-set memory fell 85%, while the largest measured build delay dropped from 14.6 minutes to 1.21 minutes.
The change addresses a mismatch between the pace of incoming events and the cost of processing them. The Zone Builder now receives roughly 700 DNS record changes per second, almost three times the rate reported in 2022, plus about 100 other events per second, including zone setting changes. Event volume alone wasn’t the bottleneck. A small number of events could trigger work proportional to the entire zone, making their cost grow as customers added records.
The new system combines a planner, which selects the smallest build it can prove correct, with a unified scheduler, which merges queued work for the same zone. Together, they avoid rebuilding large zones for many setting changes, reduce memory pressure, and let Cloudflare shorten some of the waiting periods built into the pipeline. That last step helped bring the median time to finish a build from about 7.7 seconds to about 3.3 seconds.
From a database change to DNS data
A build turns a database change into data that Cloudflare’s authoritative DNS servers can use. A customer might change a record through the dashboard or API; another Cloudflare service might change a zone setting. The database stores the update, and a system produces an event in Kafka. The DNS Kafka Aggregator combines events from several topics and passes them to the Zone Builder.
The builder reads the zone data needed for the task and converts it into keys used by the authoritative servers. It compares those keys with the existing entries in Quicksilver, Cloudflare’s distributed key-value store, then writes the changes. Quicksilver distributes them across Cloudflare’s network. The finished data isn’t a direct copy of database rows: the builder combines records with settings, applies DNS serving rules, and arranges the result for fast reads.
In 2022, Cloudflare optimized ordinary record changes. Those per-record builds were 150 times faster than full builds on average; on a test zone with one million records, they were 4,250 times faster. The next target was zone settings. The difficult part wasn’t simply skipping records. It was establishing which output could change, including data shared between zones with the same domain name, and proving that omitted work couldn’t leave stale answers at the edge.
Why settings need careful planning
A full build is straightforward: read the current settings and every record, convert the data into Quicksilver keys, compare the result with what’s already stored, then remove stale entries and write changed ones. Although the final writes include only differences, the builder still has to read, encode, and compare the whole zone. It also holds the zone’s data in memory during the process. That works for small zones, but some of the largest zones contain tens of millions of records and don’t fit in memory.
Settings complicate a targeted build because they can affect DNS output. Universal SSL provides one example. Enabling or disabling it changes zone metadata and can change Certification Authority Authorization records at the zone apex, the bare domain such as example.com. CAA records tell certificate authorities whether they may issue a certificate for a domain.
To plan less work safely, the builder has to determine which DNS names can change, whether shared metadata or the set of published zones may change, and what old Quicksilver state must be removed. Some events make those answers clear. Others don’t identify the reason for a change. A timestamp update, for example, may be routine bookkeeping or a sign that data elsewhere changed and needs to be read again. When the event lacks enough detail, the planner still requests a full build. Uncertainty never becomes a reason to do less work.
Five build choices
Quicksilver holds record keys for individual zone names and a smaller shared metadata key. Authoritative servers use the metadata to locate special information, including nameservers and wildcards. The planner classifies work into five types:
- Resources only: Rebuild records at specific DNS names without changing zone metadata. Ordinary A, AAAA, TXT, and CNAME updates usually fit here.
- Resources and metadata: Rebuild specified names and update shared metadata. Nameserver and wildcard changes take this route.
- Apex and metadata: Rebuild the zone apex and shared metadata. Universal SSL changes can use this type.
- Purge: Remove a zone that should no longer be published.
- Full zone: Rebuild all records when the planner can’t establish that a smaller task is safe.
The correctness test is strict: every key written by a targeted build must match the key a full build would produce, byte for byte. If the planner can’t establish that equivalence, it chooses a full build. For record updates and deletions, events carry the old record or indicate its deletion. The builder scans the existing keys for each name being rebuilt and removes entries the new build no longer produces.
Full builds remain necessary when a change might affect records across the zone or when the zone is being published for the first time. There is also a practical limit for the largest zones: above a fixed record-count threshold, the builder won’t escalate to a full build. It instead rebuilds the apex, metadata, and names identified by queued events. A complete rebuild of those zones can fail to finish while holding up other work on the same Kafka partition.
That limit carries a tradeoff. A few unusual cases may need additional repair, such as publishing a zone on another network. Cloudflare says it can manually rebuild individual names for those cases. Zone holds, which prevent the same name from being added to another account, reduce the chance of several same-name zones creating the need for a full rebuild.
Shared metadata across same-name zones
Each zone has its own record keys, but all Cloudflare zones with the same domain name share one metadata key. That key acts as a directory, telling the authoritative servers which internal zone IDs they can use to answer a query. During customer onboarding, offboarding, or other lifecycle changes, several internal zone objects may briefly exist for one public domain. A priority system selects a small, bounded set for publication.
For example, suppose four internal zones exist for example.com, numbered 101, 202, 303, and 404. The shared key might list 101, 202, and 303 while leaving 404 out. A TXT change at www.example.com in Zone 202 changes that zone’s record keys, but not the directory. A nameserver or wildcard change affects information represented in the shared key, so the build must update both the zone records and the directory.
The builder reads the current candidate list when it executes a metadata build. It reconstructs the shared key from that fresh view rather than patching an older version. It also writes the triggering zone’s records before publishing metadata that points to them.
Checking fresh state before writing
Events describe past changes, while a build may sit in a queue as the database continues to change. For that reason, Cloudflare plans twice. The event first creates a conservative work list. Immediately before execution, the builder reads a fresh publication snapshot: current settings, whether that deployment should publish the zone to its network, and which same-name zone IDs belong in shared metadata.
Cloudflare runs separate, isolated Quicksilver environments for different networks. A zone can be enabled in one and disabled in another; the global network and China Network are given as examples. The fresh snapshot can therefore change the planned operation. A targeted build may need to become a full build if a newly observed feature requires one. A zone that should no longer be published becomes a purge. Work may be skipped if the triggering zone no longer belongs in the shared directory, and metadata can be rebuilt using the current candidate set.
The snapshot comes from read-only database replicas, so builds don’t compete with customer writes. Kafka delivery and database replication progress independently; an event can arrive before its database change appears on a replica. The aggregator waits to cover normal replication lag, and a final database check rejects a read when the replica is still behind.
One scheduler for all build types
Previously, record builds and full-zone builds used separate schedulers. Each could combine similar work internally, but couldn’t merge a record task with a full-zone task. The unified scheduler separates the decisions: the planner defines what the build must do, and the scheduler determines when it runs.
If several events for one zone arrive before execution, the scheduler combines their requirements. An A record update at www.example.com adds that name to the work list. A wildcard change at *.example.com adds the wildcard name and a metadata update. The resulting build handles both names and the shared metadata in one run.
Merging only adds work. If any event calls for a full-zone build, the combined task becomes one. A purge overrides rebuilds because the required result is that the zone is no longer published. The scheduler tracks the DNS work and the Kafka messages awaiting acknowledgment separately. Retries preserve both, including message references that aren’t individually needed after work has become a full build or purge.
Keeping the event stream safe
Kafka orders messages within partitions and records a committed offset: the point before which every message is known to be complete. Advancing that point beyond unfinished work could lose a customer change if the builder restarted. Cloudflare uses soft commits, allowing later builds to finish out of order while holding back the committed position until every earlier message is complete.
If message 2 is still running, messages 3 through the end of the completed sequence can’t be committed. A restart clears completion marks held only in memory, so Kafka replays those messages. That can repeat work, but it avoids losing a change.
Shorter builds help the oldest unfinished message complete sooner. The scheduler also limits how often one zone can rebuild, with an interval that varies by Cloudflare plan. Events arriving during that interval are combined into the next build. To keep old queued work from holding up a partition, the scheduler can let it run before the usual interval has elapsed. Zone locks and retry delays remain in place to prevent failing work from running repeatedly without pause.
Verifying equivalence before rollout
Cloudflare added tests for nameserver, wildcard, and apex builds. Each compares a targeted build with a full build from the same database state, fixes time-dependent values, and checks every Quicksilver key the targeted plan owns byte for byte. Separate planner and scheduler tests cover event classification, same-name zones, merging, purges, and escalation.
During the initial rollout, Cloudflare also ran a full build beside each targeted build and compared the results. That method was deliberately expensive because it read every record, so it served as a temporary check. A mirror Zone Builder then provided a longer-running comparison against production conditions. It consumed the production Kafka stream and read production database state, but wrote to a separate Quicksilver environment that served no customer traffic.
A comparer checked a 20% sample of zones for missing keys, extra keys, and changed values. Because the mirror and production builders might process the same zone at different times, a temporary mismatch didn’t necessarily signal a defect. The team looked for new kinds of differences that persisted after both pipelines caught up. A dashboard built with a Worker, Durable Objects, and Cloudflare Access grouped the differences and retained examples for investigation.
After the mirror results remained healthy, Cloudflare rolled out the scheduler by hashing zone names. Zones with the same name therefore stayed on the same scheduler, preventing two schedulers from writing the same shared metadata key. The rollout increased its share of free zones first, then repeated the process for paid zones.
Measured changes in work, memory, and delay
Across 24 hours, 86.81% of builds were resources-only, 2.54% handled resources and metadata, 2.30% handled the apex and metadata, and 1.30% were purges. In total, 92.95% avoided rebuilding a complete zone. Full builds accounted for 7.04% of the reported builds.
The two metadata build types had previously required full builds, even for large zones. Together, they represented 4.84% of builds. Adding that share to the 7.04% that still required full builds gives 11.88%, about a 40% reduction in full-build share. The percentage understates the work avoided because the eliminated builds included some of the largest zones. Remaining full builds averaged 0.35 seconds; events that still trigger them, such as first publication, generally involve new zones with few records.
The unified scheduler also made a batching system planned in 2022 unnecessary. It doesn’t wait to assemble batches; it merges events only when work for the same zone is already queued. With fewer long full builds holding back Kafka’s committed position, the builder consumed events at roughly their arrival rate instead of falling behind and catching up in bursts of up to 1,550 messages per second.
Build completion delay measures the time from a record or setting change until the Zone Builder finishes writing to Quicksilver. It excludes Quicksilver’s subsequent distribution across the network, which varies by location and wasn’t affected by this work. Production, using the old schedulers, and the mirror, using the unified scheduler, processed the same events over the same 24 hours and read from the same database. The results combine build types and customer plans, so they don’t promise a particular delay for any one zone. The rebuild interval accounts for much of the measured time; the build work itself is only part of the total.
Typical delays changed little, since most builds were already small. The largest p99 spike fell from 14.6 minutes to 1.21 minutes, a 12-fold reduction. Here p99 means that 99% of events completed within the measured time at a given point.
Memory use also fell sharply. Average aggregate working-set memory, the memory actively used by the deployment, went from 49 GiB to 7 GiB, a reduction of about 86%. Peak use fell from 79 GiB to 12 GiB, or 85%.
That extra headroom allowed Cloudflare to adjust two safeguards. First, it shortened the per-zone rebuild interval on most plans because builds cost less and far fewer of them read entire zones. Second, it reduced the time the DNS Kafka Aggregator holds events while database replicas catch up. Database checks that reject reads from lagging replicas made a smaller buffer practical. Together, the changes brought median completion delay down from about 7.7 seconds to about 3.3 seconds.
The percentile estimates in the original delay chart were less precise than their plotted lines suggested. Its histogram used buckets at 1, 5, 10, and 30 seconds; percentiles between bucket edges were estimated as if delays were evenly spread through the interval. Since most delays cluster near the bottom of each bucket, that method pushed estimates upward, placing p99 close to 30 seconds and p90 above 7 seconds. The exact average, which doesn’t rely on bucket estimates, confirmed the reduction. Cloudflare has since adopted native histograms, which choose finer buckets automatically; those put p90 at about 5.5 seconds and p99 at about 12 seconds.
Why full builds still matter
Eliminating full builds entirely was once the stated goal. The current design recognizes that they remain useful: first publication needs every record written, changes with zone-wide effects need every record checked, and ambiguous events still require conservative handling. Full builds are also inexpensive for small zones.
Cloudflare expects to classify more event types for targeted work as it learns which DNS output they affect. The largest gain is already in place: settings that used to trigger rebuilds of the biggest zones no longer do. The full builds that remain mostly come from events affecting new, small zones, while targeted plans keep large zones from paying the cost of work they don’t need.