Roll out configuration changes as carefully as code

Why configuration escapes the deployment pipeline

Give every configuration change the delivery path a binary already gets. A feature flag, a policy row, a generated data file or a customer's routing rule should reach a small first stage, wait there longer than the slowest way it can fail, and roll back on its own to a version it was never allowed to overwrite. Then assume a bad version will still get through, and make every consumer keep serving when the new version is unreadable. Each of the seven incident reports reviewed here shows at least one of those two halves missing.

Configuration escapes because it rarely travels on the release pipeline. Binaries go through canaries and waves because everyone agrees they are code; configuration usually rides a separate system built to converge everywhere quickly. Cloudflare says its Quicksilver system carries a new DNS record or security rule to 90 percent of its servers within seconds. In December 2025 it wrote that its binaries must pass several gates, starting with employee traffic and widening through growing shares of customers with automatic revert, and that it had not applied that method to configuration. ^[1]

Google's SRE workbook reports that across thousands of postmortems written from 2010 to 2017, configuration pushes triggered 31 percent of outages, second only to binary pushes at 37 percent. ^[2] That sample is one company's and is dated, but the public reports from July 2024 to July 2026 show the same pattern: in none of the seven incidents below was the triggering change a binary release.

The controls that follow each trace to one of those reports: a staged path per artifact class, validation in the real consumer, stage waits longer than the slowest asynchronous failure, a last known good the change cannot overwrite, aggregate blast-radius checks, kill switches tested against the change they act on, and a stated failure mode in every consumer. The illustration shows what the first control buys. The same bad change either reaches every region before anyone reads its health, or stops at the first stage while the gate is still closed.

Source. Conceptual illustration based on the Azure Front Door PIR YKYN-BWZ and the Cloudflare December 5, 2025 and Code Orange reports. ^[7][9][1]

Method. Conceptual hand-authored illustration. Region counts and spacing are schematic and do not represent measured timing.

Accessible table and figure data

Figure 1 accessible table

Element	What it represents
Change block	The person or job that writes the configuration
Top lane	Global path: every region receives the change together
Bottom lane	Staged path: only the first stage receives it
Amber square with a cross	A region or cell running the bad change, crashing minutes later
Mint square	A region or cell still on the last known good
Gate bar	Bake time and stage-scoped health check
Dashed line	The path the change takes only after the gate opens

Figure 1 accessible table

Element	What it represents
Change block	The person or job that writes the configuration
Top lane	Global path: every region receives the change together
Bottom lane	Staged path: only the first stage receives it
Amber square with a cross	A region or cell running the bad change, crashing minutes later
Mint square	A region or cell still on the last known good
Gate bar	Bake time and stage-scoped health check
Dashed line	The path the change takes only after the gate opens

Download imageDownload table CSV

Seven incident reports, read for the delivery path

Each report is read here for one question: how the change travelled from whoever made it to the fleet that ran it. Shared dependencies are covered separately, so the retelling stops at the delivery mechanism. All seven are vendor self-reports, including every remediation described as complete.

CrowdStrike released a Rapid Response Content update for its Windows sensor at 04:09 UTC on July 19, 2024 and reverted it at 05:27 UTC, a 78-minute window in which any online Windows host on sensor 7.11 or later that received the update was exposed. ^[3] The root cause analysis describes this content as configuration data rather than code or a kernel driver, and its sixth finding is simply that each Template Instance should be deployed in a staged rollout. ^[4]

Google's Service Control binary had rolled out region by region from May 29, 2025, but its new code path ran only when a matching policy existed. That policy row, inserted at about 10:45 PDT on June 12, replicated to every region within seconds and crashed the binary everywhere at once. The code was staged; the data that activated it was not. ^[5]

Azure Front Door did stage customer configuration, through health-gated stages that also refreshed a last known good (LKG) snapshot. That protection system had caught bad tenant metadata at an early stage, and on October 9, 2025 engineers bypassed it to run a cleanup. ^[6][13] On October 29 a different metadata defect crashed the data plane about five minutes after reaching pre-production, while stages advanced about a minute apart, so the change reached most edge sites by 15:39 UTC and became the new LKG. ^[7]

On November 18, 2025 a Bot Management feature file, rebuilt every five minutes by a database query, doubled in size after a permissions change and was published to the whole network on each rebuild. ^[8] On December 5 a WAF buffer increase was moving through Cloudflare's gradual deployment system when a second change, switching off an internal test tool, went out through the global configuration system, which does not roll out gradually. ^[9]

The July 23, 2026 Azure West US incident involved no configuration file. Break-fix automation, widened by a defective blast-radius analysis, withdrew routes from every optical device leaving one datacenter, and the safety check that should have stopped it judged each device on its own. It belongs in this set because a configuration pipeline makes the same check whenever it decides a change is small. ^[10]

Read together, the reports show a narrower pattern than carelessness. In all seven some safeguard existed, and the change either took a path it did not cover or moved past it faster than it could observe a failure. That is an editorial reading, not a finding of any one report, but it says where to start: inventory the paths each artifact takes, then measure how long its failures take to appear.

Source. Times as stated in each provider's report, reviewed October 9, 2026: CrowdStrike PIR and RCA, Google Cloud incident report, Azure PIRs QNBQ-5W8, YKYN-BWZ and ZJV6-SGG, and Cloudflare's November 18 and December 5, 2025 reports. ^{[3][4][5][6][7][8][9][10]}

Method. Source-derived. Clock times are copied from the reports, in UTC except Google, which reports US/Pacific. The 78 minute CrowdStrike window is calculated as 05:27 minus 04:09 UTC. Cloudflare's November 18 report gives 11:20 UTC in its summary and 11:28 in its timeline table; both are shown.

Accessible table and figure data

Figure 2 accessible table

Incident	Change	How far it got	Stopped	Recovered
CrowdStrike, July 19, 2024	Channel File 291 released 04:09 UTC	Online Windows sensors 7.11 and later that received it	Reverted 05:27 UTC, a 78 minute window	About 99% of Windows sensors online by July 29
Google Cloud, June 12, 2025	Policy row inserted about 10:45 PDT	Every region within seconds	Red-button rollout complete within 40 minutes	us-central1 up to about 2 h 40 min
Azure Front Door, October 9, 2025	Cleanup bypassing protection from 07:30 UTC	About 26% of the data plane in Europe and Africa	Restarts from 09:08; availability back 12:50 UTC	Mitigated 16:00 UTC
Azure Front Door, October 29, 2025	Metadata change 15:35 UTC	Majority of edge sites by 15:39; written to the LKG	Propagation halted 15:43; edited LKG deploying from 17:40	Mitigated 00:05 UTC, October 30
Cloudflare, November 18, 2025	Permissions change 11:05 UTC; file rebuilt every 5 minutes	Whole network; errors from 11:20 (summary) or 11:28 (timeline)	New files stopped 14:24; good file global 14:30	All services 17:06 UTC
Cloudflare, December 5, 2025	Global flag change 08:47 UTC	Fully propagated 08:48; about 28% of HTTP traffic	Reverted 09:11 UTC	Revert propagated 09:12 UTC
Azure West US, July 23, 2026	Break-fix route withdrawals 14:44 UTC	Traffic in and out of the region	Manual rollback 17:45 to 18:26 UTC	Services recovered 19:41 UTC

Figure 2 accessible table

Incident	Change	How far it got	Stopped	Recovered
CrowdStrike, July 19, 2024	Channel File 291 released 04:09 UTC	Online Windows sensors 7.11 and later that received it	Reverted 05:27 UTC, a 78 minute window	About 99% of Windows sensors online by July 29
Google Cloud, June 12, 2025	Policy row inserted about 10:45 PDT	Every region within seconds	Red-button rollout complete within 40 minutes	us-central1 up to about 2 h 40 min
Azure Front Door, October 9, 2025	Cleanup bypassing protection from 07:30 UTC	About 26% of the data plane in Europe and Africa	Restarts from 09:08; availability back 12:50 UTC	Mitigated 16:00 UTC
Azure Front Door, October 29, 2025	Metadata change 15:35 UTC	Majority of edge sites by 15:39; written to the LKG	Propagation halted 15:43; edited LKG deploying from 17:40	Mitigated 00:05 UTC, October 30
Cloudflare, November 18, 2025	Permissions change 11:05 UTC; file rebuilt every 5 minutes	Whole network; errors from 11:20 (summary) or 11:28 (timeline)	New files stopped 14:24; good file global 14:30	All services 17:06 UTC
Cloudflare, December 5, 2025	Global flag change 08:47 UTC	Fully propagated 08:48; about 28% of HTTP traffic	Reverted 09:11 UTC	Revert propagated 09:12 UTC
Azure West US, July 23, 2026	Break-fix route withdrawals 14:44 UTC	Traffic in and out of the region	Manual rollback 17:45 to 18:26 UTC	Services recovered 19:41 UTC

Download imageDownload table CSV

Treat generated data as configuration

Start by naming the artifact classes, because each needs its own speed. Microsoft's description of Azure Front Door is a useful template. Service code moves through safe deployment practice over about two to three weeks. WAF and layer 7 DDoS platform data, such as GeoIP tables, attack signatures and IP reputation lists, changes daily and passes several stages with a bake of more than 12 hours. Customer configuration, a route or a WAF rule, is expected across the CDN industry to reach every edge site in five to ten minutes, and both October outages came from that class. ^[13] Three classes, three speeds, one fleet.

Most platforms carry at least five classes worth separating: feature flags; policy and quota records; generated data files such as model features, allow lists and routing maps; security content such as WAF rules and detection logic; and customer-authored configuration, plus plans produced by automation like the Azure break-fix request. For each, record its producer, how fast it legitimately has to move, and every consumer that parses it, with the version each runs.

Generated files deserve the hardest look because no person reviews the instance that ships. Cloudflare's feature file was rebuilt every five minutes by a ClickHouse query. The permissions change that broke it was itself rolling out gradually across the database cluster, and because only updated nodes produced bad output, the network flipped between good and bad files until every node had been updated. ^[8] The producer change was staged; its output was not. It follows that staging the job that writes a file does nothing for the file, which needs its own version, first stage and rollback.

Cloudflare's first listed remediation was to harden ingestion of files it generates itself in the same way it treats user input. ^[8] Its May 2026 completion report describes Snapstone, a system that packages any unit of configuration, whether a data file like the November one or a control flag like the December one, and releases it progressively with health monitoring and automated rollback by default. ^[11] The useful property is the default: a risky configuration pattern inherits the staged path as soon as a team moves it in.

CrowdStrike's line between content and code matters for process, not for blast radius. Its RCA calls Rapid Response Content configuration data, yet that content decided which comparison the sensor's Content Interpreter performed, and one new instance was enough to make it read past the end of its input array. ^[4] When a data file can select a code path, it carries a code path's risk and deserves a code path's rollout.

Validate against the consumer, not only the specification

CrowdStrike's validator did its job against the wrong contract. The IPC Template Type defined 21 input fields, but the sensor code that invoked the interpreter supplied 20. Tests and the first production instances used a wildcard for the 21st field, so nothing ever read it. On July 19 a new instance put a real matching criterion in that field. The Content Validator approved it, because it expected 21 inputs, and sensors performed an out-of-bounds read. ^[4]

The mitigations show the layers a validator needs. A Sensor Content Compiler patch, in production on July 27, 2024, checks the field count at build time. The Content Interpreter gained a runtime bounds check on July 25, backported to sensor 7.11 and later through a hotfix due by August 9. A validator check due in production by August 19 rejects matching criteria that cover more fields than the interpreter receives, and every new Template Instance is now tested, not only the first one used to stress-test a Template Type. ^[4] That last change is the one most configuration pipelines lack: loading each new artifact in the real consumer before release.

Cloudflare's limit was explicit and still missed. The proxy preallocated memory for at most 200 Bot Management features, against about 60 in normal use, and the duplicated rows pushed the file past 200. The newer FL2 proxy panicked and returned errors, while the older FL proxy kept running but gave every request a bot score of zero, so customers who blocked bots saw false positives. ^[8] One file, two consumer versions, two different failures: validate against every consumer version in production, not one reference build.

Azure Front Door moved validation into the consumer itself. Its incompatible metadata came from changes made across two control plane builds, so Microsoft committed to cross-version validation and to fuzzing the metadata contract between the control plane and the data plane, both by February 2026. It also added a process on every data plane server, called Food Taster, that loads each configuration change in isolation, carries no customer traffic, and lets the serving processes load the change only after it reports success. ^[13][7]

Managed configuration services provide the first layer. An AWS AppConfig configuration profile accepts up to two validators: JSON Schema, for free-form configurations and limited to version 4 for inline schemas, or an AWS Lambda function, which AppConfig calls on StartDeployment and ValidateConfigurationActivity and allows 15 seconds including start-up. ^[14] A schema catches structure, and a Lambda function can parse the artifact with the consumer's own library. Neither runs the consumer's production build against production inputs, so both are the gate in front of the first stage, not a substitute for it.

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "title": "example-feature-file",
  "type": "object",
  "additionalProperties": false,
  "required": ["schemaVersion", "generatedBy", "features"],
  "properties": {
    "schemaVersion": { "type": "integer", "enum": [3] },
    "generatedBy": { "type": "string", "minLength": 1 },
    "features": {
      "type": "array",
      "minItems": 1,
      "maxItems": 180,
      "uniqueItems": true,
      "items": { "type": "string", "pattern": "^[a-z][a-z0-9_]{0,63}$" }
    }
  }
}

Stage, bake and watch

A first stage needs a shape as well as a size. Cloudflare's plan names three progressions: geographic, to more data centers; population, from employees to customer types; and service, from one product to unrelated ones such as the dashboard. Under its health-mediated deployment model, the team that owns a service defines the metrics that mark success or failure, the rollout plan and the steps to take when a stage fails. ^[1] All three transfer directly to any configuration class.

The wait at each stage must be longer than the time a failure takes to appear. Front Door's stages waited about a minute each, and the October 29 crash surfaced about five minutes after the change reached pre-production, so every gate saw healthy signals and opened. ^[7] Google's SRE workbook states the rule for canaries: a canary must last at least as long as one unit of work takes to process, and the metrics that judge it must cover intervals no longer than the canary itself. ^[15] With configuration, the slowest unit of work is often in the background: a cleanup pass, a cache rebuild, the next regeneration of a file, an hourly batch job.

There are two ways to meet that rule, and Front Door used both. It added stages and longer bake time, and it made data plane processing synchronous so that, by Microsoft's account, a problem caused by incompatible metadata is detected within 10 seconds at every stage. ^[13] Shortening failure latency is usually the better investment, because short stages then stop being blind.

AWS AppConfig's predefined strategies give a documented baseline, shown in the chart. The longer of the two strategies AWS recommends for production, AppConfig.Linear20PercentEvery6Minutes, deploys to 20 percent of targets every six minutes over 30 minutes and then watches CloudWatch alarms for another 30, while AppConfig.Linear50PercentEvery30Seconds is meant only for testing. ^[16] Bake time is the monitoring period after every target has the change, and a custom strategy accepts up to 1,440 minutes each for deployment and bake. ^[17][18]

Two readings follow. In a linear strategy the interval between steps, six minutes in that strategy, is the figure to compare with a measured failure latency, because the final bake protects only the last step. And the sources disagree about scale: Microsoft's Well-Architected guidance says bake times should be measured in hours and days and grow with each rollout group, while AppConfig's recommended strategies complete in 30 to 60 minutes. ^[19][16] Each fits a different class. Hours suit platform data that changes daily, as Front Door's bake of more than 12 hours does, and tens of minutes suit an application flag that fails visibly within one step. A bake measured in days exceeds AppConfig's ceiling and belongs in the pipeline, as separate deployments to successive environments.

Health signals have to be scoped to the stage being judged. Front Door gates each stage on error and latency thresholds and also rechecks earlier stages for regressions, which catches a slow failure after its gate has opened. ^[13] Microsoft's guidance adds usage metrics to the health model, so a stage that served no traffic is not mistaken for a healthy one. ^[19]

Source. AWS AppConfig User Guide, Using predefined deployment strategies, table of strategies and descriptions, reviewed October 9, 2026. ^[16]

Method. Source-derived. Deployment and bake minutes are copied from each strategy's description; AllAtOnce is documented as deploying immediately and is plotted as 0. Bar totals are the sum of the two cited values. Bake time is AppConfig's alarm monitoring period after every target has the change. ^[17]

Accessible table and figure data

Figure 3 accessible table

Strategy	Strategy ID	Deployment window (minutes)	Bake time (minutes)
Linear, 20% every 6 minutes (recommended)	AppConfig.Linear20PercentEvery6Minutes	30	30
Canary, 10% growth over 20 minutes (recommended)	AppConfig.Canary10Percent20Minutes	20	10
All at once	AppConfig.AllAtOnce	0	10
Linear, 50% every 30 seconds (testing only)	AppConfig.Linear50PercentEvery30Seconds	1	1

Figure 3 accessible table

Strategy	Strategy ID	Deployment window (minutes)	Bake time (minutes)
Linear, 20% every 6 minutes (recommended)	AppConfig.Linear20PercentEvery6Minutes	30	30
Canary, 10% growth over 20 minutes (recommended)	AppConfig.Canary10Percent20Minutes	20	10
All at once	AppConfig.AllAtOnce	0	10
Linear, 50% every 30 seconds (testing only)	AppConfig.Linear50PercentEvery30Seconds	1	1

Download imageDownload table CSV

Keep the last known good out of the change's reach

An LKG snapshot is only useful if the change under test cannot become it. Front Door updated its LKG whenever a rollout completed with healthy signals, and on October 29 the bad metadata did exactly that before the crash appeared. Microsoft then chose not to revert to the latest LKG, which held the bad metadata, nor to older versions, citing system stability. Engineers edited the latest LKG by hand from 17:10 UTC, blocked all customer configuration propagation at 17:30, and had the edited snapshot at every edge site by 17:50. ^[7]

The report does not say why older snapshots were rejected. A reasonable reading is that, in a multi-tenant system, an older snapshot also discards every valid customer change made since it was taken. Either way the lesson concerns promotion: promote a version to LKG only after the final stage has baked, keep several earlier versions loadable, and make reverting to any of them a tested operation rather than a manual edit. Front Door now offers single-click revert to any previous LKG version. ^[13]

Consumers hold an LKG too, often without calling it one. Front Door already cached validated customer configurations on host storage, yet recovery took about 4.5 hours because a defect in the recovery path invalidated that cache and forced hundreds of thousands of configurations to be fetched and translated again. The repair no longer invalidates the cache by crash type and evicts only the tenant whose entry fails validation. ^[20] Cloudflare's November recovery worked at the same layer: engineers stopped the generator and manually inserted a known good file into the distribution queue, and the good file was deployed globally at 14:30 UTC, 53 minutes after rollback work began. ^[8]

Managed services differ in how long the previous version stays reachable. In AWS AppConfig, StopDeployment rolls back a deployment in progress, and after completion the AllowRevert parameter reverts it, but only within 72 hours and while ignoring alarm monitors. ^[21] On the consumer side, the AppConfig Agent's BACKUP_DIRECTORY setting saves a copy of each configuration it retrieves, and with PRELOAD_BACKUPS, which defaults to true, the agent loads those copies at startup before asking the service for anything newer. The copies are written unencrypted, so restrict the directory's permissions. ^[22]

Make rollback faster than rollout

Rollback times in the reports run from one minute to most of a working day. Cloudflare's December 5 revert started at 09:11 UTC and was fully propagated at 09:12, carried by the same seconds-scale system that had spread the fault. ^[9] Google's red-button, shipped with the original code as a safety precaution, was ready about 25 minutes in and fully rolled out within 40, but us-central1 took up to about 2 hours 40 minutes because restarting tasks overloaded the Spanner table they depended on. ^[5] Front Door's October 29 impact lasted from 15:41 UTC until 00:05 the next day. ^[7] The target is asymmetric: forward progress slow and observed, reverse progress fast, rehearsed and independent of whatever broke.

Kill switches are configuration changes that run rarely exercised code, at the worst possible moment. Cloudflare's rulesets killswitch had a standard operating procedure that had mitigated earlier incidents and was followed on December 5, but it had never been applied to a rule whose action was execute. Skipping that rule left a result object missing, a Lua lookup on it failed, and every FL1 customer using the Cloudflare Managed Ruleset received HTTP 500 errors. ^[9] After the November 18 outage, Cloudflare had listed more global kill switches among its remediations. ^[8] The goals stop conflicting once the kill switch path is itself a staged, health-mediated change, which is how Cloudflare says Snapstone now treats control flags. ^[11]

Amazon's Builders' Library gives the general reason. A fallback path that is not exercised regularly is likely to fail, or to widen the impact, when it finally triggers; the remedy it describes is to run that path continuously in production until it is failover rather than fallback. ^[23] For kill switches, that means exercising each one against every rule or feature type it can act on, in the first stage, on a schedule.

The emergency path must pass through the same gates. On October 9, Front Door's protection system was holding the bad tenant metadata at an early stage when engineers bypassed it to run a cleanup; the metadata reached later stages and about 26 percent of Front Door's data plane infrastructure in Europe and Africa was affected. ^[6] Microsoft's repair keeps the protection system always on and sends any cleanup through the same guarded stages and health checks as a normal change. ^[13] Shorter waits in an emergency are reasonable; a separate path with no stages is the October 9 incident.

Rollback must not depend on what the change broke. In the July 2026 West US incident, automated recovery started retrying at 14:54 UTC but needed the datacenter connectivity the change had cut; it alerted engineers at 16:58, and a manual rollback ran from 17:45 to 18:26. Microsoft's remaining repairs, with an estimated completion of October 2026, make that automation fail faster and escalate, and let rollback work over degraded connectivity. ^[10] Cloudflare's version of the lesson is break-glass access: backup authorization pathways for 18 key services, and an engineering-wide drill on April 7, 2026 with more than 200 people. ^[11]

A revert also restarts work. After the good feature file went out, Cloudflare's dashboard degraded again from 14:40 to 15:30 UTC as a backlog of login attempts and retries arrived. ^[8] Budget for that load before the rollback starts.

Fail small when a configuration is invalid

Some invalid versions will reach a consumer, so each consumer needs a stated behavior for that case. Cloudflare's completion report gives the order it now uses. Keep the last known good configuration where possible, which it calls fail stale; where that is impossible, choose fail open or fail closed case by case, depending on whether serving traffic with reduced function beats not serving it. For Bot Management, the system refuses data it cannot read and keeps the old version, and if no old version exists it fails open so customer traffic keeps flowing. ^[11]

Google reached a similar position for Service Control, committing to isolate its checks so they fail open and API requests are still served when a check fails. ^[5] Both are availability choices for scoring and quota that do not transfer to every consumer. For an authorization policy, a WAF rule set or endpoint detection content, failing open removes a security control when it is least observed, a decision for the control's owner to write down before an incident forces it.

The Builders' Library warns that rarely used fallback modes hide latent bugs, which seems to argue against fail stale. ^[23] The two reconcile when the stale path is the normal path. Front Door's workers load each update into a staging area, validate it and swap it in atomically, while in-flight requests keep the old configuration. ^[20] A consumer built that way holds the previous version through every update, so the code that serves it runs constantly, long before an incident needs it.

The table is an editorial starting point derived from the reports: a default per consumer for a team to confirm or overrule, not a provider rule.

What to measure

Exposure at first alarm sums up a configuration pipeline in one number: the share of the fleet or traffic holding the change when the first alarm fired. Cloudflare's December 5 change was fully propagated at 08:48 UTC, two minutes before automated alerts declared the incident, so the whole affected population was exposed before detection. ^[9] Front Door's October 29 change had reached most edge sites by 15:39, four minutes before the protection system halted propagation. ^[7] A staged pipeline exists to make this number small, and deployment logs are enough to compute it.

Five more measures belong on the same dashboard:

• Failure latency per artifact class: time from applying a change to the first error signal, measured in the first stage and compared with the wait between stages.
• Rollback time against rollout time for the same change, from the decision to revert to a fully propagated revert.
• LKG depth and age: how many earlier versions are loadable, and how old the newest promoted one is.
• Kill switch freshness: when each kill switch last worked, per rule or feature type.
• Aggregate scope in flight: total regions, cells, devices or tenants touched by every running change, checked as one number.

Speed came back after failures surfaced faster

Front Door's published figures show the trade between speed and safety. Propagation took five to ten minutes before October 2025, about 45 minutes once stages and bake were added, about 20 minutes by March 2026, and was back at pre-incident levels by July 2026, after synchronous processing and Food Taster had shortened how long a bad change takes to show itself. ^{[7][13][20][12]}

Microsoft still lists the added stages and extended bake among its completed repairs, so the reading here is that speed returned because failure latency fell from minutes to seconds, letting short stages see the failures they exist to catch. Recovery moved the same way, from about 4.5 hours to about an hour once the local cache was preserved, then under 10 minutes once workers loaded only tenants with active traffic. ^[20][12] These are Microsoft's own statements, not independent measurements.

Write a rollout contract for each artifact class

Turn the controls into one record per artifact class, checked by a reviewer before the class may reach the fleet. The comparison shows what changes when a class leaves the global path; the fields below are what the record holds.

• Owner, producer and every consumer, with the versions running in production.
• Stage order and size: pre-canary, then one cell or low-traffic region, then widening waves.
• Validation: schema, a load test in each consumer version, and a cross-version check against the previous producer.
• Stage wait: longer than the measured failure latency plus one metric interval, judged on stage-scoped health.
• LKG: where it lives, how many versions are kept, and the rule that promotes a version.
• Rollback: an automatic trigger, a time target shorter than one stage, and a path independent of the change.
• Kill switch: what it can disable, which types it was tested against, and when it last worked.
• Consumer failure mode for each consumer, signed off by the owner of any security control it affects.
• Emergency path: the same gates with shorter waits, and who may shorten them.

Source. Conceptual comparison based on the Azure Front Door, Cloudflare and CrowdStrike reports. ^{[7][9][8][11][4]}

Method. Conceptual comparison. Each row pairs a delivery behavior described in the reports with the control their remediations adopted; no environment was tested.

Accessible table and figure data

Figure 4 accessible table

Aspect	Before	After
First exposure	Every server within seconds	One pre-canary stage or cell
Validation	Schema check at the producer	Schema plus a load test in each consumer version
Wait between stages	None	Longer than the slowest asynchronous failure
Last known good	Replaced as soon as propagation completes	Promoted only after the final bake
Kill switch	Same fast path, untested for this rule type	Exercised per rule type on a schedule
Invalid file at a consumer	Panic, crash or dropped traffic	Keep serving the previous version
Rollback	Manual repair while the fleet is down	Automatic, and faster than one stage

Figure 4 accessible table

Aspect	Before	After
First exposure	Every server within seconds	One pre-canary stage or cell
Validation	Schema check at the producer	Schema plus a load test in each consumer version
Wait between stages	None	Longer than the slowest asynchronous failure
Last known good	Replaced as soon as propagation completes	Promoted only after the final bake
Kill switch	Same fast path, untested for this rule type	Exercised per rule type on a schedule
Invalid file at a consumer	Panic, crash or dropped traffic	Keep serving the previous version
Rollback	Manual repair while the fleet is down	Automatic, and faster than one stage

Download imageDownload table CSV

A worked AWS AppConfig rollout

The example applies the contract to a hypothetical team that ships its flags as a free-form configuration profile in AWS AppConfig. AppConfig has regional endpoints and per-Region quotas, and its deployment strategy governs how a configuration rolls out across the instances of one application environment. ^[24] A reasonable reading is that the strategy staggers exposure inside a Region, while the order of Regions, and any bake longer than a strategy can hold, belongs to the pipeline that calls AppConfig once per Region.

The team has measured in staging that a bad flag value shows up in its HTTP 5xx alarm within about four minutes, so it creates a linear strategy of 10 percent steps over 60 minutes, six minutes per step, with a 30-minute final bake. AppConfig's monitoring documentation lists both ALARM and INSUFFICIENT_DATA as states that trigger rollback during a deployment, so decide in advance what a quiet alarm means; and no rollback happens if the alarm's actions are disabled, which the pipeline checks before every deployment. ^[25]

Three limits shape the rest of the pipeline. The revert in the last command is the 72-hour AllowRevert described earlier, so restrict who may run it. ^[21] The agent polls every 45 seconds by default, so it follows that a step can reach a host up to one poll interval after AppConfig moves to it. ^[22] Deployment and bake each top out at 1,440 minutes, so the second Region starts only after the first reports COMPLETE and the pipeline's own hold has passed. ^[18]

# One Region of a staged AppConfig rollout. IDs, ARNs and names are placeholders.
REGION=us-west-2
APP_ID=abc1234      # application
ENV_ID=def5678      # production environment in this Region
PROFILE_ID=ghi9012  # free-form profile with a JSON Schema validator

# 1. Step interval (60 min / 10 steps = 6 min) exceeds the measured failure
#    latency; 30 minute bake after 100 percent of targets.
aws appconfig create-deployment-strategy --region "$REGION" \
  --name example-linear-10pct-60min \
  --deployment-duration-in-minutes 60 \
  --growth-type LINEAR \
  --growth-factor 10 \
  --final-bake-time-in-minutes 30 \
  --replicate-to NONE

# 2. Attach the 5xx alarm. The role lets AppConfig call cloudwatch:DescribeAlarms.
aws appconfig update-environment --region "$REGION" \
  --application-id "$APP_ID" --environment-id "$ENV_ID" \
  --monitors "AlarmArn=arn:aws:cloudwatch:us-west-2:111122223333:alarm:example-api-5xx,AlarmRoleArn=arn:aws:iam::111122223333:role/example-appconfig-alarm-role"

# 3. Stop the pipeline if alarm actions are off; AppConfig would not roll back.
aws cloudwatch describe-alarms --region "$REGION" --alarm-names example-api-5xx \
  --query 'MetricAlarms[0].ActionsEnabled' --output text | grep -qx True

# 4. Deploy configuration version 42 with the strategy from step 1.
aws appconfig start-deployment --region "$REGION" \
  --application-id "$APP_ID" --environment-id "$ENV_ID" \
  --configuration-profile-id "$PROFILE_ID" --configuration-version 42 \
  --deployment-strategy-id jkl3456 \
  --description "flags v42, Region 1 of 3"

# 5. Emergency only: revert completed deployment 7 (within 72 hours).
#    Warning: a revert ignores alarm monitors.
aws appconfig stop-deployment --region "$REGION" \
  --application-id "$APP_ID" --environment-id "$ENV_ID" \
  --deployment-number 7 --allow-revert

When global and immediate is the right choice

Some configuration has to move faster than any bake. Cloudflare's feature file existed to react to new bot attacks, CrowdStrike designed its content to respond to new attack techniques at operational speed, and CDN customers expect a route change at every edge in minutes. ^[8][3][13] Cloudflare still concluded that near-instant global deployment is useful in many cases but rarely necessary ^[1], and Google committed to incremental propagation of globally replicated data whatever the business need for near-instant consistency. ^[5] The default is therefore staged, unless a specific case meets every condition below.

A global, immediate push is defensible only when all four of these hold, and each can be checked before the change is made:

• Delay is the larger risk: the change answers an active attack or exploit, and a staged path would leave the fleet exposed longer than a bad version would hurt it.
• Every consumer already fails stale, loading the new version beside the old one and keeping the old one if the new one does not load.
• This exact type of change, on this exact rule or feature type, has been through the fast path before without harm. December 5 fails this test.
• Reverting is faster than the spread and uses nothing the change can break.