Separate stopping a fault experiment from recovering the service

Write three completion states

Stopping an AWS Fault Injection Service experiment does not prove that the fault's effects have been removed or that the application has recovered. Write those as three separate states before approving the experiment: execution stopped, action effects resolved and service accepted. Each state needs an owner and evidence appropriate to the claim.

AWS documents that stop conditions use CloudWatch alarms to halt an experiment and that a stopped experiment cannot resume. It also explains that pending post-actions complete before an experiment stops. The stop mechanism is therefore part of experiment control, not a universal statement about the resulting application state. ^[1][3]

An experiment might stop while a database connection pool remains unhealthy, queued work still needs reconciliation or a terminated resource requires replacement. These are illustrative possibilities, not observations from a test. The experiment's control state cannot substitute for application-specific checks of those consequences.

Make the distinction visible in the approval record. The experiment owner can explain how execution is halted. The resource owner can describe which effects the action reverses and which require repair. The service owner can define what useful recovery looks like. One person may hold several responsibilities, but the responsibilities should not disappear into a single complete checkbox.

This guide does not run an experiment, generate a preview or change an alarm or safety lever. It provides an action-level safety framework based on public documentation. No stop latency, outage reduction or recovery success rate was measured.

The goal is an experiment whose boundaries remain understandable when it is aborted. An early stop can be a valuable result if it reveals a control weakness and the resulting state is handled responsibly. It should not be relabeled as a successful recovery test merely because the experiment no longer runs.

Classify the action before approval

Review each selected action's documented duration, parameters and post-action behavior. AWS describes rollback or cleanup where an action supports it, but an irreversible action such as terminating an instance cannot restore that same terminated instance. Avoid a blanket promise that FIS will undo everything when the experiment ends. ^[2]

Create an action ledger with the target, intended effect, stop behavior, supported post-action, remaining repair and acceptance evidence. This is an original planning recommendation. Its purpose is to expose the difference between an action that removes a temporary constraint and one that changes a resource or application state in a way that needs separate recovery.

Include indirect effects in the review without pretending to predict all of them. A bounded infrastructure fault can cause application retries, delayed work or an operator intervention. The resource action's cleanup may not reconcile those consequences. Ask the service owner which outcomes must be checked before ordinary work resumes.

For a hypothetical experiment that impairs connectivity, distinguish removal of the injected network effect from recovery of established sessions and application work. The first may be action-specific cleanup; the latter requires observations at the application boundary. No universal waiting period is justified without the reader's actual system behavior.

For a hypothetical termination action, the review must identify how replacement and acceptance occur. An experiment report showing termination completed is not a restoration plan. If the repair owner or replacement prerequisites are missing, the experiment is not ready merely because the target selector is syntactically valid.

Keep the ledger scoped to the actual action configuration. A statement about one supported action should not be transferred to another action with a similar name. Revisit the entry when parameters, resource type or provider behavior changes, and preserve the reviewed version of the plan with the experiment record.

Source. Original action-review matrix informed by AWS documentation. ^[2][3][5][6]

Method. Conceptual distinctions without invented scores or recovery times.

Accessible table and figure data

Figure 1 accessible table

Case	What it can establish	What remains owned
Supported post-action	Documented cleanup result	Application and side-effect checks
Irreversible action	Action completed or stopped	Replacement and recovery
Target preview	Observed target selection	Action permissions and actual impact
Experiment report	Reported action and dashboard evidence	Full recovery conclusion

Figure 1 accessible table

Case	What it can establish	What remains owned
Supported post-action	Documented cleanup result	Application and side-effect checks
Irreversible action	Action completed or stopped	Replacement and recovery
Target preview	Observed target selection	Action permissions and actual impact
Experiment report	Reported action and dashboard evidence	Full recovery conclusion

Download imageDownload table CSV

Choose a stop signal that can survive the fault

A stop condition should correspond to unacceptable service risk, not merely a metric that is easy to find. FIS integrates CloudWatch alarms as stop conditions, but the reader must review the alarm's actual configuration and meaning. The presence of an alarm identifier does not establish that it will detect the intended unsafe condition. ^[1]

Define the steady state in terms the service owner can evaluate. Which user operation, resource condition or dependency failure requires stopping? Which signal observes it, and what can make that signal unavailable or misleading during the experiment? These questions should be answered before choosing an action that might impair the monitoring path itself.

Inspect evaluation settings, metric scope and missing-data treatment in the real alarm configuration. This guide does not invent a universal threshold or evaluation duration. A threshold appropriate to one workload may be irrelevant or unsafe for another, and a missing metric should not be casually treated as evidence that the system remains healthy.

Consider a hypothetical fault that affects both an application dependency and the path used to report its health. A dashboard may become quiet precisely when observation is most important. The review should identify an independent signal or an explicit operator decision for that condition rather than assuming that silence means the stop boundary was never crossed.

Separate detection delay from stopping and recovery. Even a correctly configured alarm has an observation and evaluation path. Experiment control has its own transition, and application recovery has another. Do not add undocumented durations together or promise that the service returns to normal within a fixed period after the metric changes.

A readiness check should preserve the alarm definition, intended signal, owner and evidence that its observation path is usable. If those facts cannot be established, record the gap as a precondition failure. The safest experiment is not defined by optimistic stop settings but by a credible way to recognize and act on an unacceptable state.

Make the emergency stop reachable

An operator must be able to stop the experiment through an authorized path that remains usable during the fault. Review the required identity, account, Region and operational interface before the exercise. Do not assume that the same application access path being tested can also support emergency intervention.

AWS FIS safety levers are scoped to an account and Region. The documented Engaging state differs from Engaged: the former indicates the transition is still in progress, while the latter indicates running experiments are stopped and new starts are blocked in that scope. A lever in one scope is not a universal cross-account stop switch. ^[4]

Document which control the operator will use and how its state will be observed. A manual stop request and an engaged safety lever serve related but different operational purposes. The change record should not use the words stopped and blocked interchangeably without stating the scope and actual result.

Because a stopped experiment cannot resume, the recovery plan should describe what happens after an abort. A future attempt is a new decision based on the retained evidence, corrected prerequisites and approval. Do not treat restarting the work as an automatic continuation that inherits every assumption from the failed run. ^[1]

A tabletop can check whether the emergency operator knows the target account, Region and experiment identifier, and whether the authorized access path is available. It cannot by itself prove action cleanup or service recovery. Label tabletop evidence accordingly and avoid claiming a live stop test when only the procedure was reviewed.

If emergency access depends on another team or identity service, identify that dependency explicitly. The experiment's approval should include the relevant coordination, not simply a contact name in a document. A fault plan is incomplete when the only person able to stop it cannot reach the control during the approved window.

Treat preview as a target check only

A target preview is useful but narrower than a dry-run guarantee. AWS documents that preview skips actions, can resolve a different target set from a later real run and does not validate the experiment role's action permissions. It therefore cannot establish that an eventual experiment will behave safely or succeed. ^[5]

Preview is also an experiment operation, not merely a local read-only query. This article did not invoke it. Use it only within the reader's authorized workflow, with a clear understanding that observing target selection does not grant authority to run the fault-producing action.

Review selectors against the intended resource inventory and impact boundary. Tags, resource state and other selection inputs can change between review and execution. Preserve the preview time and configuration, then define which target facts must be rechecked before the approved run.

Keep permissions review separate. An action may have the intended target yet fail because its role cannot perform a required operation. Conversely, a broad role can exceed the authority the experiment needs even when a preview selects a small set. Review target scope and permission scope as two independent boundaries.

Include negative target expectations. The plan should name resources that must remain outside the experiment, such as another service or an unapproved failure domain. A preview that shows some expected targets is insufficient if it also includes an unintended one. This is a proposed acceptance method, not a report of inspected resources.

A useful preflight result is a bounded statement: this configuration selected this observed set at this time, while action permissions, stop behavior and service consequences require separate evidence. That wording preserves the preview's real value without allowing a convenient tool output to stand in for the whole safety case.

Observe the post actions and the application

After a stop request, observe the action-specific state rather than assuming immediate cleanup. AWS states that pending post-actions complete before the experiment stops. That ordering does not establish that every effect has a post-action or that the application is already ready for traffic. ^[2][3]

The sequence diagram therefore extends beyond the stop control. It includes supported post-actions, resource verification, any remaining repair and application acceptance. The extra stages are not implied provider automation. They are responsibilities the experiment plan must assign where the selected action and service require them.

For a temporary fault, verify that the intended constraint is actually removed through the available resource evidence. Then check the application operation the fault was expected to affect. If removal succeeds but the operation remains unhealthy, keep the experiment-control result separate from the open recovery issue.

For an irreversible effect, follow the approved replacement or repair procedure. Preserve the distinction between creating a replacement and restoring the original resource. That distinction matters for identity, data, connections and audit interpretation, even when the service eventually returns to an acceptable state.

Do not let a recovery check create unreviewed side effects. An application validation should use an authorized operation with defined cleanup or reconciliation where necessary. The service owner should know whether the check reads state, writes test data or triggers external work. This article provides no live test or claim of validated recovery.

The action-effect matrix is a qualitative responsibility map. It does not rank actions by invented risk scores or promise a common recovery duration. Its role is to keep unsupported cleanup assumptions visible until a responsible owner supplies the actual evidence.

Source. Original recovery sequence based on AWS FIS action and stop documentation. ^[1][2][3][4]

Method. Conceptual ownership sequence, not a latency model.

Accessible table and figure data

Figure 2 accessible table

Stage	Evidence	Remaining question
Stop requested	Control request and state	Has transition completed?
Supported post-action	Action-specific result	Any irreversible or residual effect?
Resource review	Observed resource state	Can the application perform useful work?
Service acceptance	Owned application checks	Any unresolved side effects?

Figure 2 accessible table

Stage	Evidence	Remaining question
Stop requested	Control request and state	Has transition completed?
Supported post-action	Action-specific result	Any irreversible or residual effect?
Resource review	Observed resource state	Can the application perform useful work?
Service acceptance	Owned application checks	Any unresolved side effects?

Download imageDownload table CSV

Save evidence beyond the report

FIS experiment reports can include action information and optional CloudWatch dashboard snapshots, but their scope is bounded. AWS documents limitations such as excluded cross-Region metric widgets and report availability differences for preview or canceled experiments. A report is therefore useful evidence, not a complete record of application recovery. ^[6]

Retain the experiment configuration, target observations, action states, stop condition, alarm evidence and operator decisions. Add the service's acceptance results and any repair record separately. This makes the final account understandable even when a dashboard snapshot misses the relevant symptom or a report is not produced.

Preserve timestamps with their meanings. Alarm evaluation, stop request, action completion and application acceptance are different events. A single end time can conceal the interval during which the experiment was stopped but the service was still recovering. No universal timing model is asserted here.

Record failures and missing observations as part of the result. If a health signal was unavailable, name it and explain how the owner decided what to do. If a post-action did not establish the expected resource state, retain the repair and its authorization rather than replacing the history with a final healthy screenshot.

A concise evidence index can point to restricted raw material without exposing sensitive resource details in a broad retrospective. The summary should still explain the scope, decision and limitation of each artifact. A report attachment with no interpretation is not a substitute for an accountable recovery conclusion.

Record the control path used for an emergency intervention separately from the automation's normal actions. If an operator repaired a resource manually, preserve what was changed, under whose authority and which acceptance check followed. That intervention can be appropriate, but it changes what the experiment demonstrated. A later review should not credit an automated recovery mechanism for work that a person performed through another path.

Close with an explicit recovery owner

The final approval and completion record should identify who can abort, which effects are reversible, who repairs what remains and who accepts the service. These roles turn an experiment from a collection of actions into a bounded operational decision.

An aborted experiment can produce useful learning without demonstrating successful recovery. State what was established and what remains untested. If the stop path worked but repair required an improvised intervention, the next action is to improve the recovery plan, not to claim that the original plan passed.

Before another run, resolve the specific gaps: target ambiguity, weak stop observation, unavailable emergency access, unsupported cleanup or incomplete application acceptance. Keep the corrections narrow enough to verify, and reapprove changes that alter the fault's scope.

The standard is not that every experiment finishes as planned. It is that the organization can halt the authorized activity, understand its remaining effects and restore or accept the service under explicit ownership. Experiment stopped, effect resolved and service accepted should remain separate statements all the way to the final report.