Make secret rotation reach every running application

The secret changed but the application did not

A secret rotation can succeed in the secret manager while a running application continues to use an older credential. The value may have changed in storage, reached a mounted file and still never been reloaded into the process that opens connections. Treat rotation as convergence across the credential backend, delivery mechanism, application memory and fresh use of the credential. The completion test belongs at the consumer, not only in the management service. ^[1][2][3]

Consider a hypothetical database client that reads a mounted password during startup. A controller updates the file, but the client keeps its original configuration object. Existing pooled connections continue to serve requests, so ordinary health checks remain successful. The next new connection may reveal the stale credential. This example describes a possible application design and a testable failure path; it is not a reported behavior of every database driver or a measured production incident.

The appropriate fix depends on how the process consumes the secret. An environment variable, a projected Kubernetes Secret, a Secrets Store CSI mount and a direct secret-manager SDK call have different update contracts. None should be treated as an interchangeable promise that every application will reload itself. Kubernetes and provider documentation distinguish delivery from application behavior, including cases where a restart is required. ^[1][2][3][4]

Start by identifying the actual consumers and the credential authority they use. A service can include web processes, workers, scheduled jobs and recovery deployments with different configuration paths. Record each class separately. This guide focuses on shared-secret delivery and application reload; Entra application certificate registration and signing-key adoption are a related but distinct workflow. The common need for consumer evidence does not make their management APIs or retirement semantics identical.

Use a rotation plan that can explain where a replacement has stopped. If the backend changed but the store did not, the problem differs from a current mount feeding a stale process. That distinction is useful during both routine rotation and incident response. It also prevents an operator from restarting every pod when the actual failure is an unsuccessful credential update in the external system.

Inventory the way each process consumes credentials

For environment-based consumption, the process receives the value when its environment is established. Updating the Kubernetes Secret does not rewrite the environment of an already running container. The application needs an appropriate restart or redeployment to obtain the replacement through that path. Record which workload controller owns the restart and how its health checks establish that new instances can use the credential. Do not assume that a secret-store event automatically triggers this behavior. ^[1][5]

Mounted files have a different contract. Kubernetes can update projected Secret data, but the application still needs to read the replacement. A program that loads a file only at startup behaves differently from a program with a supported reload mechanism. Record whether the consumer rereads for each operation, watches for changes or reloads through an explicit signal or management action. Those are application-specific choices; the delivery controller cannot infer them from the existence of the mount. ^[1][2]

A direct SDK consumer needs similar scrutiny. It may fetch the current secret at startup, cache it for a period or request it when opening a new connection. The secret manager's current version is therefore only one input to the consumer's behavior. The inventory should name the supported client library and the application's cache or refresh policy, rather than describing all direct reads as automatically current. Any concrete implementation must be checked against its own library contract.

Record non-secret version evidence wherever the integration supports it. A secret version identifier, mounted-version status or application configuration generation can help connect delivery to consumer state. The identifier must have a defined meaning: a controller reporting a loaded version does not necessarily report the application's memory. Do not log passwords, connection strings containing credentials or decrypted secret payloads to make the observation easier. ^[2]

Include the connection factory and pool in the inventory. A process can reload a password while an existing pool continues to reuse sessions established earlier. The relevant acceptance test is whether the application can create the connections it will need with the replacement credential, under the backend's actual authentication behavior. This does not require forcefully terminating every production connection; it requires a controlled way to exercise new credential use.

Finally, assign an owner to each consumption mechanism. The platform team may control mounts and deployment rollouts, while the application team controls reload and connection creation. The database or external-service owner controls whether credentials can overlap and when the old credential stops working. A rotation with no owner for one of those stages can remain incomplete even when every other team reports success.

Read the update contract for the actual delivery path

Kubernetes documents projected Secret updates as eventually consistent. The delay depends on kubelet synchronization and the configured change-detection strategy. This is a delivery property, not a guarantee that an application will use the value by a particular deadline. If an operational objective requires a bounded rotation time, measure the relevant path in an authorized environment and include the consumer behavior in that measurement. Do not derive an end-to-end bound from one controller setting. ^[1]

The subPath exception is important. A container using a Secret through a subPath volume mount does not receive automated Secret updates through that mount. A runbook that assumes every mounted secret changes in place can therefore fail for a workload using this pattern. Identify the mount construction during the inventory and choose an update strategy that the actual consumption path supports. A successful update to the underlying Secret object does not remove the exception. ^[1]

Secrets Store CSI Driver documentation reviewed on August 28, 2026 describes v1.6.0 behavior using RequiresRepublish. In that documented implementation, rotation-poll-interval acts as a minimum cached duration before a subsequent kubelet republish can refresh the content. Its default two-minute setting should not be presented as a fixed application-refresh timer or a delivery SLA. Preserve the distinction between the driver's refresh eligibility and the application's later use of the secret. ^[2]

Managed-provider documentation can describe different supported settings and behavior. The AKS provider documentation discusses auto-rotation of mounted content and synchronized Kubernetes Secrets, along with the application's responsibility to observe updates. Google Secret Manager synchronization has its own supported controller and configuration contract. Verify the provider and version actually in use rather than importing a setting from one platform into another because both ultimately produce a Kubernetes Secret. ^[3][4]

Test the reload mechanism against the real delivery behavior. A file watcher, for example, must detect the update pattern produced by the mount or synchronization mechanism and trigger the application's supported reload path. This guide does not claim that a generic watcher snippet has been validated for every runtime. Use a controlled change and observe the application's non-secret version evidence, then test new credential use. A notification received by a watcher is another intermediate event.

Keep delivery errors visible. A controller can fail to retrieve a replacement because of permissions, provider availability or configuration, while the application continues using content already present. Continuing service may be desirable during that failure, but it can conceal stale state. The operational record should distinguish current delivery, retained older content and an unknown state, with alerts or review steps appropriate to the application's rotation objective.

Rotate the backend and the consumer as one change

Changing a stored value is not necessarily the same as changing the credential accepted by the external system. The rotation mechanism must coordinate the secret manager with the database or service that validates the credential. AWS Secrets Manager's rotation tutorials illustrate that relationship: the rotation process updates a supported backend and the managed secret, rather than merely replacing arbitrary text in storage. Use the supported implementation for the actual backend. ^[6][7][8]

AWS documents single-user and alternating-user rotation approaches with different dependencies. Single-user rotation changes the credential associated with one user. Alternating-user rotation uses a different arrangement that can preserve an overlap between credential versions, but it introduces additional user and administrative dependencies. The appropriate choice depends on backend support, permissions and the application's transition behavior. Do not assume that every database or external service supports both approaches. ^[6][7]

An overlap window can give consumers time to move, but it needs an exit condition. Identify which credential states are accepted during the transition, how the application selects the current version and when the old credential may be retired. The interval should reflect known consumers and supported rollback behavior rather than an unexplained universal duration. An infrequently running job may require a controlled test before the window closes.

Rollback also needs a backend-aware definition. Reverting an application deployment does not necessarily restore the credential that the backend accepts. An older release may read a pinned secret version or use a consumption mechanism that cannot reload the replacement. State whether the rollback option still works after each rotation stage. If the backend cannot restore the old credential safely, a deployment rollback command alone is not a complete recovery procedure.

Keep rotation permissions scoped to the chosen strategy. A mechanism that changes a database credential or manages an alternate user has different authority from an application that only reads its current secret. Record those roles and avoid giving every consumer the privileges needed by the rotation service. AWS's database-specific templates and tutorials are the relevant starting point for supported implementation details, not a reason to invent a common administrator role for every backend. ^[6][7][8]

Treat suspected credential exposure differently from routine maintenance. The incident owner may choose to revoke the old credential sooner than a normal overlap plan allows, accepting a service interruption to reduce continued use. That is an explicit containment decision. The article's convergence model still helps identify stale consumers, but it does not imply that preserving the old credential is always appropriate when its confidentiality is in doubt.

Source. Secrets Store CSI Driver rotation ^[2]; AKS CSI provider configuration ^[3]; AWS single-user secret rotation ^[6]; AWS alternating-user secret rotation ^[7]. Sources checked August 28, 2026.

Method. Original conceptual model synthesized from the cited documentation. The relationships and proposed tests explain design choices; they are not measured results or a claim about a deployed environment.

Accessible table and figure data

Figure 1 accessible table

Stage	Owner	Evidence	Common gap
Replace backend credential	Secret rotation service	Backend-supported state transition	Assuming all backends permit overlap
Publish replacement	Secret manager	Non-secret version identifier	Treating storage as application adoption
Deliver	CSI or synchronization controller	Mounted or synchronized version	subPath or provider-specific timing
Reload	Application process	Loaded-version evidence	Read-once configuration
Use	Connection factory	Fresh connection result	Old pooled session
Retire where supported	Credential owner	Controlled old-credential rejection	Unobserved stale consumer

Figure 1 accessible table

Stage	Owner	Evidence	Common gap
Replace backend credential	Secret rotation service	Backend-supported state transition	Assuming all backends permit overlap
Publish replacement	Secret manager	Non-secret version identifier	Treating storage as application adoption
Deliver	CSI or synchronization controller	Mounted or synchronized version	subPath or provider-specific timing
Reload	Application process	Loaded-version evidence	Read-once configuration
Use	Connection factory	Fresh connection result	Old pooled session
Retire where supported	Credential owner	Controlled old-credential rejection	Unobserved stale consumer

Download imageDownload table CSV

Diagnose a current mount feeding a stale connection factory

Return to the hypothetical database client, now with a concrete diagnostic trace. Its connection factory reads a mounted credential only when it starts. A web process and a scheduled worker use the same supported secret-delivery integration but initialize independently. The labels v41 and v42 below are invented non-secret version identifiers, not credentials. The trace is a proposed exercise with synthetic observations, not a report of a rotation run or a timing claim about any provider.

Before the change, the backend accepts the credential represented by v41, the delivery status names v41, and both consumers report configuration generation v41. A fresh test connection succeeds. The exercise owner records which controlled operation established that baseline and which backend rotation strategy will be used. The owner also confirms the rollback conditions before changing anything. A green endpoint health check alone is insufficient because it might reuse a connection opened before the exercise.

At the next checkpoint, the supported rotation process reports that the replacement is usable at the backend and publishes v42 in the secret manager. Delivery metadata also names v42. The web process still reports v41, however, because its connection factory reads only at startup. Existing pooled requests continue to succeed in this synthetic trace. These observations locate a stalled application-adoption stage; they do not show that rotation failed at the backend or that CSI failed to deliver the value. CSI version status describes the mount, not process memory. ^[2][3]

The application owner now uses the application's supported restart or reload procedure in the authorized test environment. That choice depends on the actual client, deployment controller and availability requirements. The exercise should not invent a generic file-watcher fix. After the action, non-secret application instrumentation names v42. The owner then asks the connection factory to create a fresh connection and performs the permitted test operation. A loaded-version observation and a successful fresh connection answer different questions, so both belong in the record.

The scheduled worker remains unresolved. It has not restarted or created a new connection since publication of v42, so the web-process result cannot stand in for it. The owner invokes a permitted test job or waits for an agreed controlled execution that exposes the worker's actual credential-consumption path. Until that happens, the record says worker adoption unknown. It must not infer adoption from an updated mount, a successful web request or the fact that both consumers share a repository.

Retiring the earlier credential is a separate checkpoint governed by the backend strategy. Where a controlled retirement test is supported, record rejection of the old credential on a fresh connection and continued success with the replacement. Do not infer that this rejection terminates every session established earlier. Single-user and alternating-user rotation have different transition behavior; the exercise must follow the chosen implementation rather than demand an overlap it cannot provide. A suspected compromise may require earlier containment and an explicit availability tradeoff. ^[6][7]

The compact record below captures the stalled checkpoint, before reload and the scheduled-worker test. It contains no password or connection string. A real implementation needs instrumentation whose generation label actually changes when the connection factory adopts new configuration; merely echoing the latest secret-manager version would repeat the original mistake. Keep that instrumentation contract with the test so reviewers know what each observation establishes and which state remains unknown.

Use the same trace to choose the next owner when observations differ. If the store is current but the mount is stale, investigate delivery permissions and driver or provider status. If the mount is current but the factory is stale, investigate application reload. If the factory reports the replacement yet a fresh connection fails, check the backend state and the client's connection configuration. Those are diagnostic branches, not universal root causes. Preserve the contradictory evidence rather than forcing it into a single rotation-succeeded status.

{
  "exampleOnly": true,
  "checkpoint": "after-delivery-before-application-reload",
  "versionLabels": "Synthetic metadata only; no credential values",
  "backendReplacementUsable": "v42",
  "secretManagerVersion": "v42",
  "mountedVersion": "v42",
  "webConnectionFactoryVersion": "v41",
  "existingPooledRequest": "succeeds-in-this-synthetic-trace",
  "freshConnectionAfterReload": "not-yet-observed",
  "scheduledWorkerAdoption": "unknown",
  "decision": "rotation-incomplete",
  "nextOwner": "Example application owner",
  "nextAction": "Use supported reload; then test fresh web and worker connections"
}

Source. Kubernetes Secrets ^[1]; Secrets Store CSI Driver rotation ^[2]; AWS single-user secret rotation ^[6]. Sources checked August 28, 2026.

Method. Original conceptual model synthesized from the cited documentation. The relationships and proposed tests explain design choices; they are not measured results or a claim about a deployed environment.

Accessible table and figure data

Figure 2 accessible table

Boundary	Before	After
Secret backend	Replacement exists but consumer state is unknown	Backend credential state is recorded
Delivery	A mount or synchronized object changed	Delivered version identifier is observed
Application	Process may retain startup credentials	Reloaded version is observed without secret values
Connection	Existing pooled connections hide adoption	A fresh connection succeeds with the replacement

Figure 2 accessible table

Boundary	Before	After
Secret backend	Replacement exists but consumer state is unknown	Backend credential state is recorded
Delivery	A mount or synchronized object changed	Delivered version identifier is observed
Application	Process may retain startup credentials	Reloaded version is observed without secret values
Connection	Existing pooled connections hide adoption	A fresh connection succeeds with the replacement

Download imageDownload table CSV

Use failure signals to find the stalled stage

Operational monitoring should identify which part of rotation has stopped progressing. A backend update failure, a secret delivery error, an unchanged application version and a failed fresh connection call for different owners. Tie alerts and review records to those states where the implementation supports them. A single rotation succeeded message is too broad to explain a later authentication failure.

Maintain the consumer inventory as the application changes. A new worker, a different mount pattern, a library upgrade or a revised connection-pool policy can invalidate the assumptions behind an earlier rotation test. Reuse the evidence framework, but test the changed behavior instead of treating a previous successful drill as permanent coverage. Provider and driver updates also deserve review when they alter the delivery contract. ^[1][2][3][4]

Include a recovery deployment when it consumes the same credential. A successful production rollout does not prove that an older recovery configuration can authenticate after the backend has retired its previous value.

The finished procedure should make two statements defensible: the backend accepts the replacement under the chosen strategy, and the required running consumers can use it through their real configuration paths. Delivery, reload and fresh authentication each contribute evidence. Keeping those stages visible makes routine rotation easier to diagnose and gives incident operators a clear account of which credentials and consumers remain unresolved.