Automation Readiness Review: Inputs, State, Permissions, Rollback, and Audit
Automation does not fail because teams lack scripts; it fails because the platform cannot prove the script is safe enough to run.
Situation
Platform teams are being asked to automate everything that used to require a ticket, a meeting, or a senior engineer at a keyboard: environment creation, database migrations, feature flag rollout, certificate rotation, cache purges, dependency updates, access grants, incident mitigations, and production deploys.
That pressure is rational. Manual operations do not scale, and human approval queues become their own outage mode. The mature response is not to reject automation. It is to make automation reviewable before it becomes executable.
A useful automation readiness review asks five questions before the first production run: are the inputs bounded, is state understood, are permissions scoped, is rollback credible, and is the audit trail durable?
The Problem
Most internal automation starts as a successful local procedure. Someone documents commands, another person wraps them in a script, a CI job appears, and eventually the platform has a button labeled “Run.” The button feels like maturity, but it may only be concealment.
The risk is that automation removes friction without replacing judgment. A human operator may notice that the target environment is wrong, that a database is already in a degraded state, or that a command is about to mutate more resources than intended. A pipeline will usually do exactly what it was told.
The failure modes are familiar:
- Inputs are strings when they should be constrained types.
- State is fetched once and assumed stable for the rest of the run.
- Permissions belong to the pipeline, not the operation.
- Rollback is described as “rerun the previous job.”
- Audit records show that something ran, but not why it was allowed.
The core question is: what must a platform prove before it is allowed to automate a production change?
The Readiness Contract
The answer is to treat automation as a contract, not a script. The contract does not guarantee that every run succeeds. It guarantees that every run is bounded, observable, reversible where possible, and attributable.
flowchart TD
A[Change request — desired outcome] --> B[Input contract — typed parameters]
B --> C[State contract — inventory and locks]
C --> D[Permission contract — scoped identity]
D --> E[Execution plan — dry run and gates]
E --> F[Rollback plan — inverse action and stop points]
F --> G[Audit record — evidence and decision trail]
G --> H[Promotion decision — run or reject]
E -->|approved| I[Production execution — bounded mutation]
E -->|rejected| J[No execution — recorded reason]
I --> K[Postcheck — observed state]
K --> G
The input contract defines what the automation accepts. It should prefer enums, resource identifiers, validated ranges, and explicit environment names over free-form text. If a workflow accepts prod and production and main-prod, it has already delegated policy to string parsing.
The state contract defines what the automation believes is true before it acts. This includes the target resource inventory, current version, dependency health, outstanding locks, and any concurrent change windows. Automation that mutates shared systems without checking state is not automation; it is remote execution.
The permission contract binds authority to the operation. A deployment job should not have permanent access to every secret and every cluster because one step needs to update one service. Credentials should be short-lived where possible, scoped to the target, and tied to the request.
The rollback contract is not a promise that time can move backward. Some operations are reversible, some are compensating, and some are one-way. The readiness review should force the distinction. For a schema migration, rollback may mean restoring from backup, running a forward fix, or stopping before a destructive step. For an access change, rollback may be immediate revocation. For a message replay, rollback may be impossible, so the guardrail must move earlier.
The audit contract records who requested the change, what was evaluated, which gates passed, which version ran, which identity executed, what state changed, and what evidence was produced afterward. Logs alone are insufficient if they cannot connect decision, authority, and effect.
In Practice
Context
The documented pattern across mature systems is that automation is safest when desired state, authorization, and observed state are separated.
Kubernetes does this through declarative resources, controllers, admission control, and RBAC. A user submits desired state; the API server validates and authorizes it; controllers reconcile actual state toward that intent. The architectural lesson is not “use Kubernetes for everything.” The lesson is that mutation should pass through a control plane that can validate intent before execution.
Terraform’s documented state model gives another example. Terraform compares configuration with state, produces a plan, and then applies changes. Remote state locking exists because infrastructure state is shared and concurrent writers can corrupt intent. The learning is that a plan without state discipline is only a guess.
Google’s Site Reliability Engineering material repeatedly emphasizes safe rollout, progressive change, observability, and rollback planning. The documented pattern is that production change is an operational risk surface, not a build artifact. The release mechanism must expose enough evidence for operators to decide whether to continue, pause, or revert.
GitHub Actions environments and deployment protection rules show the same concern in CI form. A workflow may be syntactically valid and still require environment-specific review, secrets, or approval before deployment. The learning is that a pipeline stage is not equivalent to permission.
Action
An automation readiness review should be run before an internal workflow receives production authority. The review can be lightweight, but it should be explicit.
First, require an input schema. Each parameter should have a type, validation rule, default policy, and owner. Avoid hidden defaults for environment, region, account, cluster, or tenant. Those are blast-radius controls.
Second, require a state read. The workflow should show what it will touch and what it believes the current state is. If it cannot enumerate targets, it should not mutate them. If state can change during execution, the workflow needs locks, leases, version checks, or idempotent reconciliation.
Third, require an execution identity. The identity should be named, scoped, rotated, and separable from the developer who wrote the automation. Long-lived shared credentials are a readiness failure.
Fourth, require rollback classification. Mark each step as reversible, compensating, or irreversible. Reversible steps need tested inverse actions. Compensating steps need an approved forward repair. Irreversible steps need stronger prechecks and smaller batches.
Fifth, require audit evidence. A completed run should leave behind the request, plan, approvals, artifact version, actor, execution identity, target set, result, and postcheck evidence.
Result
The result is a platform that can say no before production says no. Bad inputs fail at validation. Stale assumptions fail at planning. Overbroad permissions fail before credentials are issued. Weak rollback plans fail before the change is scheduled. Missing audit data fails before the run disappears into logs.
This does not remove human judgment. It moves judgment to the point where it is cheapest: before execution.
Learning
The documented pattern is consistent across Kubernetes, Terraform, SRE release practices, and protected CI deployments: automation becomes reliable when intent, authority, state, and evidence are first-class objects. A script can perform an action. A platform must justify it.
Where It Breaks
| Failure mode | Why it happens | Readiness response |
|---|---|---|
| Overvalidated inputs | The schema blocks legitimate emergency work | Add an emergency path with stronger audit and narrower scope |
| Stale plans | State changes between review and execution | Use locks, version checks, leases, or short plan lifetimes |
| Fake rollback | The inverse path was never tested | Run rollback drills in non-production and classify irreversible steps |
| Permission sprawl | One job accumulates every capability | Issue scoped, short-lived credentials per operation |
| Audit noise | Logs exist but decisions are not reconstructable | Record request, plan, approval, actor, identity, target, and result |
| Slow approvals | Every run needs human review | Promote proven workflows to policy-based approval after evidence accumulates |
What to Do Next
- Problem: Your automation may be executable before it is reviewable.
- Solution: Add a readiness contract covering inputs, state, permissions, rollback, and audit before granting production authority.
- Proof: Compare the workflow against documented control-plane patterns from Kubernetes, Terraform, SRE release engineering, and protected deployment environments.
- Action: Pick one high-risk automation path this week and require a typed input schema, preflight state plan, scoped execution identity, rollback classification, and durable audit record before the next production run.