CI/CD Pipelines Are Distributed Systems With Bad Observability
CI/CD failures rarely start as broken scripts; they start as distributed coordination failures hiding behind a green-or-red build badge.
Situation
Modern delivery systems no longer look like a shell script running on one box. A single change can fan out across source control webhooks, workflow schedulers, hosted runners, container registries, package mirrors, secret stores, test environments, deployment controllers, approval gates, and chat notifications.
Platform teams often describe this as automation. That framing is too small. A CI/CD platform is a distributed system whose primary job is to turn intent into verified change. It accepts an event, constructs a graph, assigns work to workers, moves artifacts through storage systems, evaluates policy, and coordinates rollout across environments.
The industry has improved the ergonomics of defining pipelines. YAML made workflows reviewable. Hosted runners reduced fleet maintenance. GitOps moved deployment intent into version control. Preview environments made validation more realistic. None of these removed the distributed nature of the system. They mostly made the control plane easier to use.
The operational gap is that most teams still observe CI/CD as if it were a linear process. They look at job logs, duration charts, and final status. That is equivalent to debugging a distributed database by tailing one replica.
The Problem
A failing pipeline is not always a failing command. It may be a queueing problem, cache invalidation problem, dependency outage, lease contention issue, permission drift, artifact corruption, stale environment, policy mismatch, or scheduler bug.
The difficulty is that CI/CD systems collapse many failure domains into the same user experience: the build is red, the deployment is blocked, or the job is still running. The developer sees a pipeline failure. The platform team sees a ticket with a link to logs. The real failure may be several hops away from the visible symptom.
This causes three recurring mistakes.
First, teams over-index on step logs. Logs explain what a worker process saw after it started. They often say little about why the job waited 42 minutes before scheduling, why a runner was selected, which cache key was used, which deployment controller reconciled the change, or which external dependency was degraded.
Second, teams treat pipeline duration as a single metric. End-to-end latency matters, but it is not diagnostic. Queue time, setup time, dependency fetch time, test execution time, artifact upload time, approval wait time, and rollout convergence time are different signals. Aggregating them into “build took 27 minutes” destroys the shape of the problem.
Third, teams optimize locally. A service team adds retries. A platform team increases runner capacity. A security team adds another scan. A release team adds a manual gate. Each change may be reasonable in isolation, but the resulting system accumulates hidden coupling.
The core question is not “how do we make the pipeline faster?” It is: how do we operate CI/CD as a distributed control plane whose failure modes are visible, attributable, and recoverable?
Core Concept
The answer is to model CI/CD as a distributed system with explicit state transitions, ownership boundaries, and telemetry at every handoff.
A pipeline has a data plane and a control plane. The data plane is the actual work: compilation, test execution, image building, scanning, and deployment. The control plane decides what should happen, when it should happen, where it should run, and whether the result is acceptable.
Most observability work should start at the control plane.
flowchart TD
A[commit event — source control] --> B[pipeline scheduler — workflow graph]
B --> C[queue — runner capacity]
C --> D[runner — isolated execution]
D --> E[artifact store — build outputs]
E --> F[policy gate — checks and approvals]
F --> G[deployment controller — desired state]
G --> H[runtime environment — observed state]
H --> I[feedback channel — status and alerts]
B --> J[metadata store — run state]
C --> J
D --> J
E --> J
F --> J
G --> J
H --> J
The first requirement is traceability. Every pipeline run needs a stable correlation identifier that follows the commit, workflow, jobs, artifacts, environments, approvals, and deployment events. Without that, the system cannot answer basic questions such as “which artifact reached staging?” or “which approval allowed production rollout?”
The second requirement is state modeling. A job should not merely be “running” or “failed.” The useful states are more specific: admitted, queued, assigned, preparing, executing, uploading artifacts, waiting for policy, deploying, converging, and completed. These states let teams separate execution failure from orchestration failure.
The third requirement is dependency visibility. CI/CD systems rely on package registries, container registries, secret stores, identity providers, cloud APIs, artifact stores, test databases, and deployment targets. If those dependencies are not part of the pipeline trace, every incident starts with guesswork.
The fourth requirement is replayability. A good pipeline can tell you what it did. A better one can tell you what it would do again. That means preserving inputs: commit SHA, workflow version, runner image, dependency lockfiles, environment variables that are safe to retain, policy versions, artifact digests, and deployment manifests.
In Practice
Context: GitHub Actions documents workflows as event-driven graphs composed of jobs and steps, with dependencies expressed through needs, runner selection, artifacts, caches, environments, and deployment protection rules. The documented pattern is a scheduler assigning graph nodes to execution environments while preserving workflow state.
Action: Treat each job boundary as a distributed-system boundary. Capture queue duration, runner label, runner image, cache hit status, artifact digest, dependency installation time, environment wait time, and deployment approval time as first-class telemetry.
Result: The operational question changes from “why did the build fail?” to “which handoff failed?” A job that waited 30 minutes for a runner has a capacity problem. A job that repeatedly misses cache has a keying or dependency drift problem. A deployment waiting on an environment rule has a policy or approval bottleneck, not a test failure.
Learning: The documented GitHub Actions model already exposes many control-plane concepts. The missing piece in many organizations is not another YAML abstraction. It is disciplined observability over the graph GitHub is already executing.
Context: Argo CD documents a reconciliation model where the desired application state in Git is compared with the observed state in Kubernetes, producing sync and health status. That is not a command runner; it is a controller loop.
Action: Observe deployment as convergence, not as a final shell step. Track desired revision, applied revision, sync status, health status, reconciliation time, Kubernetes events, and rollback decisions in the same trace as the build artifact.
Result: Production deployment stops being a black box after “kubectl apply” or a Git commit. The platform can distinguish “manifest accepted,” “controller applied desired state,” “workload became healthy,” and “runtime stayed healthy after rollout.”
Learning: GitOps makes deployment intent auditable, but intent alone is not delivery. The operational truth is the gap between desired state and observed state.
Context: Bazel’s remote caching and remote execution documentation describes builds as graphs of actions whose outputs can be reused when inputs match. The documented pattern is content-addressed work rather than step-by-step scripting.
Action: Apply the same thinking to CI performance. Measure cacheability, invalidation causes, dependency fanout, action duration, and artifact reuse instead of only measuring total pipeline time.
Result: Optimization becomes structural. Teams can identify whether slow delivery comes from unnecessary work, low cache hit rates, oversized test targets, or serialized graph edges.
Learning: A pipeline is faster when less unnecessary work is scheduled, not merely when larger machines run the same opaque sequence.
Where It Breaks
| Failure mode | What it looks like | What to observe | Better response |
|---|---|---|---|
| Runner starvation | Jobs sit pending | Queue time by label and repository | Capacity planning and concurrency limits |
| Cache drift | Builds get slower without code changes | Cache hit rate and key churn | Stable keys and dependency discipline |
| Artifact ambiguity | Wrong version reaches an environment | Artifact digest and commit correlation | Immutable promotion |
| Policy opacity | Deployments appear stuck | Approval state and rule evaluation | Visible gates with owners |
| Environment decay | Tests fail only in CI | Environment version and fixture state | Rebuildable test environments |
| Retry masking | Pipelines pass after repeated attempts | Retry count and failure class | Fix root cause before adding retries |
| Deployment blind spot | Build is green but release is bad | Sync, health, and runtime signals | Treat rollout as part of CI/CD |
What to Do Next
- Problem: Your pipeline is probably already a distributed system, but its observability is still organized around step logs and final status.
- Solution: Model the pipeline as a control plane. Trace every handoff from source event to runtime convergence.
- Proof: Use documented behavior from systems such as GitHub Actions, Argo CD, and Bazel as the baseline: graph scheduling, reconciliation, and content-addressed work are distributed patterns.
- Action: Add correlation IDs, state transition metrics, artifact digests, queue time, cache telemetry, policy visibility, and deployment health to the pipeline before adding another abstraction layer.