Self-Service Database Provisioning: Catalog Request, Terraform Module, Policy, and Audit
The hard part of self-service databases is not creating the database. It is creating the right database, under the right constraints, with enough evidence that operations, security, finance, and application teams can all trust what happened later.
Situation
Engineering organizations want product teams to move without waiting on a central database team for every PostgreSQL schema, MySQL instance, Redis cache, read replica, or analytics warehouse. The old ticket queue made sense when infrastructure changed slowly and a small group of specialists held all production access. It breaks down when teams deploy daily, cloud providers expose hundreds of database options, and every environment needs reproducibility.
Platform engineering changes the interface. Instead of asking a DBA to run commands, an application team requests a database capability from an internal catalog. Behind that request is infrastructure as code, policy as code, CI/CD, secrets management, and audit logging.
The goal is not to remove database expertise. The goal is to encode the repeatable parts of that expertise so specialists spend less time provisioning standard resources and more time improving the platform.
The Problem
A naive self-service workflow turns database provisioning into a button that creates risk faster.
If the catalog form exposes every cloud setting, application teams inherit provider complexity. If it exposes too little, teams open escape-hatch tickets. If Terraform modules are copied per team, drift appears immediately. If policy runs after infrastructure creation, bad resources already exist. If approvals live only in chat, auditors cannot reconstruct who requested what, which policy evaluated it, and which commit changed production.
The database team still owns the failure domain. A mis-sized instance can hurt availability. A missing backup policy can turn a routine incident into data loss. A public endpoint can become an exposure event. A missing cost tag can make chargeback impossible. A missing owner can leave production data orphaned.
The core question is: how do you let teams provision databases themselves while keeping the control plane opinionated, reviewable, and auditable?
The Answer: Catalog-Driven Provisioning
The architecture should separate the user interface from the execution path.
The service catalog is the product surface. It asks for intent: engine, environment, data classification, region, durability tier, expected workload, owning team, and cost center. It should not ask an application engineer to select every subnet group, parameter group, backup flag, encryption option, or IAM binding.
The Terraform module is the implementation contract. It maps approved intent into provider resources. It should set secure defaults, hide incidental provider detail, and expose only the variables the platform team is willing to support.
Policy is the guardrail. It validates the request and the Terraform plan before apply. It should reject unsafe combinations early: production without backups, public access for restricted data, missing ownership metadata, unsupported regions, weak encryption, excessive instance classes, or nonstandard maintenance windows.
Audit is the evidence stream. Every request, policy result, approval, plan, apply, output, secret reference, and lifecycle action should be traceable.
flowchart TD
A[developer — database request] --> B[service catalog — intent form]
B --> C[request record — owner and purpose]
C --> D[ci pipeline — plan workflow]
D --> E[terraform module — approved database pattern]
E --> F[terraform plan — proposed change]
F --> G[policy engine — guardrail evaluation]
G -->|approved| H[manual approval — production gate]
G -->|rejected| I[feedback — failed checks]
H --> J[terraform apply — provision resources]
J --> K[secrets manager — connection material]
J --> L[audit log — request policy apply]
J --> M[database service — managed instance]
This gives each layer a clear responsibility.
The catalog owns ergonomics. The module owns repeatability. Policy owns constraints. CI/CD owns execution. Audit owns reconstruction.
A good module should encode database lifecycle decisions explicitly. For example, a production PostgreSQL request might always enable encryption at rest, automated backups, deletion protection, private networking, monitoring, parameter baselines, owner tags, and backup retention. A development database might use smaller defaults but still require tags, private access, and an expiration date.
A good catalog should make the paved road obvious. Most teams should choose from tiers such as dev, staging, production-standard, and production-critical. These are business and operational promises, not raw instance sizes. The module can translate the tier into backup retention, high availability, monitoring, maintenance policy, and allowed sizes.
A good policy layer should evaluate both request metadata and infrastructure plans. Request policy catches missing owners and unsupported combinations before Terraform runs. Plan policy catches what the provider resources will actually do. That second check matters because module changes, provider defaults, and conditional logic can produce surprising plans.
In Practice
Context: AWS Service Catalog documents the pattern of centrally managing approved infrastructure products that end users can launch without receiving broad cloud permissions. The documented pattern is a controlled catalog of products, portfolios, constraints, and launch roles, rather than direct access to every cloud API.
Action: Apply the same pattern internally for databases. The product team requests “managed PostgreSQL for production” through the catalog. The platform workflow resolves that request into a versioned Terraform module and runs policy checks before apply.
Result: The request path becomes standardized. Teams do not need direct administrative access to database APIs, and the platform team can evolve the underlying module without changing the catalog interface for every consumer.
Learning: Self-service works when the abstraction is a supported product, not a thin wrapper around provider configuration.
Context: HashiCorp’s Terraform module pattern documents reusable infrastructure packages with inputs, outputs, versions, and composition. The documented pattern is that common infrastructure should be packaged and reused instead of copied across workspaces.
Action: Put database defaults in a small number of versioned modules: one for PostgreSQL, one for MySQL, one for Redis, and one for warehouse datasets if needed. Treat module version upgrades as platform releases with changelogs, tests, and migration notes.
Result: The same defaults apply across teams. Drift becomes easier to detect because supported variation flows through module inputs rather than hand-edited resources.
Learning: The module is not just code reuse. It is the operational contract between platform engineering and application teams.
Context: Open Policy Agent documents policy as code as a way to make authorization and compliance decisions using declarative rules. The documented pattern is externalizing policy decisions from application logic so they can be reviewed, tested, and versioned.
Action: Evaluate database requests and Terraform plans against policy before provisioning. Reject production databases without deletion protection, private networking, backups, owner tags, and approved regions. Require extra approval for high-cost classes or sensitive data tiers.
Result: The workflow fails before infrastructure changes when a request violates guardrails. The rejection can return a specific policy message rather than a vague platform denial.
Learning: Policy should be close enough to the workflow to block unsafe changes, but separate enough from the module to remain reviewable by security and operations.
Context: Cloud audit systems such as Google Cloud Audit Logs and AWS CloudTrail document the pattern of recording administrative activity for later investigation and compliance review.
Action: Store the catalog request ID in every downstream system: CI run metadata, Terraform workspace variables, resource tags, policy result records, and approval comments. Emit a durable event when the request is submitted, approved, rejected, applied, rotated, modified, or destroyed.
Result: During an incident or audit, the team can reconstruct who requested the database, what was approved, what Terraform planned, which policies passed, when it changed, and which resources were created.
Learning: Audit is not a screenshot of an approval. It is a chain of evidence across systems.
Where It Breaks
| Failure mode | Why it happens | Mitigation |
|---|---|---|
| Catalog sprawl | Every team asks for a custom product | Keep few supported tiers and require platform review for new offerings |
| Module escape hatches | Teams need unsupported settings | Add explicit extension points with ownership and review |
| Policy noise | Rules block valid work without context | Version policies, test them, and return actionable failure messages |
| Approval theater | Humans approve changes they cannot evaluate | Approve intent and exceptions, not raw provider diffs alone |
| Secret leakage | Outputs expose credentials in CI logs | Store credentials only in a secrets manager and output references |
| Drift | Operators change resources outside Terraform | Detect drift on schedule and route fixes through the same workflow |
| Cost surprises | Self-service hides spend impact | Show estimated monthly cost before approval and tag every resource |
| Ownership decay | Teams reorganize and databases remain | Require owner validation and periodic recertification |
What to Do Next
- Problem: Database provisioning is slow because the control process lives in tickets and expert memory.
- Solution: Move the request into a service catalog backed by versioned Terraform modules, pre-apply policy checks, CI/CD execution, and durable audit records.
- Proof: This follows documented patterns from service catalogs, Terraform modules, policy as code, and cloud audit logging rather than relying on ad hoc approval threads.
- Action: Start with one supported database product. Define the catalog fields, write the module contract, add five non-negotiable policies, emit a request ID through the pipeline, and run the first production provisioning workflow as a reviewed platform release.