If you blindly enable every database metric exporter without understanding high-cardinality data, your monitoring stack will collapse before your database does.

Situation

Managed observability platforms like Datadog and CloudWatch are exceptionally powerful, but their pricing models are fundamentally misaligned with high-volume database metrics. If you operate massive, self-managed database fleets on bare metal or Kubernetes, sending every connection state, wait event, and table-level metric to a SaaS provider quickly becomes a top-three line item on your cloud bill.

For teams running their own infrastructure, the Prometheus and Grafana stack remains the definitive open-source baseline. OpenTelemetry’s unified model for logs, metrics, and traces provides the standard vocabulary, but Prometheus is the engine that pulls the metrics. However, database engineers often struggle with Prometheus because its pull-based architecture and label-based querying (PromQL) require a different mental model than traditional agent-based monitoring.

The Problem

Out of the box, a tool like postgres_exporter or mysqld_exporter will scrape hundreds of metrics. The immediate trap that database teams fall into is “cardinality explosion.”

If you configure an exporter to scrape the execution count of every unique normalized SQL query from pg_stat_statements, and you have a high-churn ORM generating thousands of unique query shapes, Prometheus will attempt to store each of those as a unique time series. Memory consumption on the Prometheus server will skyrocket, OOM kills will follow, and you will lose visibility precisely when you need it most.

The Open-Source Database Observability Stack

A production-grade open-source monitoring stack for databases requires three strictly managed layers:

  1. The Exporter Layer: This is a lightweight process (e.g., postgres_exporter) running alongside the database. It translates internal database states into the text-based exposition format Prometheus expects.
  2. The Scrape Configuration: The Prometheus server pulls data from the exporter at a defined interval (e.g., every 15 seconds). This is where you must aggressively filter out high-cardinality labels using metric_relabel_configs to drop metrics you do not actively alert on.
  3. The Alerting Rules: Raw metrics are useless during an incident. You must define Prometheus recording rules to pre-calculate expensive metrics (like the 5-minute rate of disk I/O) and alerting rules (e.g., alert if the connection pool is >90% saturated for 3 minutes).

In Practice

The documented pattern for surviving Prometheus at scale involves ruthless metric dropping.

Context: The mysqld_exporter default configuration exposes mysql_perf_schema_events_statements_total, which creates one time series per unique normalized query digest tracked by the Performance Schema. On an ORM-driven application generating thousands of unique query shapes, this single metric produces hundreds of thousands of unique time series. Prometheus’s documentation on instrumentation best practices explicitly warns that unbounded label values — like digest or query_hash — cause memory growth proportional to the number of unique label combinations, and recommends against high-cardinality dimensions in metric labels (Prometheus: Instrumentation best practices).

Action: The documented mitigation is a metric_relabel_configs block with a drop action targeting mysql_perf_schema_events_statements_total in the Prometheus scrape configuration, combined with a replacement custom collector query that exports only the top-N slowest statements by total execution time from performance_schema.events_statements_summary_by_digest.

Result: The Prometheus TSDB status page (/tsdb-status) exposes the top-10 highest-cardinality metrics by series count — this is the diagnostic that reveals which exporter metric is consuming the majority of Prometheus server memory before it OOM-kills.

Learning: Prometheus is an operational alerting database, not a data lake. The test for any scraped metric: does it drive an alert or a live dashboard panel? If not, drop it at the scrape layer rather than ingesting it and paying the memory cost.

Where It Breaks

Relying on Prometheus and Grafana involves significant operational tradeoffs compared to managed services:

ApproachAdvantageDisadvantageFailure Mode
Prometheus (Self-Hosted)Zero variable cost for high data volume; complete control over scrape intervals.You must manage the storage, backups, and high availability of the monitoring stack yourself.The Prometheus server runs out of disk space and stops recording metrics during an outage.
Datadog / Managed SaaSZero maintenance; built-in correlation between logs, traces, and metrics.High-cardinality custom metrics incur massive monthly costs.Finance forces engineering to drop critical metrics to meet budget constraints.

What to Do Next

  • Problem: Database teams deploy postgres_exporter or mysqld_exporter with default settings, then watch the Prometheus server OOM-kill itself from cardinality explosion within days — the monitoring stack fails before the database does.
  • Solution: Apply metric_relabel_configs to drop high-cardinality per-query metrics on every new exporter deployment, and replace them with a targeted custom collector that exports only top-N slowest queries by total execution time.
  • Proof: Check your Prometheus TSDB status page (/tsdb-status) — if any single metric family consumes more than 10% of total series, you have a cardinality problem that will eventually crash the server under incident load.
  • Action: Audit current exporters via the TSDB status page this week and drop any metric not tied to an active alerting rule or dashboard panel — treat every unalerted metric as operational overhead with a memory cost.