Embedding Backfill in Postgres: Batch Size, WAL, Autovacuum, and Bloat
How large embedding backfills stress PostgreSQL through batch size, WAL growth, checkpoints, autovacuum lag, bloat, index timing, throttling, and rollback planning.
Practical write-ups across PostgreSQL, MySQL, MongoDB, Cassandra, Valkey, Elasticsearch, OpenSearch, and vector search — plus the cloud platforms and AI agent patterns that operate them.
Nine engines and operating layers, each at a different point on its own learning curve — some run for twenty years, some picked up recently over the last couple of years. AI engineering is the lens applied across all of them, not a separate silo. The posts under each one show the depth better than any label could.
Agents, evals, MCP, and the guardrails that connect back to every system below.
MVCC, autovacuum, replication, Aurora and Citus scaling, pgvector.
InnoDB internals, replication lag, Aurora MySQL, sharded write scaling.
WiredTiger cache, shard hotspots, queryable encryption, query performance.
Write path fundamentals, compaction strategy, tombstone observability.
AWS and on-prem infrastructure — Terraform, Kubernetes, migration cost.
Eviction policy, licensing history, ElastiCache performance triage.
Shard and heap troubleshooting, hybrid search, kNN architecture.
Deep pgvector internals, plus newer ground with Qdrant and Weaviate as the vector landscape expands.
New to the site? These posts are ordered as a reading path, while Latest Notes below is ordered by publication date.
Recent field notes and breakdowns across AI engineering, databases, cloud, and system design.
How large embedding backfills stress PostgreSQL through batch size, WAL growth, checkpoints, autovacuum lag, bloat, index timing, throttling, and rollback planning.
Why pgvector changes backup and restore planning for RAG systems, including vector column size, index rebuilds, embedding reproducibility, source-of-truth design, and DR runbooks.
A DBA watchlist for running pgvector on Aurora PostgreSQL, covering extension support, memory, I/O, WAL, replicas, failover, backups, parameters, and cost.
A staged playbook for moving from Postgres and pgvector prototypes to hybrid search, agents, and GraphRAG only when the workload requires it.
A production decision guide for GraphRAG, entity graphs, community summaries, relationship reasoning, and where graph retrieval is overkill.
A production framework for adding agents only when users need planning, tool calls, validation, retries, and governed actions.
Browse by the production problems the notes are written around.
Agents, context engineering, harness design, MCP, evaluation, token efficiency, and AI-assisted engineering workflows.
PostgreSQL, Aurora, MySQL, Oracle, Cassandra, MongoDB, pgvector, replication, migrations, indexing, and database operations.
AWS, Azure, GCP, OCI, Terraform, Kubernetes, CI/CD, Cloudflare, developer platforms, and operational control planes.
Architecture reviews, scalability, failure modes, guardrails, distributed systems, reliability boundaries, and production tradeoffs.
Core engineering principles, debugging workflows, observability, performance basics, reviews, and practical operating habits.
Short practical observations, checklists, production lessons, debugging notes, and decision patterns from real engineering work.
Multi-post arcs that connect practical decisions across a topic.
14 posts · Start: AI Token Cost Overruns: Why AI Coding Assistants Are Becoming the New Cloud Bill Problem
Databases63 posts · Start: The Database Observability Baseline: What Every DBA Dashboard Must Show
Databases67 posts · Start: PostgreSQL Slow Query Triage Workflow