L1 Field Note
Databases From POC to Production: A Natural Language Search Migration Playbook
A staged playbook for moving from Postgres and pgvector prototypes to hybrid search, agents, and GraphRAG only when the workload requires it.
Topic
Field Notes
Short practical observations, checklists, production lessons, debugging notes, and decision patterns from real engineering work.
89 posts 0 deep dives
Related Series
Latest in Field Notes
L1 Field Note
Databases GraphRAG: Use It When Relationships Matter, Not When Search Is Hard
A production decision guide for GraphRAG, entity graphs, community summaries, relationship reasoning, and where graph retrieval is overkill.
L1 Field Note
AI Engineering Agentic RAG: When Natural Language Search Becomes an Action System
A production framework for adding agents only when users need planning, tool calls, validation, retries, and governed actions.
L1 Field Note
Databases When Search Becomes a Product: OpenSearch and Elasticsearch Architecture
Why marketplace, travel, retail, local commerce, and support search often need a dedicated search platform instead of only pgvector.
L1 Field Note
Databases When Postgres and pgvector Are Enough for Natural Language Search
A database-first decision guide for using PostgreSQL full-text search and pgvector before adding a search engine or vector database.
L1 Field Note
Databases Hybrid Search Before RAG: The Production Baseline for Natural Language Search
Why production search systems should combine BM25, vector retrieval, filters, fusion, ranking, and reranking before relying on LLM answers.
All Field Notes Posts
L1 Field Note
Databases From POC to Production: A Natural Language Search Migration Playbook
A staged playbook for moving from Postgres and pgvector prototypes to hybrid search, agents, and GraphRAG only when the workload requires it.
L1 Field Note
Databases GraphRAG: Use It When Relationships Matter, Not When Search Is Hard
A production decision guide for GraphRAG, entity graphs, community summaries, relationship reasoning, and where graph retrieval is overkill.
L1 Field Note
AI Engineering Agentic RAG: When Natural Language Search Becomes an Action System
A production framework for adding agents only when users need planning, tool calls, validation, retries, and governed actions.
L1 Field Note
Databases When Search Becomes a Product: OpenSearch and Elasticsearch Architecture
Why marketplace, travel, retail, local commerce, and support search often need a dedicated search platform instead of only pgvector.
L1 Field Note
Databases When Postgres and pgvector Are Enough for Natural Language Search
A database-first decision guide for using PostgreSQL full-text search and pgvector before adding a search engine or vector database.
L1 Field Note
Databases Hybrid Search Before RAG: The Production Baseline for Natural Language Search
Why production search systems should combine BM25, vector retrieval, filters, fusion, ranking, and reranking before relying on LLM answers.
L1 Field Note
Databases Natural Language Search Architecture: A Database Architect's Decision Framework
A production decision framework for choosing lexical search, vector search, hybrid retrieval, RAG, agents, or GraphRAG by workload shape.
L1 Field Note
Databases Datadog DBM: What Database Teams Should Actually Monitor
Datadog Database Monitoring can surface enormous detail — and bill for it. The skill is choosing the few signals that answer real cost and reliability questions, and not paying to collect noise nobody acts on.
L1 Field Note
AI Engineering AI Token Cost Is the New Cloud Bill
Token spend behaves differently from compute and storage — it scales with usage and prompt design. Treating it like an engineering cost line, the way you treat a database bill, is how you bring it under control.
L1 Field Note
Databases Why Database Engineers Should Care About AI Cost Engineering
The skills that make a good cost-aware DBA — measuring usage, finding structural waste, balancing cost against reliability — transfer almost directly to AI workloads. Database engineers are unusually well positioned to own AI cost.
L1 Field Note
Databases How to Run a Database Cost & Reliability Review
A practitioner walkthrough of the review method: what to look at, in what order, how to quantify an opportunity honestly, and how to turn findings into a prioritized 30/60/90-day plan.
L1 Field Note
Databases Aurora Cost Optimization: The Hidden Database Bill
Aurora cost hides in places the console doesn't foreground — I/O charges, oversized writers and readers, replica sprawl, and storage. A structured way to find and reduce each without hurting reliability.
L1 Field Note
Databases PostgreSQL Bloat, Index Waste, and Cloud Cost
Table and index bloat and unused indexes are well-known Postgres problems — and direct cloud-cost problems: wasted storage, write amplification, and extra I/O. How to measure both with read-only queries and remediate safely.
L1 Field Note
AI Engineering AI Coding Assistant ROI: When $200/Developer/Month Is Cheap — and When It Is Waste
Why treating AI assistant seats like standard SaaS licenses obscures their true infrastructure cost profile, and how to measure ROI using cloud compute parallels.
L1 Field Note
AI Engineering Token Budgeting for Engineering Teams: Daily, Weekly, Monthly Controls by Developer and Repository
How to implement token quotas, chargebacks, and spend controls for AI engineering teams, drawing parallels from cloud database cost management.
L1 Field Note
Databases SQL Server to PostgreSQL Migration Cost Defense Checklist
A pragmatic checklist to defend the business case for migrating away from Microsoft SQL Server.
L1 Field Note
Engineering Fundamentals AI Cost Observability Dashboard: LangSmith vs Helicone
How to build an AI FinOps dashboard and choose between proxy-based and instrumentation-based observability.
L1 Field Note
Databases Beyond the Slow Query Log: AI-Driven Troubleshooting Across the Application-Database Boundary
How to use LLMs to holistically diagnose complex production incidents by correlating database metrics, application ORM models, and driver configurations.
L1 Field Note
System Design Why Your Non-Prod Databases Cost as Much as Production
Architectural strategies to eliminate waste in Dev, Test, and Staging database environments.
L1 Field Note
AI Engineering Why Agentic AI Costs Explode: Context Size, Tool Calls, MCP Servers, Repo Size, and Retry Loops
Agentic AI systems can quietly accumulate massive API bills due to compounding context windows, retry loops, and unconstrained workspace parsing.
L1 Field Note
Cloud & Platform The Math Behind Database Reserved Instances: When to Wait
Why committing to 3-year database reserved instances too early locks in architectural waste.
L1 Field Note
AI Engineering Codex Credits and Cost Controls for Business Teams
Practical strategies for managing OpenAI Codex API consumption, workspace credits, and governance across your organization.
L1 Field Note
Databases Oracle Cloud BYOL: True Cost Analysis Beyond the Headline Rate
Understanding the financial nuances, OCPU conversions, and hidden costs of bringing your Oracle licenses to OCI.
L1 Field Note
Cloud & Platform BigQuery Cost Optimization: On-Demand vs Slot Commitments
How to stop runaway BigQuery costs by analyzing query scans, enforcing partitions, and moving to capacity-based pricing.
L1 Field Note
AI Engineering The New AI FinOps Model: Seat Cost vs Token Cost vs Agent Runtime Cost
Why traditional SaaS spend models fail for agentic AI, and how platform teams are treating LLM compute like database provisioned IOPS.
L1 Field Note
Databases Oracle to Aurora PostgreSQL: License Cost Elimination in Practice
The engineering reality and ROI of migrating from Oracle to Amazon Aurora PostgreSQL.
L1 Field Note
Databases AWS RDS Oracle and SQL Server: The License Cost Nobody Talks About
Why the default License-Included model on AWS RDS is a financial trap for enterprise database workloads.
L1 Field Note
AI Engineering Context Anxiety and Harness Decay
Why agent harnesses become stale when they overfit today's model weaknesses instead of stable execution contracts.
L1 Field Note
AI Engineering Programmatic Tool Calling for DB Automation
A reference pattern for keeping large database outputs out of model context by using scripts that summarize evidence before the agent sees it.
L1 Field Note
AI Engineering Tool Search vs Loading Every MCP Tool
Why production agents need discoverable tools and context budgets instead of one giant always-loaded MCP surface.
L1 Field Note
Databases Azure Synapse Cost Optimization: DWU Right-Sizing, Serverless, and Hybrid Benefit
How to reduce your Azure Synapse compute bill by right-sizing dedicated pools and offloading to serverless.
L1 Field Note
AI Engineering Token-Efficient Tool Use
How to design agent tool surfaces that preserve context budget for reasoning instead of wasting it on tool metadata and raw output.
L1 Field Note
AI Engineering Application Legibility for Agents
A reference architecture for making logs, metrics, test output, schemas, and deployment history readable by coding agents.
L1 Field Note
Cloud & Platform Database Licensing Cost Across AWS, Azure, GCP, and OCI
A framework for managing commercial database licensing costs across the four major cloud providers.
L1 Field Note
AI Engineering Agent-to-Agent Review Loops
A practical review pattern where one agent creates a change and specialized agents review risk, rollback, security, and observability.
L1 Field Note
Cloud & Platform Cloud Database Cost Engineering: How to Reduce Database, Data Warehouse, and Licensing Spend Across Azure, AWS, GCP, and OCI
A comprehensive framework for reigning in cloud database costs, focusing on licensing, right-sizing, and architectural tradeoffs.
L1 Field Note
AI Engineering Harness Engineering: The 2026 Breakthrough Concept
Why the real engineering surface around agents is the harness of tools, scripts, context, review, and telemetry.
L1 Field Note
Databases Database Runbooks as Agent Contracts
A reference operating model for turning human database runbooks into machine-usable agent contracts.
L1 Field Note
AI Engineering The New Engineer Role: Implementer to Orchestrator
Why agentic coding shifts senior engineering work toward decomposition, verification, and operating-model design.
L1 Field Note
Databases Repo-Embedded Skills for Database Teams
Why database teams should store agent instructions, runbook contracts, and review policies in the repository instead of in memory.
L1 Field Note
Databases Agentic Code Review for Database Repositories
Database repositories contain hidden rules human reviewers know: never add a blocking index at peak hours, never widen IAM without owner approval. Agent review surfaces these violations before merge — without displacing the human judgment that set the rules.
L1 Field Note
AI Engineering Agent Autonomy Ladder: Manual, Confirm, Auto-Approve, Supervised
A governance model for deciding which database and cloud agent actions require approval and which can run automatically.
L1 Field Note
Databases CloudWatch Database Insights: Alerting on What Actually Matters
How to move beyond useless CPU alerts and use CloudWatch Database Insights to track locks, burst balance, and connection queues.
L1 Field Note
AI Engineering Outcome-Based Agent Evaluation vs Transcript Review
A field note on why agent evaluation should measure verified state changes instead of polished reasoning traces.
L1 Field Note
AI Engineering Evals Are the New Unit Tests for Agents
Why database and cloud teams need agent eval harnesses that grade outcomes, not persuasive transcripts.
L1 Field Note
Engineering Fundamentals Alert Fatigue Engineering: How to Build Fewer, Better, Actionable Alerts
A dashboard is not observability, and an alert without a specific action is just operational debt masquerading as monitoring.
L1 Field Note
AI Engineering Can an LLM Safely Recommend Database Performance Changes?
A security and change-control architecture for turning LLM-generated database performance hypotheses into auditable proposals without giving the model production authority.
L1 Field Note
AI Engineering Finding Performance Anomalies Before Asking the LLM
A deterministic anomaly-detection layer that turns normalized database telemetry into attributable, operationally meaningful evidence before LLM correlation begins.
L1 Field Note
AI Engineering Building the Database Incident Evidence Pack for an LLM
A practical evidence contract for collecting, normalizing, redacting, storing, and presenting time-bounded database telemetry to an LLM without sending an uncontrolled production dump.
L1 Field Note
AI Engineering From Metrics to Root Cause: Why Database Troubleshooting Needs Evidence Correlation
Dashboards expose database symptoms one layer at a time. This architecture turns time-bounded telemetry into ranked, testable root-cause hypotheses without giving an LLM production control.
L1 Field Note
Databases Stop Guessing: Solving Invisible Latency with the MySQL Performance Schema
How to move from trial-and-error database tuning to mathematical proof using the underutilized MySQL Performance Schema.
L1 Field Note
Databases Surviving Aurora MySQL: I/O Billing Traps and Small-Instance Cliffs
The brutal realities of scaling Amazon Aurora MySQL, from IOPS billing surprises to network limits on smaller instances.
L1 Field Note
AI Engineering The Agent Should Not Have Your App Credentials
Giving an AI coding agent your application's Postgres credentials is the default mistake — the agent inherits every permission the app has. Database-enforced read-only roles, replica routing, query limits, and project-scoped MCP config are the alternative that actually fails closed.
L1 Field Note
Databases Prometheus + Grafana for Database Engineers: Open-Source Monitoring That Actually Works
How to position Prometheus and Grafana as the open-source baseline for teams that cannot send every byte of database telemetry to managed services.
L1 Field Note
Databases Why pgcrypto Is Not a Full Key Management Strategy
PostgreSQL's pgcrypto is a cryptographic function library, not a key management system. Treating it as one guarantees your encryption keys will eventually leak.
L1 Field Note
Databases MySQL RDS 8.4 vs Amazon Aurora: The Write Scaling and Storage Tradeoff
An architectural comparison of Amazon RDS for MySQL 8.4 against Aurora MySQL, focusing on write path physics, EBS bottlenecks, and distributed storage IOPS.
L1 Field Note
Databases The Database Observability Baseline: What Every DBA Dashboard Must Show
Before you can adopt AI-assisted triage, your database dashboard needs a foundation built on saturation, locking, and lag metrics.
L1 Field Note
AI Engineering Database Security Review for AI Access
Granting an autonomous AI agent access to your database breaks every assumption of traditional RBAC. How to secure databases against unpredictable, unbounded AI queries.
L1 Field Note
Databases MySQL 8.4 LTS: What DBAs Should Check Before Upgrade
MySQL 8.4 is the first long-term support release in the 8.x line — five breaking changes that require verification before any production upgrade.
L1 Field Note
Engineering Fundamentals Consistency Models Your Application Actually Needs
The difference between read committed, repeatable read, and serializable isolation in operational terms — and why most applications are running with weaker guarantees than engineers assume.
L1 Field Note
Databases Vector Search on GPU Databases
A DBA-friendly explanation of how vector search works, why GPUs help, and where vector retrieval fits inside modern database and AI systems.
L1 Field Note
Databases How a 10 Billion Row SQL Query Runs in 200ms on a GPU Database
A DBA-friendly walkthrough of how modern GPU databases execute large analytical SQL queries using columnar storage, parallel scans, and GPU aggregation.
L1 Field Note
Databases Why Databases Are Moving Toward GPU Execution Engines
A practical, DBA-friendly explanation of why modern analytical databases are increasingly using GPUs for scans, joins, aggregations, and AI-adjacent workloads.
L1 Field Note
Engineering Fundamentals SIMD vs SIMT Explained for Database Engineers
A DBA-friendly explanation of SIMD and SIMT using query execution, vectorized processing, and GPU mental models instead of hardware jargon.
L1 Field Note
Engineering Fundamentals CPU vs GPU vs TPU Explained for Database Engineers
How CPU, GPU, and TPU architectures differ in ways that matter for databases and AI workloads — and which compute class to reach for when adding vector search, embedding generation, or GPU-accelerated analytics.
L1 Field Note
Databases Aurora Global Database: What It Solves and What It Does Not
Aurora Global Database delivers sub-second cross-region replication and under-one-minute RTO for disaster recovery — but it is not active-active, and application failover is never automatic.
L1 Field Note
Engineering Fundamentals CAP Theorem in Operational Terms
What CAP theorem actually says about distributed database tradeoffs, why the CP vs AP framing is more useful than the theory, and what it means for your system when the network fails.
L1 Field Note
Engineering Fundamentals Caches, Queues, and Databases: When to Use Each
The decision framework for choosing between a cache, a queue, and a database — including the failure modes that appear when engineers use the wrong one for the job.
L1 Field Note
Databases Why SELECT * Still Hurts Production Systems
SELECT * causes four distinct problems that compound at scale: it prevents covering index usage, transfers unnecessary data, breaks application code silently, and defeats column pruning in analytical systems.
L1 Field Note
Engineering Fundamentals Cardinality Estimation: Why the Query Planner Gets It Wrong
How PostgreSQL estimates row counts, why those estimates are wrong for correlated columns and skewed distributions, and what engineers can do when the planner picks a bad plan.
L1 Field Note
Engineering Fundamentals Index Selectivity: Why Cardinality Changes Everything
Why a low-cardinality index is often worse than no index, how the query planner uses selectivity estimates, and when to build a partial index instead.
L1 Field Note
Databases MySQL Binlog Format: Row vs Statement vs Mixed
Choosing the wrong MySQL binary log format silently breaks replication or bloats the binlog — this is the decision tree for picking the right one.
L1 Field Note
Engineering Fundamentals Reading a Query Plan Without Getting Lost
How to read PostgreSQL EXPLAIN output, what seq scan vs index scan actually means in practice, and the three numbers that matter most in any query plan.
L1 Field Note
Databases Read Replicas Are Not Free Scale
Read replicas add read throughput but they do not reduce write load, do not eliminate replication lag, and silently serve stale data under write bursts — understanding those constraints before you add replicas is the decision engineers skip.
L1 Field Note
Engineering Fundamentals Connection Pooling Explained
Why PostgreSQL connections are expensive, what a connection pool actually does, and the difference between session mode, transaction mode, and statement mode in PgBouncer.
L1 Field Note
Databases MongoDB WiredTiger Cache: Practical Basics
WiredTiger's internal cache is MongoDB's primary memory tier — how to read its metrics, recognize eviction pressure, and size it correctly for your working set.
L1 Field Note
Databases MySQL Cardinality and Index Selectivity
MySQL ignores an index when the optimizer estimates a full scan is cheaper — which happens when cardinality is too low, statistics are stale, or the query shape doesn't match index selectivity. How to diagnose which problem it is and what to do about each.
L1 Field Note
Engineering Fundamentals Replication Lag Explained
What replication lag actually measures in PostgreSQL, the three distinct lag components that most monitoring tools conflate, and which one matters for your RPO.
L1 Field Note
Databases PostgreSQL Statistics: Why the Optimizer Gets It Wrong
PostgreSQL's query planner depends entirely on per-column statistics that go stale after bulk loads — here is what that means for query plan quality and how to fix it.
L1 Field Note
Engineering Fundamentals Checkpoint and Flush: What Your Database Does Before It Can Rest
What a checkpoint actually does in PostgreSQL, why dirty page flush matters for recovery time, and what engineers should monitor to avoid checkpoint pressure.
L1 Field Note
Databases Redis Memory Eviction Policies Explained
Redis has eight eviction policies and a maxmemory limit. The policy you pick determines whether your cache degrades safely or silently corrupts your hit rate under load.
L1 Field Note
Databases MongoDB Index Basics: Why Your Query Became Slow
MongoDB's default behavior is a full collection scan when no index supports the query. Here is what you need to know about single-field, compound, and multikey indexes before your collection grows past 10K documents.
L1 Field Note
Engineering Fundamentals Redo vs Undo: How Databases Recover from Crashes
The two mechanisms databases use to survive crashes — redo brings committed changes forward, undo rolls back uncommitted ones — and why the distinction matters operationally.
L1 Field Note
Engineering Fundamentals B-tree vs LSM Tree: The Storage Engine Tradeoff
Why PostgreSQL and MySQL use B-trees while Cassandra and RocksDB use LSM trees — the read/write tradeoff that determines which storage engine fits your workload.
L1 Field Note
Databases MySQL EXPLAIN: Reading the Plan Without Guessing
How to read MySQL EXPLAIN output systematically — type column, key column, rows estimate, and Extra flags — so you stop adding indexes blindly.
L1 Field Note
Databases MySQL InnoDB Buffer Pool: The First Thing to Check
The InnoDB buffer pool hit ratio and size are the first metrics to verify on any MySQL server — a default 128MB pool on a 32GB machine sends every query to disk.
L1 Field Note
Databases PostgreSQL Autovacuum: What Every Engineer Should Know
Autovacuum is not optional maintenance — it is the mechanism that prevents table bloat and transaction ID wraparound from taking your database offline.
L1 Field Note
Engineering Fundamentals WAL Explained for Database Engineers
What write-ahead logging is, why every ACID database uses it, and what engineers need to know about LSN ordering, crash recovery, and replication lag.
L1 Field Note
Engineering Fundamentals MVCC Explained Like a Database Engineer
How multi-version concurrency control lets readers and writers run without blocking each other — and why misunderstanding it causes table bloat, undo log growth, and stalled vacuums.