Catalog Sync Workflow: Database, Search Index, CDN, and Cache Invalidation
A catalog update is not complete when the database transaction commits; it is complete when every reader that can show the product has converged, or has been explicitly allowed to serve stale data.
Situation
Product catalogs have become multi-surface systems. A price change may be read from the primary database by checkout, from a search index by the browse page, from a CDN edge by a product detail page, and from an application cache by recommendation or inventory services.
Each surface exists for a good reason. The database gives transactional truth. The search index gives relevance and filtering. The CDN absorbs global read traffic. The cache keeps hot paths fast and isolates dependencies. None of these systems share the same consistency model.
That means catalog sync is not a background detail. It is part of the product correctness boundary. If the architecture treats it as a best-effort side effect, the user experience will eventually split: checkout rejects a price shown on the page, search returns deleted products, category pages show stale availability, or a CDN edge keeps serving a retired SKU after the origin has been fixed.
The Problem
The common failure is coupling the catalog write path to too many downstream effects.
A simple implementation writes the database row, updates the search document, purges CDN URLs, deletes cache keys, and returns success. It feels direct, but it creates a distributed transaction without transaction semantics. If the database commit succeeds and the search update times out, the system now needs to know whether to retry, reconcile, or roll back. If CDN invalidation is slow, the product page can remain stale even though every internal API is correct. If the cache delete happens before commit, readers can refill old data.
The reverse design is also dangerous. If sync is fully asynchronous but invisible, operational teams lose the ability to answer basic questions: Which SKUs are behind? Which downstream system is blocking convergence? Is the stale page caused by search lag, cache refill, CDN propagation, or a missing event?
The core question is this: how do you make catalog updates fast enough for product teams while preserving a clear correctness model across database, search, CDN, and cache?
The Catalog Sync Control Plane
The answer is to separate the catalog write from catalog propagation, while making propagation observable, replayable, and bounded by explicit freshness contracts.
The database remains the source of truth. Every catalog mutation writes both the business row and an outbox event in the same transaction. A sync worker reads the outbox, writes derived projections, and records per-target delivery state. Search indexing, CDN invalidation, and cache invalidation are treated as independent subscribers with their own retry policies.
flowchart TD
A[admin change — price update] --> B[database transaction — catalog row]
B --> C[outbox event — committed with row]
C --> D[sync dispatcher — ordered work]
D --> E[search index writer — product document]
D --> F[cache invalidator — key set]
D --> G[CDN invalidator — URL set]
E --> H[delivery ledger — search status]
F --> I[delivery ledger — cache status]
G --> J[delivery ledger — CDN status]
H --> K[read freshness view — catalog convergence]
I --> K
J --> K
This is not just an event-driven architecture. The important part is the control plane around the events.
First, the outbox is the durable handoff. A catalog change is not considered emitted because an HTTP call was attempted. It is emitted because an outbox record exists in the same commit as the catalog mutation.
Second, the dispatcher owns idempotency. Every downstream write carries a stable catalog version, such as product_id plus catalog_version. Search indexing can safely retry the same document version. Cache invalidation can safely delete the same key more than once. CDN invalidation can deduplicate by path set and version window.
Third, the read paths are explicit about freshness. Checkout should read the database or a strongly controlled projection. Browse can tolerate search lag if the UI and ranking contracts allow it. CDN-backed pages need short TTLs, versioned URLs, or active invalidation for fields that cannot remain stale.
Fourth, reconciliation is a first-class workflow. A periodic job compares database versions against search document versions, cache metadata, and CDN invalidation completion records. This catches missed events, poison messages, and downstream outages that retry queues alone may hide.
In Practice
Context. The documented pattern is the transactional outbox: persist the state change and the message in the same database transaction, then relay the message asynchronously. This pattern is widely described by Chris Richardson at microservices.io as a way to avoid dual writes between a database and a message broker.
Action. For catalog sync, the action is to treat the outbox table as the only source of propagation work. The application does not call Elasticsearch, Redis, or CloudFront inside the request transaction. It commits the catalog row and the outbox event, then lets workers advance downstream projections.
Result. The result is not instant consistency. The result is recoverable inconsistency. If the search cluster is unavailable, the database remains correct, the outbox backlog grows, and operators can see exactly which catalog versions have not reached search.
Learning. The practical lesson is that asynchronous does not mean best effort. It means the system accepts temporary lag in exchange for durable retry, replay, and isolation from downstream failures.
Context. PostgreSQL behavior reinforces the same lesson. A committed row is durable according to the database configuration, but LISTEN and NOTIFY are not a durable queue. Notifications can wake workers, but they should not be the only record of catalog work.
Action. Use database polling, logical decoding, or a durable queue fed by the outbox as the real work source. Notifications can reduce latency, but workers must be able to recover from the table itself.
Result. A worker restart no longer loses product updates. The backlog is still present in the database, ordered by commit metadata or monotonically assigned outbox IDs.
Learning. Do not confuse a signal with a ledger. Catalog propagation needs a ledger.
Context. Elasticsearch and OpenSearch are near-real-time search systems. Indexed documents are not necessarily visible to search immediately after the write; refresh behavior controls when changes become searchable.
Action. Store the catalog version in every indexed document and expose sync lag by comparing the latest database version with the searchable version. Use forced refresh only for narrow operational cases, not as the default path for every product edit.
Result. Search freshness becomes measurable instead of anecdotal. Product teams can decide whether a five-second lag is acceptable for title edits and whether price or availability requires a different path.
Learning. Search is a projection, not the catalog authority.
Context. CDN invalidation is also not a transaction. Providers such as Amazon CloudFront document invalidation as an asynchronous operation. Edge caches may continue serving old content until expiration or invalidation propagation completes.
Action. Use versioned asset URLs where possible, short TTLs for volatile catalog HTML, and targeted invalidations for pages whose stale content creates business risk. Record invalidation request IDs and completion state.
Result. CDN behavior stops being mysterious. A stale product page can be traced to a known invalidation request, an expected TTL, or a missing path mapping.
Learning. CDN freshness must be designed into URL and TTL strategy; it cannot be patched reliably with broad emergency purges.
Where It Breaks
| Failure mode | Why it happens | Mitigation |
|---|---|---|
| Database updated, search stale | Search write failed or refresh has not exposed the document | Outbox retry, versioned documents, search lag dashboards |
| Cache refilled with old data | Cache delete happened before commit or readers raced the writer | Commit first, then invalidate; use versioned cache keys for critical reads |
| CDN serves retired page | Edge TTL or invalidation propagation delay | Versioned URLs, targeted invalidation, volatile content TTL limits |
| Worker poison message blocks queue | One malformed SKU or payload fails repeatedly | Dead letter queue, per-target isolation, replay tooling |
| Reindex overwrites newer data | Bulk job writes an older document version | Compare versions before write, reject stale projection updates |
| Operators cannot explain staleness | No per-target delivery ledger | Track catalog version, target, status, attempt count, and last error |
The hardest tradeoff is deciding which surfaces are allowed to be stale. A product description can usually tolerate propagation delay. Price, legal restrictions, and availability often cannot. The architecture should encode that distinction rather than pretending all catalog fields have the same consistency requirements.
For high-risk fields, route reads through stronger sources. Checkout should validate against the database or a strongly consistent pricing service. Search can display a product, but checkout must make the final decision. CDN pages can show cached marketing content, but price and availability may need client-side hydration from a fresher API.
What to Do Next
Problem: Catalog updates fail operationally when the database, search index, CDN, and cache are treated as one implicit transaction.
Solution: Use a transactional outbox, independent downstream subscribers, idempotent versioned writes, and a delivery ledger for every propagation target.
Proof: The design follows documented behavior of durable database commits, near-real-time search visibility, asynchronous CDN invalidation, and repeatable cache invalidation patterns.
Action: Start by adding catalog_version to the database row, search document, and cache payload. Then add an outbox table and a dashboard that shows, for each changed SKU, the latest version committed and the latest version visible in search, cache, and CDN.