How Razorpay refreshes its data warehouse 10x faster
Strong data-infra example of reframing materialized-table refresh as graph maintenance, plus a nice case for batch-plus-incremental beating naive streaming when stateful joins get too expensive.
Logged at IST: 2026-07-15 10:45 IST
What it is: Piyush Goel sharing Razorpay Engineering’s writeup on refreshing warehouse facts 10x faster with graphs and indexes
Gist: Razorpay moved from expensive full-refresh fact generation toward incremental fact maintenance by treating each denormalized fact as a dependency graph. They pair change-driven processing with secondary indexes on the lake, graph traversal to discover affected ancestors and descendants, and selective runtime joins for high-cardinality dimensions. The result is much faster warehouse refreshes with lower compute cost, restored historical coverage, and less dependency on the warm store.
Newsletter angle: Strong data-infra example of reframing materialized-table refresh as graph maintenance, plus a nice case for batch-plus-incremental beating naive streaming when stateful joins get too expensive.