95% Cache Hits, 5% Database Load: How Meta Handles Billions of Requests

October 15, 2026

Imagine you’re building an application that serves 100 million users. Every time someone opens their feed, your backend needs to retrieve posts, profile information, relationships, preferences, and other data. Your database handles the first million requests reasonably well. Then traffic grows. Suddenly, database CPU utilization spikes, query latency increases, and requests begin timing out. Your first instinct might be to buy larger database servers.

But what if the real problem isn’t that your database is slow?

What if your database is doing far too much work in the first place? This is where caching becomes one of the most powerful system design techniques in large-scale infrastructure. And Meta offers a fascinating real-world case study of how to make it work at an enormous scale.

1. The problem: Every request doesn’t need a database query

Consider a simplified social media application.

A user opens their feed, and the application needs to retrieve information such as:

• The names and profile pictures of people they follow.
• Post metadata and engagement counts.
• Friend relationships.
• Frequently accessed account settings.

Some of this data changes frequently. Other information might remain unchanged for hours or days. Yet, without caching, every request could trigger another database query.

The problem becomes especially painful when millions of users request the same popular information.

Suppose a celebrity’s profile receives 50,000 requests per second. If every request hits the database, the application spends enormous resources repeatedly retrieving essentially the same data.

This is a classic example of a system that can become expensive and unreliable even when individual database queries are fast.

2. The solution: Put a cache between the application and the database

The basic idea is straightforward: keep frequently requested data in a fast, temporary storage layer.

Instead of querying the database for every request, the application checks the cache first.

This pattern is commonly known as cache-aside or lazy loading. The application first checks the cache. If the key exists, it returns the cached value. Otherwise, it reads from the database, stores the result in the cache, and returns it. Redis and Memcached are common technologies used to implement this pattern. However, the interesting part isn’t adding a cache. It’s making the cache reliable when millions of requests arrive simultaneously.

3. The real-world case study: Meta’s Memcache infrastructure

Meta, formerly Facebook, has published extensive engineering research on its Memcache deployment and the infrastructure behind it.

Its systems have used Memcache, a distributed in-memory caching system, to reduce the workload on persistent storage and make frequently accessed information available with low latency. The challenge is fundamentally different from caching a few API responses in a small application.

At Meta’s scale, engineers have to address several interconnected problems:

• Extreme request volume: Cache infrastructure must handle enormous numbers of requests without becoming a bottleneck itself.
• Uneven popularity: A small fraction of data can receive a disproportionate share of traffic.
• Consistency: Cached values can become outdated when the underlying database changes.
• Failures: A cache server or network connection can fail, suddenly redirecting traffic toward the database.
• Operational cost: Keeping everything in memory is expensive, so the system needs to use memory efficiently.

Meta’s engineering work on Memcache illustrates an important principle: at large scale, a cache is not merely a performance optimization. It becomes a distributed infrastructure system that must be designed for failures, traffic patterns, and operational efficiency.

4. The first major problem: Cache stampedes

Imagine a popular product’s details are cached for five minutes. At the five-minute mark, the cached entry expires. Suddenly, 20,000 requests arrive for that product. Without coordination, all 20,000 requests may discover the cache miss and query the database simultaneously. This is called a cache stampede.

The solution is to coordinate cache fills.

For example, the application can use a per-key lock or single-flight mechanism so that only one request fetches the missing value. Other requests wait briefly for the result or receive a controlled fallback.

Other useful techniques include:

• TTL jitter: Randomize expiration times to prevent many keys from expiring together.
• Early refresh: Refresh frequently accessed entries before they expire.
• Stale-while-revalidate: Serve a slightly outdated value while refreshing it in the background, when the data’s correctness requirements permit it.

The key insight: a cache miss can be more dangerous than it looks because many requests can experience it at the same time.

5. The second problem: A hot key can overwhelm the cache

Caching does not automatically eliminate bottlenecks.

Suppose one cache key receives 100,000 requests per second, while most other keys receive only a handful.

Even if the cache cluster has plenty of overall capacity, the server responsible for that particular key can become overloaded. This is known as the hot-key problem.

Possible solutions include:

• Local caching: Keep a short-lived copy in each application server’s memory. This reduces repeated network requests to the distributed cache, but introduces additional invalidation and consistency challenges.
• Replication: Keep frequently accessed data on multiple cache servers so requests can be distributed across replicas.
• Request coalescing: Combine concurrent requests for the same key within an application instance.
• Traffic shaping: Rate-limit, prioritize, or shed nonessential requests when a dependency is overloaded.

The correct solution depends on the data. A public profile can often tolerate short-lived replication. A rapidly changing financial balance requires much stricter correctness guarantees.

6. The third problem: What happens when the cache fails?

This is one of the most important system design interview questions. Suppose your application normally handles 100,000 requests per second, and 95% of reads are served by the cache. Under normal conditions, the database receives only about 5,000 reads per second. Now imagine the cache becomes unavailable. If the application immediately sends every request to the database, the database suddenly faces up to 100,000 reads per second instead of 5,000.

That is a potential 20-fold increase in read traffic. These are illustrative numbers, not measurements from Meta. The failure can trigger a vicious cycle:

1. The cache becomes unavailable.
2. Database traffic spikes.
3. Database latency increases.
4. Application requests remain open longer.
5. Connection pools fill up.
6. Retries generate even more traffic.
7. The entire service becomes unavailable.

This is a classic cascading failure. A resilient design needs to account for it before production traffic arrives.

Useful safeguards include:

• Circuit breakers to stop repeatedly calling an unhealthy dependency.
• Bounded retries with exponential backoff and jitter to avoid synchronized retry storms.
• Concurrency limits to protect the database from sudden surges.
• Stale-value fallbacks for data that can safely be served temporarily out of date.
• Load shedding to preserve essential operations when capacity is exhausted.

The most important principle is this:

Your database must be able to survive the failure of the system designed to protect it.

That means capacity planning should consider degraded operation, not just normal traffic.

7. Keeping cached data correct

Caching introduces another difficult trade-off: performance versus freshness. Imagine a user changes their account settings. The database contains the new value, but the cache still contains the old one. One common strategy is cache invalidation after a database update.

A simplified flow looks like this:

1. Update the database.
2. Invalidate the corresponding cache key.
3. Let the next request fetch the latest value and repopulate the cache.

But this approach has a subtle race condition. A request that started before the update might read an old database value and write it into the cache after invalidation. Production systems need to consider these races, along with delayed events, retries, and failed invalidation messages. For more demanding workloads, engineers may use versioned cache entries, write-through strategies, event-driven invalidation, or carefully designed consistency protocols.

There is no single strategy that works for every type of data.

The lesson is that cache design must follow the correctness requirements of the data, not just its access frequency.

8. What the architecture might look like at scale

Putting the pieces together, a simplified large-scale read path could look like this:

Notice that caching is only one component of the design. Request coordination, replication, data consistency, monitoring, and failure protection all influence whether the system remains stable under pressure.

9. How would you estimate the impact?

Let’s use a hypothetical service receiving 100,000 read requests per second.

Assume its cache hit rate is 95%.

Now imagine the hit rate falls from 95% to 80%.

Database reads increase from 5,000 to 20,000 per second: four times the previous database load.

That is why monitoring the cache hit rate matters, but it is not sufficient on its own. A cache can have an excellent hit rate and still suffer from hot keys, high network latency, memory pressure, or expensive misses.

A robust monitoring dashboard should track:

• Cache hit rate, broken down by endpoint and key category.
• p95 and p99 request latency.
• Eviction rate and memory utilization.
• Hot keys and uneven traffic distribution.
• Database query volume and connection-pool saturation.
• Cache errors, timeouts, and retry volume.

These metrics help engineers understand whether the cache is actually protecting the system.

10. The bigger lesson from Meta’s engineering

The most interesting part of this case study is not that in-memory lookups are faster than database queries. That is the easy part. The hard part is maintaining the benefits of caching when the workload is enormous, data changes continuously, and individual components inevitably fail. Meta’s published work on Memcache demonstrates why large-scale caching requires careful engineering of the entire data path, rather than simply installing Redis or Memcached and hoping the problem disappears.

For engineers designing systems today, the practical lessons are clear:

1. Reduce unnecessary work before adding capacity. Avoid repeatedly fetching identical information.
2. Design for cache misses. Expiration and cold starts can create sudden traffic spikes.
3. Protect against correlated failures. Retries, hot keys, and dependency failures can amplify one another.
4. Treat consistency as a product requirement. Not every piece of data can tolerate the same degree of staleness.
5. Measure the failure path. A system that works only when every cache server is healthy is not resilient.

Caching is often introduced as a simple performance optimization. At scale, it becomes a central part of the system’s reliability strategy. And that is the difference between making a request faster and designing infrastructure that can keep working when millions of requests arrive at once.

#systemdesign #ai #ml #software

← Back to all posts