Core Cloud Architecture · Part 7 of 13

Caching

Caching patterns, and the consistency cost of keeping a second copy of the data.

A cache keeps a copy of frequently accessed data somewhere faster to read from than its original source, usually in memory instead of a database's disk-backed storage. It cuts latency and takes read load off the source. A cache is a second copy of the data, and the two copies can disagree.

Cache-aside

In the cache-aside pattern (also called lazy loading), the application checks the cache first; on a hit, it uses the cached value directly. On a miss, it reads from the database, then writes that value into the cache before returning it, so the next request for the same data is a hit. This is the most common pattern because it's simple to reason about and only caches data that's being requested; nothing is pre-loaded speculatively.

Write-through

In write-through caching, every write goes to both the cache and the database, so the cache stays in step with the database for data written this way, unless one of the two writes fails. It trades a small amount of write latency (every write now touches two systems instead of one) for stronger consistency than cache-aside provides on its own.

TTL and invalidation

A TTL (time to live) is an expiration set on a cached value; once it elapses, the cache treats the entry as gone, forcing the next read to go back to the source and refresh it. It keeps a value from getting too stale without guaranteeing that it isn't stale, because the cache is never told the data changed. Invalidation removes or updates a cache entry when its underlying data changes, so the cache reflects a change sooner than a TTL alone. It covers only changes that go through code that invalidates the entry: Microsoft's cache-aside guidance notes that an external process can change the data store at any time, and the change doesn't appear in the cache until the item loads again.

Hot keys

A hot key is a single cache entry receiving a disproportionate share of all traffic: a trending post, a widely shared link, a piece of global configuration read on every request. Even a cache built to handle enormous aggregate load can struggle when nearly all of that load lands on one key, since most caching systems shard data across multiple nodes by key, and a hot key means one shard doing far more work than the rest. Mitigations include replicating the hot value across multiple cache nodes deliberately, or adding a low-TTL layer of local (in-process) caching in front of the shared cache for the handful of keys that need it.

Managed caching services

What caching costs: consistency risk

The latency benefit is nearly automatic, since reading from memory is faster than reading from a database. The cost that needs design attention is consistency: the moment a cache exists, there are two answers to "what is this value right now," and they can diverge.

A stale read happens when the underlying data has changed but the cache hasn't caught up: a user updates their profile, reloads the page, and briefly sees the old value, because the read hit a cache entry that hasn't expired or been invalidated. Cache/DB divergence is the more serious version of the same problem: a bug in invalidation logic, a race between a write and a cache refresh, or a cache node that fell out of sync means the cache keeps serving a wrong value indefinitely.