Autoscaling and Elasticity

Adding and removing capacity automatically as demand changes.

Elasticity, introduced in the lesson on the cloud and AI shift, is capacity that grows and shrinks with demand instead of being sized in advance for a guess about peak load. Autoscaling is the mechanism that delivers it.

Horizontal vs. vertical scaling

Vertical scaling means making an existing instance bigger: more CPU, more memory, on the same machine. It requires no changes to how the application is built, but it has a ceiling (there's a largest instance size available), and doing it live usually means downtime or a restart. Horizontal scaling means adding more instances running the same code, side by side behind a load balancer. It scales far past any single machine, and each instance can be added or removed without touching the others, but the application has to support running many copies at once: no state stored only in one instance's memory, no assumption that "the" server is a single machine.

Reactive vs. predictive scaling

Reactive scaling responds to what's happening right now (CPU usage crossed a threshold, request queue length grew) and adds or removes capacity after the fact. Scheduled scaling changes capacity at set times, such as more instances at 9 a.m. on weekdays, and also covers one-time events like a marketing push. Predictive scaling forecasts demand from historical load and adds capacity before it arrives. AWS EC2 Auto Scaling, Google Compute Engine managed instance groups, and Azure virtual machine scale sets each offer it, based on daily and weekly patterns. Scheduled and predictive scaling help most when instances take a long time to start, because reactive scaling can lag behind the spike.

Request-based vs. queue-based scaling

Request-based scaling reacts to inbound HTTP traffic: requests per second, concurrent connections, CPU load caused by serving them. It's the natural fit for a web application or API. Queue-based scaling reacts to the depth of a work queue instead, how many jobs are waiting to be processed, and fits background or asynchronous work. A system commonly needs both: request-based scaling for the part serving live traffic, queue-based scaling for the part chewing through background jobs.

Cold starts, minimum instances, and scale-to-zero

A serverless function or serverless container that scales to zero instances when idle costs nothing while unused, but the next request has to wait for a fresh instance to start, initialize, and become ready. That delay is a cold start, and it ranges from tens of milliseconds to several seconds depending on the runtime and how much setup code runs before the first request is handled. Setting a minimum instance count above zero keeps that many instances warm at all times, trading ongoing cost for fewer cold starts. That's a common choice for a user-facing API, and unnecessary for something that runs a few times a day and can tolerate the delay.

A related, often-overlooked limit is connection limits: every instance holds some number of open connections (to a database, to another service), and a fleet that scales out fast can hit a connection ceiling on the other end well before it hits any limit on itself.

Scaling past the database's limits

Autoscaling adds capacity to whichever layer is configured to scale, and it's easy to configure only the layer that's visibly under load (usually the application servers) without noticing that a downstream dependency has a hard ceiling of its own.

A concrete version of this: a burst of traffic triggers autoscaling on the application tier, which goes from 4 instances to 40 in a few minutes. Each instance opens its own pool of connections to the database. The database has a fixed maximum connection count, a usually modest limit that doesn't scale itself just because the application tier did. Once the connection ceiling is hit, new connection attempts start failing, application instances start timing out waiting for a database connection to free up, and those timeouts cause the load balancer to see slow, failing instances, which can trigger scaling out even further, adding still more instances fighting over the same fixed number of database connections.

flowchart TD
  A[Traffic spike] --> B[App tier scales: 4→40 instances]
  B --> C[Each instance opens a DB pool]
  C --> D{DB max connections reached?}
  D -- Yes --> E[Connections rejected]
  E --> F[App times out on DB]
  F --> G[LB sees failing instances]
  D -- No --> H[System handles the spike]

The application tier scaled as designed and the system still went down, because the database's connection ceiling was not part of the scaling story.

The usual fix is a connection pooler. A connection pooler (PgBouncer for Postgres is the common example, and several managed database services now offer a built-in equivalent) sits between the application and the database, holding a small, stable set of connections to the database and multiplexing many more application-side connections through them. Every new instance connects to the pooler instead of the database, so the application tier can scale without exhausting the database's connections. A simpler first control is capping the maximum instance count at what the database can serve.

Databases can autoscale too, but they have ceilings of their own. Aurora Serverless v2 derives its connection limit from the maximum capacity you configure, and DynamoDB on-demand can throttle traffic that more than doubles its previous peak within 30 minutes.

The same trap extends past databases: any downstream dependency with a fixed or slower-scaling capacity (a third-party API with a rate limit, a cache with fixed memory, another internal service that hasn't been scaled the same way) can become this kind of wall. Autoscaling only helps the layer it's applied to. Reliability in Distributed Systems covers the patterns (backpressure, load shedding, circuit breakers) that keep a bottleneck like this from taking the whole system down with it.

Previous ← Compute Options

Next Storage Fundamentals: Object, Block, and File →