Scaling, Queues, and Backpressure
Why autoscaling reacts too late, Little's law applied to a real queue, what a growing backlog actually means, and the retry storm that turns a blip into an outage.
Scaling questions are really queueing questions. The arithmetic is simple and almost nobody does it out loud, which is why doing it stands out.
Autoscaling reacts after the fact
aws autoscaling describe-policies --auto-scaling-group-name app-asg \
--query 'ScalingPolicies[].{name:PolicyName,type:PolicyType,target:TargetTrackingConfiguration.TargetValue,cooldown:Cooldown}' \
--output table--------------------------------------------------------------
| DescribePolicies |
+------------+----------------+-----------+------------------+
| cooldown | name | target | type |
+------------+----------------+-----------+------------------+
| 300 | cpu-target | 70.0 | TargetTracking |
+------------+----------------+-----------+------------------+# reaction.py
DETECT_S = 180 # CPU alarm needs 3 data points at 60s
DECIDE_S = 30 # ASG evaluates and calls RunInstances
BOOT_S = 90 # instance boots
WARM_S = 60 # app starts, JIT warms, connection pool fills
HEALTH_S = 30 # load balancer health checks pass
total = DETECT_S + DECIDE_S + BOOT_S + WARM_S + HEALTH_S
print(f"detect {DETECT_S}s + decide {DECIDE_S}s + boot {BOOT_S}s "
f"+ warm {WARM_S}s + health {HEALTH_S}s")
print(f"total time to serve traffic: {total}s = {total/60:.1f} minutes")
BASE_RPS, SPIKE_RPS, CAPACITY_PER = 2_000, 10_000, 500
have = BASE_RPS / CAPACITY_PER
need = SPIKE_RPS / CAPACITY_PER
print(f"\nhave {have:.0f} instances ({have*CAPACITY_PER:,} rps capacity)")
print(f"need {need:.0f} instances ({SPIKE_RPS:,} rps arriving)")
print(f"shortfall for {total/60:.1f} min: {SPIKE_RPS - have*CAPACITY_PER:,} rps dropped or queued")$ python reaction.py
detect 180s + decide 30s + boot 90s + warm 60s + health 30s
total time to serve traffic: 390s = 6.5 minutes
have 4 instances (2,000 rps capacity)
need 20 instances (10,000 rps arriving)
shortfall for 6.5 min: 8,000 rps dropped or queuedSix and a half minutes. For a spike that arrives in thirty seconds, autoscaling does nothing useful — you serve it with the fleet you already had, or you shed load.
“Autoscaling handles gradual growth and diurnal patterns well. It does not handle spikes, because the reaction time is minutes and a spike is seconds. For known events I’d use scheduled scaling ahead of time; for unknown ones I’d carry headroom and shed load gracefully rather than pretend the scaler will catch it.”
What actually shortens each term:
TERM FIX TYPICAL GAIN
detect request-count target instead of CPU; 180s → 60s
CPU lags the load it is caused by
decide predictive or scheduled scaling to zero for known events
boot pre-baked image instead of boot-time config 90s → 30s
warm warm pool of stopped instances 60s → ~0
health faster interval, fewer required checks 30s → 10s
everything containers instead of VMs 390s → ~40s
everything serverless → cold start onlyNaming the request-count target instead of CPU is worth a mark on its own: CPU is a lagging indicator of a load that already arrived.
Little’s law
# little.py
# L = lambda * W items in system = arrival rate x time in system
def items_in_system(arrival_rate, time_in_system):
return arrival_rate * time_in_system
def required_concurrency(rps, latency_s):
return rps * latency_s
for rps, latency in [(1000, 0.050), (1000, 0.200), (1000, 2.000), (5000, 0.100)]:
conc = required_concurrency(rps, latency)
print(f"{rps:>6,} rps at {latency*1000:>6.0f} ms → {conc:>7,.0f} concurrent requests"
f" → {conc/250:>5.1f} instances at 250 threads each")$ python little.py
1,000 rps at 50 ms → 50 concurrent requests → 0.2 instances at 250 threads each
1,000 rps at 200 ms → 200 concurrent requests → 0.8 instances at 250 threads each
1,000 rps at 2,000 ms → 2,000 concurrent requests → 8.0 instances at 250 threads each
5,000 rps at 100 ms → 500 concurrent requests → 2.0 instances at 250 threads eachSame throughput, 40x the concurrency, purely because latency went from 50 ms to 2 s. Latency is a capacity multiplier, which is why a slow dependency exhausts a connection pool long before it exhausts CPU.
The interview use is the reverse direction — sanity-checking a claim:
# "the queue has 2 million messages and we process 500/sec"
backlog, rate = 2_000_000, 500
print(f"drain time: {backlog/rate/3600:.1f} hours at current rate")
for multiple in (2, 5, 10):
print(f" with {multiple}x consumers: {backlog/(rate*multiple)/3600:.1f} hours")drain time: 1.1 hours at current rate
with 2x consumers: 0.6 hours
with 5x consumers: 0.2 hours
with 10x consumers: 0.1 hoursThat takes ten seconds and turns “the queue is backed up” into a decision.
What a growing queue means
aws sqs get-queue-attributes --queue-url https://sqs.us-east-1.amazonaws.com/123/jobs \
--attribute-names ApproximateNumberOfMessages ApproximateAgeOfOldestMessage \
--query 'Attributes'{
"ApproximateNumberOfMessages": "1847293",
"ApproximateAgeOfOldestMessage": "4127"
}1.8 million messages, oldest is 69 minutes old. The depth is the alarming number; the age is the useful one, because it is what a user experiences.
# queue.py
producers_per_sec, consumers_per_sec, hours = 1200, 1000, 6
net = producers_per_sec - consumers_per_sec
print(f"in {producers_per_sec}/s, out {consumers_per_sec}/s, net +{net}/s")
for h in range(0, hours + 1, 2):
depth = net * h * 3600
age_min = depth / consumers_per_sec / 60 if consumers_per_sec else float('inf')
print(f" t+{h}h depth {depth:>10,} oldest message {age_min:>7,.0f} min behind")$ python queue.py
in 1200/s, out 1000/s, net +200/s
t+0h depth 0 oldest message 0 min behind
t+2h depth 1,440,000 oldest message 24 min behind
t+4h depth 2,880,000 oldest message 48 min behind
t+6h depth 4,320,000 oldest message 72 min behindA 20% shortfall in consumer capacity produces a backlog growing by 1.4 million messages every two hours, indefinitely. The queue is not absorbing the problem — it is recording it.
The three responses, and when each is right:
SCALE CONSUMERS correct when the shortfall is temporary and the work
is parallelisable. Check the downstream dependency
can take 5x the load before you scale 5x.
SHED LOAD correct when the work is not all equally valuable.
Drop or defer low-priority messages; keep the rest
current. Requires priority to exist in the design.
BACKPRESSURE correct when producers can be slowed. Reject at the
edge with 429 rather than accepting work you cannot
do — a fast rejection is better than an unbounded wait.Backpressure is the answer most candidates never give, and it is often the right one. An unbounded queue converts an overload into a latency problem that is invisible until someone looks at the age metric.
The retry storm
# retries.py
NORMAL_RPS = 1000
MAX_RETRIES = 3
print(f"{'scenario':<34} {'load on dependency':>20}")
print(f"{'healthy, no retries needed':<34} {NORMAL_RPS:>19,}")
naive = NORMAL_RPS * (1 + MAX_RETRIES)
print(f"{'degraded, naive retry x3':<34} {naive:>19,} ({naive/NORMAL_RPS:.0f}x)")
# clients also time out and the caller above them retries
cascade = NORMAL_RPS * (1 + MAX_RETRIES) ** 2
print(f"{'two layers each retrying x3':<34} {cascade:>19,} ({cascade/NORMAL_RPS:.0f}x)")
# circuit breaker: after N failures, stop calling entirely
print(f"{'circuit breaker open':<34} {'~0':>19} (fail fast, shed load)")$ python retries.py
scenario load on dependency
healthy, no retries needed 1,000
degraded, naive retry x3 4,000 (4x)
two layers each retrying x3 16,000 (16x)
circuit breaker open ~0 (fail fast, shed load)Sixteen times the load, arriving exactly when the dependency is least able to serve it. Retries multiply across layers, which is why a service mesh retrying, plus a client library retrying, plus application code retrying, is a genuine outage amplifier.
The three mitigations, in order of how much they help:
import random
def backoff(attempt, base=0.1, cap=30):
exponential = min(cap, base * 2 ** attempt)
return random.uniform(0, exponential) # full jitter
for a in range(5):
print(f"attempt {a}: max {min(30, 0.1 * 2**a):>5.1f}s, actual {backoff(a):>5.2f}s")attempt 0: max 0.1s, actual 0.07s
attempt 1: max 0.2s, actual 0.03s
attempt 2: max 0.4s, actual 0.31s
attempt 3: max 0.8s, actual 0.55s
attempt 4: max 1.6s, actual 1.42s- Jitter — without it, every client that failed at the same moment retries at the same moment, producing a synchronised wave. Full jitter (uniform between 0 and the backoff) spreads them.
- Retry budget — cap retries at a percentage of total requests, so amplification is bounded regardless of how many things are failing.
- Circuit breaker — after N consecutive failures, stop calling entirely for a cooldown and fail fast. This is the only one that reduces load to near zero, and it is what actually stops the storm.
Two more worth naming: retry only idempotent operations, and retry only once per layer — decide where retries live and remove them everywhere else.
Scaling the database
TECHNIQUE WHAT IT SOLVES WHAT IT COSTS
read replicas read-heavy load replication lag; stale reads
connection pooling connection exhaustion one more thing to operate
caching repeated reads invalidation, and a cold-cache
thundering herd
vertical scaling everything, for a while a hard ceiling and downtime
to resize
sharding write throughput cross-shard queries and
transactions become hard
CQRS divergent read/write shapes eventual consistencyRead replicas are the first reach and the one with the sharpest edge. Replication lag means a user can write and then not see their own write:
SELECT now() - pg_last_xact_replay_timestamp() AS replication_lag; replication_lag
-----------------
00:00:00.847847 milliseconds. Long enough that a redirect after a form submission reads stale data — the classic “I saved it and it disappeared” bug.
The mitigation is read-your-own-writes: route a user’s reads to the primary for a short window after they write, or pin their session. Naming that as the specific problem, rather than saying “eventual consistency”, is what shows you have hit it.
Recognising it
SIGNAL ANSWER
"traffic spikes at 9am every day" scheduled scaling, not reactive
"unpredictable viral spikes" headroom + load shedding; scaling is too slow
"the queue is backing up" compare producer and consumer rates first
"how do we handle 10x traffic?" Little's law, then which component saturates
"a dependency got slow and we went down" retry storm — circuit breaker
"users report saving and losing data" replication lag; read-your-own-writes
"how do we scale writes?" sharding, and name what it costs
"how do we scale reads?" cache, then replicas, in that order
"the cache went down and so did we" thundering herd; request coalescingPractice
1. Add up autoscaling's reaction time.
detect 180 + decide 30 + boot 90 + warm 60 + health 30 = 390s = 6.5 minFor a spike arriving in thirty seconds, you serve it with the fleet you already have. Scheduled scaling and headroom are what work.
2. Apply L = λW at 1,000 rps with 50 ms and 2 s latency.
50 concurrent → 2,000 concurrent, a 40x difference at identical throughputLatency is a capacity multiplier. A slow dependency exhausts a connection pool long before it exhausts CPU.
3. Project a queue where producers exceed consumers by 20%.
t+6h: 4.3M deep, oldest message 72 minutes behindThe queue records the problem rather than absorbing it. Watch the age metric, not the depth.
4. Compound retries across two layers.
1,000 rps → 16,000 rps at 3 retries per layerRetries multiply. A circuit breaker is the only mitigation that takes the load to near zero.
Next: cost optimisation — the FinOps round, and where the money actually goes.