DesignFlix 13: Slow the Producer Before the Queue Fills

A queue can absorb a rush of work. It can't make a permanently slower consumer catch up by itself.

Suppose a producer sends 200 jobs a second while workers finish 120. The queue gains 80 jobs every second, or 4,800 in a minute. We haven't removed the overload; we've given it somewhere to wait.

Backpressure lets the receiving side influence how much more work arrives. Let's look at a pause-and-resume controller, then ask what happens when the producer can't pause.

A buffer buys time, not extra throughput

Imagine a pipeline that generates preview images for uploaded videos. A burst of uploads can wait in a bounded queue while workers process them.

That works if the burst ends and the workers can drain the backlog. If arrivals stay above completion capacity, even a large buffer eventually fills. Jobs also wait longer, so the user experience can become poor before memory or storage runs out.

Track arrival rate, completion rate and oldest-job age together. Queue depth tells us how much work is waiting; age tells us how long somebody may already have waited.

Pause early, resume after some breathing room

One simple controller uses two thresholds. At the high watermark, pause or slow the producer. At the lower watermark, let it resume.

For an illustrative buffer with capacity for 100 similar jobs, we might pause at 80 and resume at 30. Using two thresholds avoids repeatedly switching on and off whenever the depth fluctuates around one number.

Here's the state decision in Python:

def next_paused(depth, paused):
    if depth >= 80:
        return True
    if depth <= 30:
        return False
    return paused

assert next_paused(80, False) is True
assert next_paused(79, True) is True
assert next_paused(30, True) is False
assert next_paused(31, False) is False

The middle range keeps the previous state. This is just the decision logic; the actual pipeline must communicate and honour it. We also need room for work already in flight when the pause signal arrives.

I said... slow down.

Let the signal travel back to the sender

Backpressure only helps if an upstream component responds. A worker may stop requesting more deliveries, a stream writer may wait for available capacity, or an API may refuse additional work under a documented overload policy.

In Node.js writable streams, a write() result of false asks the producer to wait for the drain event before continuing. The Node.js backpressure guide explains why ignoring that signal lets buffers and memory grow.

The same idea applies across longer pipelines. If one stage pauses but an earlier stage keeps accepting unlimited uploads into another buffer, we've moved the backlog rather than controlled it. Follow the signal far enough to find where new work is admitted.

Choose a policy when the source cannot wait

Some input arrives whether we're ready or not. A live feed can't always be paused without losing information, and a user-facing API can't hold every incoming request indefinitely.

We need an explicit choice:

• Reject excess work and tell callers how to retry.
• Keep only the newest update when old updates no longer matter.
• Drop a permitted class of optional work.
• Spill to durable storage, with a defined capacity and recovery plan.

Those policies suit different data. Dropping an old live preview can be acceptable; dropping a committed order is a different decision. Backpressure doesn't choose what the business may lose.

When the buffer is full, which promise are we still able to keep?

Count bytes and downstream capacity too

A hundred jobs can mean a hundred small records or a hundred large payloads. A count limit alone may not protect memory. Bound payload sizes, buffered bytes and in-flight processing as appropriate to the pipeline.

Adding workers can increase completion capacity, but check what they depend on. If the database or image-processing service is already saturated, extra concurrency can increase timeouts and retries instead of clearing the queue.

Separate those decisions from the controller. Pausing input can protect the system now; scaling capacity may help later if the limiting resource can actually grow.

Test recovery as well as overload

Slow the consumers in a test and watch arrivals, depth, age and memory. Check that the high threshold changes admission before the hard limit, and that the low threshold resumes work without constant toggling.

Then restore consumer capacity. Does the backlog drain? Do callers retry gradually, or does resuming release another rush? A controller that survives the first peak but immediately creates a second one still needs work.

Backpressure makes overload visible in the flow of work rather than hiding it in an expanding queue. A useful design has a bounded buffer, a signal the sender honours, and a clear policy for input we cannot accept. Start there before treating a bigger queue as the answer.