DesignFlix 14: Share Queue Work Across Competing Consumers

One image worker is busy resizing uploads. More photos keep arriving, and the oldest job is waiting longer every minute. Can we add workers without asking every worker to resize every photo?

That's the job of competing consumers. Several workers share a source of work, and the system distributes jobs among them. The important detail is what happens when a worker starts a job but doesn't finish it.

Let's use thumbnail generation to see how sharing, acknowledgements and capacity fit together.

Share jobs, not copies of every job

Suppose an application queues one thumbnail task for each uploaded photo. Three workers consume from the same work queue.

One may handle photo 17, another photo 18, and the third photo 19. They aren't three independent subscribers that each need a copy of all three jobs. They are sharing this processing responsibility.

This distinction matters when choosing broker configuration. A broadcast-style subscription and a shared work queue solve different problems. If you create a separate subscription for every worker, you may have created three copies of the work rather than three workers sharing it.

You brought the whole crew?

Acknowledgement marks progress, not a magical guarantee

The worker reads the uploaded photo, produces a thumbnail, saves it and acknowledges the message. That acknowledgement tells the queue the delivery was handled under its protocol.

Now imagine the worker crashes after saving but before acknowledging. Depending on the queue's recovery rules, the task can be delivered again. Another worker may encounter a thumbnail that already exists.

Use a stable task identity and make repeated execution safe. For example, a job can refer to an upload version and a specific thumbnail size. Its output path can identify that same intended result, with storage writes and job state managed to prevent conflicting output.

The right rule is not "every message is processed once forever." We distribute work while still planning for a task to return after uncertain completion. Microsoft's competing-consumer guidance discusses recovery and repeat handling as part of the pattern.

Give each worker a manageable amount of work

If a worker receives hundreds of jobs ahead of time, other workers can be idle while those jobs wait in its local buffer. Large buffers also consume memory and complicate recovery.

Control how many deliveries a worker can hold without acknowledging. RabbitMQ calls this consumer prefetch. A small setting may improve distribution for expensive jobs; too small can waste time waiting for the next delivery. Measure with the task duration and network behaviour your workload has.

This setting isn't a substitute for limiting the worker's actual concurrency. A worker may receive several jobs but process only one at a time, or it may run a bounded number together. Know which limit your client and application enforce.

More workers help only while capacity can grow

Suppose one worker takes about two seconds per photo. Ignoring other limits, its throughput is around half a photo per second. Three similarly equipped workers could approach one and a half per second.

That's a useful starting estimate, not a performance promise. Image decoding costs vary, and all workers may compete for CPU, storage bandwidth or database connections. Adding ten more could make those shared resources slower.

Measure completed jobs per second, oldest-job age, failure rate and downstream resource use. If the queue grows despite idle CPU, investigate the waiting time before assuming the answer is more processes.

Is the worker doing the work slowly, or waiting for another system to let it continue?

Independent photos are easier than ordered updates

Thumbnail jobs for unrelated uploads can usually finish in any order. Changes to one order or account may not be so forgiving.

If an operation requires ordered handling for one entity, choose partitioning, routing and concurrency that preserve that requirement. A queue's delivery order doesn't automatically mean several workers complete jobs in the same order.

Kafka consumer groups share partitions among consumers, rather than handing arbitrary individual records to any available worker. With one consumer group, additional consumers beyond the relevant partitions won't increase active partition-level parallelism. Check the broker's model before applying the work-queue picture literally.

Keep one bad job from owning a worker forever

An unreadable photo can fail repeatedly. Give retries a limit and keep enough context to investigate the failed task. A dead letter queue or other failure store can preserve it while ordinary uploads continue.

Also handle shutdown deliberately. Stop accepting new work, finish or release in-flight jobs under the queue's rules, and allow another worker to recover unfinished tasks. Killing a process isn't a recovery strategy by itself.

Scale the part that actually limits throughput

Competing consumers let us add processing capacity without changing the meaning of each thumbnail job. That is the attraction: a shared backlog and workers we can increase or replace.

To make it useful, pair distribution with bounded in-flight work, safe repeats and measured capacity. Start with a few workers, stop one mid-job, and inspect the result. The system should eventually produce the intended thumbnail without losing the upload or multiplying its effects.