Skip to content

Queue Consumer Stuck

A queue consumer (BaseConsumer implementation) has stopped processing messages. The queue is building up; no messages are being acknowledged.

Symptoms

  • Queue depth increasing with no drain
  • No ack() calls visible in OTel spans
  • Consumer container is running but subscribe() loop has stalled
  • ping() returns True (broker is reachable) but messages are not processed

Diagnosis

1. Check if the handler is hanging.

modal app logs <app-name> | grep "consumer\|subscribe\|ack\|nack"

Look for the last message that was started but never acknowledged. A handler that raises an unhandled exception without triggering nack() will stall the consumer.

2. Check broker connectivity.

asyncio.run(consumer.ping())   # True = broker reachable, False = connection lost

If ping() returns False, this is a Redis Connection Lost or broker outage, not a stuck consumer.

Recovery

If the handler is hanging on a specific message:

Stop the consumer, drain or skip the problem message at the broker level, restart.

modal app stop <app-name>
# Skip / dead-letter the stuck message via broker admin UI or CLI
modal deploy modal_app.py

If the consumer loop crashed silently:

modal app stop <app-name>
modal deploy modal_app.py

Modal will cold-start a new container. record_lifecycle_event("cold_start") will fire and be visible in metrics.

Prevention

Wrap all handler logic in try/except AdapterError and call nack() explicitly on failure. Never let an exception propagate out of the handler without an explicit ack() or nack() call.