Data Ingestion Recipe Parsing Workflows

Celery vs RQ for Menu-Sync Queues

This page helps a food-tech developer choose the task queue that runs menu-sync, cost roll-up, and report jobs. It is a decision companion to async batch processing workflows; read that for the queue-based architecture and idempotency contract, then use this to pick between the two dominant Python options — a choice that trades operational simplicity against routing and scheduling power.

Both run functions off the request path with retries. Celery is a full task framework with multiple brokers, complex routing, scheduled beats, and workflows; RQ (Redis Queue) is deliberately minimal — Redis only, a tiny API, and far less to operate.

Prerequisites and Data Contract

  • Python 3.11+; Celery 5.x (with Redis or RabbitMQ) or RQ 1.x (Redis only).
  • Tasks must be idempotent — keyed on a natural id — because either queue can deliver a job more than once, exactly as the parent guide requires.
  • The output contract is the same: an enqueued job that a worker executes with bounded retries and a dead-letter path for permanent failures.
Celery versus RQ for menu-sync queues A comparison of two task queues. Celery supports multiple brokers such as Redis and RabbitMQ, rich routing to dedicated queues, scheduled beats, and chained workflows, at the cost of more configuration and operational surface. RQ is Redis-only with a minimal API, is simple to run and reason about, but offers fewer routing and scheduling primitives and no built-in periodic scheduler. Celery RQ • Redis or RabbitMQ brokers • rich routing + priority queues • beat scheduler + workflows • more to configure and operate • best for complex estates • Redis only • minimal API, easy to reason about • no built-in periodic scheduler • low operational surface • best for simpler pipelines

Step-by-Step Implementation

Step 1 — Define an idempotent sync task in Celery

Celery’s decorator adds retries and routing; the task stays idempotent on a natural key.

from celery import Celery

app = Celery("menu", broker="redis://localhost:6379/0")


@app.task(bind=True, max_retries=4, acks_late=True)
def sync_location_menu(self, location_id: str, run_id: str) -> str:
    try:
        return upsert_menu(location_id, run_id)      # idempotent on (location_id, run_id)
    except TransientError as exc:
        raise self.retry(exc=exc, countdown=2 ** self.request.retries)

Step 2 — Define the same task in RQ

RQ enqueues a plain function; retries are configured on enqueue.

from redis import Redis
from rq import Queue, Retry

q = Queue("menu", connection=Redis())


def enqueue_sync(location_id: str, run_id: str) -> None:
    q.enqueue(
        upsert_menu, location_id, run_id,
        retry=Retry(max=4, interval=[2, 4, 8, 16]),   # bounded backoff
        job_id=f"sync:{location_id}:{run_id}",        # idempotency guard
    )

Step 3 — Choose by operational complexity

Pick Celery when the estate needs dedicated priority queues (report generation must not starve behind a nightly bulk import), a periodic scheduler for nightly roll-ups, or chained workflows where one stage triggers the next. Pick RQ when the pipeline is a handful of job types on Redis and operational simplicity is worth more than routing power — fewer moving parts means fewer failure modes at 3 a.m. Many teams start on RQ and migrate to Celery only when routing or scheduling needs actually arrive, not preemptively. Either way, the async batch processing workflows idempotency and dead-letter contract is unchanged.

Verification and Validation

  • Idempotency under redelivery. Enqueue the same (location_id, run_id) twice; the sync must converge to one result. In RQ, a duplicate job_id is rejected; in Celery, the natural-key upsert absorbs the retry.
  • Retry backoff. Force a TransientError and confirm both retry with increasing intervals and stop at the max, then dead-letter.
  • Scheduling need. If you require periodic runs, confirm Celery beat fires on schedule; RQ needs an external scheduler (rq-scheduler or cron), which is a point in the decision.
  • Isolation. Confirm a slow bulk job does not block a report job — trivial with Celery priority queues, requires separate RQ queues and workers.

Gotchas and Edge Cases

  • acks_late and visibility timeout. Celery with acks_late=True re-delivers a job if a worker dies mid-task; ensure the task is idempotent or a crash double-applies. The visibility timeout must exceed the longest task or Redis re-queues a still-running job.
  • RQ has no native periodic scheduler. Nightly roll-ups need rq-scheduler or an external cron enqueuing jobs; do not assume RQ schedules on its own.
  • Result backend growth. Both can persist results in Redis; unbounded result retention fills memory. Set a result TTL, or disable the backend for fire-and-forget syncs.
  • Serialization of large payloads. Passing a whole menu payload through the broker is slow and memory-heavy. Enqueue an id and let the worker fetch the data, keeping the queue message small.

For library specifics, see the official Celery documentation and RQ documentation.