Python Celery alternatives (and prior art from other ecosystems): research for Potatoq¶

Research date: 2026-10-07. Versions checked: Dramatiq 2.2.1 (2026-09), RQ 2.12 (2026-08), Huey 3.4.0 (2026-09), arq 0.28.0 (2026-04, maintenance-only), Taskiq 0.13.0 (2026-09), Procrastinate 3.10.0 (2026-09), SAQ (master), Chancy 0.25.1, Hatchet v0.110.x, django-q2 1.11.1, django-tasks 0.12.0, Django 6.0/6.1, River (Go), Oban 2.24.

Defaults were checked against source code wherever possible (raw GitHub files). Where a number comes from memory or a secondary source, the text says so.

0. Why people leave Celery (the problems Potatoq has to fix)¶

Most-reacted Celery issues (search):

HN comments on Celery's defaults: - "long standing bugs / poor defaults, like prefetching tasks so they can get stuck behind long running tasks" (HN 30127847); "weird performance gotchas (like the workers prefetch jobs which is terrible if they aren't all the same size)" (HN 34339618); one user found prefetch "a deal breaker" for long tasks and switched to RQ (HN 30129111). - "It was difficult to debug, going through Celery's layers of code that try to make various backends present the same interface" (Dramatiq Show HN). - Celery's own author (asksol) acknowledged that the defaults are not tuned for small jobs (HN 1710446). - From the Hatchet launch thread: "the observability is pretty bad. Even if you use Celery Flower, it still just doesn't give me enough insight" (HN 39643136).

Dramatiq's motivation page lists these Celery problems: tasks are acked as soon as a worker pulls them; delayed tasks sit on the normal queue and are held in worker memory (which breaks queue-size-based autoscaling); the code is "spread across 3 different projects (celery, billiard and kombu) and it's impenetrable", with runtime stack-frame manipulation; there is no global prioritization.

Celery defaults for reference (from memory of the Celery docs; worth re-verifying in the Celery-specific research): worker_prefetch_multiplier=4, task_acks_late=False, task_reject_on_worker_lost=False, no time limit, Redis visibility_timeout=3600, result_expires=1 day, max_retries=3 / default_retry_delay=180s (used only when self.retry() is called explicitly), confirm_publish=False.

1. Dramatiq¶

Links: docs, motivation, cookbook, GitHub. LGPL (it started as AGPL plus a paid commercial license, and the AGPL drew heavy pushback in the 2017 Show HN).

Defaults (from source)¶

Note on retry span: 20 retries from a 15 s base, doubling, jittered and capped at 7 days, add up to roughly 3–6 weeks before a message is dead-lettered. That is a long time for most web-app tasks.

Architecture worth copying¶

• Redis Lua dispatch script (dispatch.lua). Each queue is a list of message IDs (ns:queue) plus a hash of payloads (ns:queue.msgs). Each worker has an ack set (ns:__acks__.<worker_id>.<queue>). A heartbeats ZSET (ns:__heartbeats__) holds worker liveness, and the dead-letter queue is queue.XQ (a ZSET by timestamp plus a msgs hash). The fetch command pops up to prefetch IDs into the worker's ack set atomically. Maintenance moves unacked messages owned by dead workers (heartbeat older than timeout) back to the queue and purges expired DLQ entries.
• Delay queues (queue.DQ) are separate queues, so the main queue length stays an honest autoscaling signal.
• Middleware architecture with hooks (before_enqueue, before_process_message, after_process_message, after_skip_message, worker boot/shutdown, and so on). Retries, time limits, age limits, callbacks, pipelines, results and Prometheus are all middleware.
• Rate limiters are separate from tasks: ConcurrentRateLimiter (a distributed mutex when limit=1), BucketRateLimiter and WindowRateLimiter, on Redis or Memcached backends. When the limit is hit, the default raises and the task is retried with backoff.
• TimeLimit injects TimeLimitExceeded into the worker thread with PyThreadState_SetAsyncExc. The source admits that it "can't cancel system calls" and only fires when the thread next acquires the GIL.
• Composition: group, pipeline (a.message() | b.message()), pipe_ignore.
• StubBroker for tests; join(fail_fast=True) became the default in 2.0.

What they got wrong / community complaints¶

• Delayed messages are still held in worker memory. On both brokers the worker consumes the .DQ queue with delay_prefetch (8000 by default) and keeps messages in an in-process PriorityQueue until their ETA, then re-enqueues them to the canonical queue and acks the DQ copy. That is better than Celery (separate queue) but has the same failure modes:
• #431 "Delayed messages are being duplicated on work queue": copy-then-ack is not atomic, so a crash between the two steps duplicates the message.
• PR #434: with many delayed messages, the Lua stack overflows in the prefetch.
• An external guide recommends lowering the default delay prefetch to avoid cold-start memory pressure (markaicode).
• confirm_delivery=False on RabbitMQ is the same unsafe default as Celery's.
• Probabilistic maintenance (1 in a million) makes dead-worker recovery latency unpredictable on quiet queues.
• No built-in cron (the docs point to APScheduler or periodiq). No Redis priorities (#412). No pipelines of groups (#103). Actor registry is coupled to the broker (#324).
• Configuring logging inside the library was a top complaint (#48). Lesson: never call logging.basicConfig.
• Bus factor: Bogdan posted "Maintainer Wanted" #680, and a team now maintains it (LincolnPuzey and others). 2.0 shipped in Nov 2025.
• Asyncio support was added late, through the AsyncIO middleware.

2. RQ (python-rq)¶

Links: docs, workers, defaults.py, CHANGES.

Defaults¶

Design¶

• Fork per job. The worker forks a "work horse" for every job, which gives isolation and no memory leaks across jobs. SimpleWorker runs in-process. SpawnWorker (2.2, 2025) covers Windows and macOS. rq worker-pool (2.5) runs N workers from one CLI.
• Registries: Started, Finished, Failed, Deferred (dependencies), Scheduled, Canceled (registries). depends_on provides job dependencies.
• 2.x: Redis Streams for results, multiple executions per job, a built-in scheduler (--with-scheduler), Repeat (2.3), rq cron / CronScheduler (2.4/2.5), and Valkey support. rq-scheduler is now mostly superseded.
• Dequeue strategies: default (strict queue order), round_robin, random.

Complaints / what they got wrong¶

• No acks. Abandoned jobs (worker killed mid-job) are moved to the FailedJobRegistry with AbandonedJobError, not requeued (exceptions docs). That is effectively at-most-once unless you retry from the failed registry.
• No in-process concurrency. #45 "Worker concurrency?" has been open since 2012. Fork per job costs CPU: Redash reported RQ workers needing much more CPU than Celery for the same load (dev.to/redash).
• macOS fork-safety crash (#1418, 101 reactions). Lesson: prefork on macOS needs OBJC_DISABLE_INITIALIZE_FORK_SAFETY or spawn.
• Zombie workers (#787), no per-queue rate limit (#725), cancel running jobs (#684), key prefix (#1855).
• Redis only.
• Community: praised as "rock solid for years", "5 minutes setup", with an API that is "pretty small and comprehensible" (HN 35533590, HN 20433061). One user noted the circular dependency between app, Redis and Postgres (HN 21944154).

3. Huey (Charles Leifer)¶

Links: API, consumer, guide, releases.

Defaults¶

Strengths¶

• Periodic tasks built in (@huey.periodic_task(crontab(...))). Only one consumer should run the scheduler (-n disables it).
• Immediate mode (huey.immediate = True) runs tasks synchronously with in-memory storage, which suits tests.
• lock_task (TTL added in 3.4), expires, revoke, RetryTask, CancelExecution, signals (SIGNAL_ERROR, SIGNAL_INTERRUPTED, SIGNAL_TIMEOUT, and others), and pipelines with .then().
• 3.0 (Apr 2026) added group() / chord(), timeout (SIGALRM for processes, gevent.Timeout for greenlets, cooperative only for threads), a fixed-window rate_limit(), and low-latency results via notify_result=True (3.0.0). 3.2 added a Django admin stats dashboard.
• Very small; no broker needed with SQLite (HN 28568473, HN 32731032).

What they got wrong¶

• At-most-once by design. "Huey does not guarantee at-least-once delivery" and in-flight tasks "will be lost" if the consumer dies. The suggested workaround is a SIGNAL_INTERRUPTED handler that re-enqueues, which only helps on SIGTERM, not SIGKILL or OOM (consumer docs, recipes).
• pickle by default; destructive result reads surprise people; one default worker; no retries by default.
• Maintainer tone was criticized as "dismissive/borderline rude" (HN 21950594).

4. arq (Samuel Colvin / Pydantic)¶

Links: docs, worker.py, #437 Future plan, #510 Maintenance only.

Defaults¶

Design¶

• The queue is a ZSET scored by the time the job should run, so deferred jobs (_defer_until, _defer_by) and immediate jobs share one structure. _job_id gives uniqueness: a job with the same ID cannot be re-enqueued until its result expires.
• "Pessimistic execution." A job stays in the queue until it completes. A worker claims it with WATCH/MULTI and PSETEX in_progress_key (timeout+10s). If the worker dies, the in-progress key expires and another worker re-runs the job.
• raise Retry(defer=ctx['job_try'] * 5) for custom backoff; cron jobs built in; asyncio only.

What they got wrong¶

• Polling every 0.5 s plus optimistic WATCH/MULTI claiming means every worker contends on the head of one ZSET. SAQ claims to be up to 8× faster because it uses BLMOVE (SAQ README), and streaq claims 14× (#437 comment).
• Abandoned. Colvin wrote "no one is more disappointed that we haven't found a way to work more on arq than me", and since 2025 it is in maintenance-only mode (#510). Long-standing asks include cancel/abort (#246, #290), monitoring (#297), and newer Redis versions (#454). Downloads are still about 3.5M/month, so an async Redis user base is waiting for a home.
• The planned redesign in #437 is a useful spec for Potatoq: ParamSpec type safety, enqueue by name when worker code isn't importable (tiangolo: "That's the main reason I had gone for Celery over RQ"), and running sync def functions on threads alongside async ones (also tiangolo).

5. Taskiq¶

Links: architecture, CLI, middlewares, FastAPI, taskiq-redis.

Defaults¶

Strengths¶

• A clean broker / result-backend / middleware / schedule-source abstraction. Middleware hooks are pre_send, post_send, pre_execute, on_error, post_execute and post_save.
• Dependency injection (TaskiqDepends, FastAPI-style). taskiq-fastapi init(broker, "app:app") lets tasks reuse FastAPI dependencies such as Request and app.state.
• InMemoryBroker for tests. Async-native, and kicks with .kiq().

What they got wrong¶

• Data-loss footguns in broker choice. taskiq-redis PubSubBroker and ListQueueBroker "don't support acknowledgements… the message is going to be lost". Only RedisStreamBroker acks (taskiq-redis).
• Results and retries are off unless you wire them up. Missing liveness/readiness commands (#240, #303), unique jobs (#271), and per-process Prometheus metrics (#590).
• Many knobs and small separate packages; quality varies between broker packages.

6. Procrastinate (Postgres)¶

Links: discussions, reference, retry, stalled jobs, Django, HN 2022.

Defaults¶

Design¶

• Postgres with LISTEN/NOTIFY and SELECT … FOR UPDATE SKIP LOCKED. Job states: todo, doing, succeeded, failed, cancelled, aborted. Async at the core; sync tasks run through asgiref.sync_to_async, and there are separate sync and async defer paths.
• lock: jobs that share a lock string run sequentially (serialized execution). queueing_lock: at most one job with a given lock can be in todo (enqueue-time dedup through a unique index).
• Periodic tasks via @app.periodic(cron=...), deduplicated in the DB (procrastinate_periodic_defers).
• Django: uses the Django DB settings, ships Django migrations, read-only admin models, and the management command worker.

What they got wrong¶

• Stalled-job recovery is DIY. You write a periodic task that calls get_stalled_jobs() and then retry_job() (docs).
• No retries, no timeout and no pruning by default (jobs are kept forever). Concurrency 1. Polling fallback of 5 s.
• The docs say they'd "like to develop real monitoring tools before we call this really ready for production".
• Community sentiment is very positive: "moved all our celery tasks to procrastinate… it has been great" (HN 46086198); "codebase is like one-tenth that of Celery" (HN 46084615); "Django with Procrastinate… works like a dream" (HN 47330075).

7. SAQ (Simple Async Queue, Toby Mao)¶

Links: GitHub, job.py.

• Backends are Redis and Postgres (SKIP LOCKED plus LISTEN/NOTIFY). The Redis backend uses BLMOVE/RPOPLPUSH instead of polling, claiming under 5 ms latency against arq's 0.5 s and "up to 8x faster than arq". Includes a web UI, heartbeats and a sweeper for abandoned jobs, and cron.
• Job defaults: timeout=10 s (aggressive), heartbeat=0 (disabled), retries=1 (that is, one attempt and no retry), ttl=600 s (result and info retention), retry_delay=0.0, retry_backoff=False, priority=0 (Postgres only).
• Lesson: a 10 s default timeout is too aggressive for a general-purpose replacement, but low-latency blocking dequeue is the right call.

8. Chancy (TkTech, Postgres)¶

Links: GitHub, docs, HN 46084714.

• Postgres-only and async-first, with only psycopg3 as a dependency. Features: "rate limiting, global uniqueness, timeouts, memory limits, mix asyncio/processes/threads/sub-interpreters in the same worker, workflows, cron jobs, dashboard, metrics, django integrations, reprioritization, triggers, pruning, Windows support, queue tagging".
• Queues are DB rows with concurrency, executor, rate limit and polling interval, and can be created, paused or changed at runtime. Workers can be tagged so that queues route to particular machines (GPU, OS). Plugins: Pruner, Recovery, Leadership, Cron, Workflow, Metrics, Reprioritize.
• Planned django-tasks integration. Small adoption (about 290 stars) and pre-1.0 (0.25), but the feature list is close to "Oban for Python".

9. Hatchet¶

Links: docs, timeouts, retries, HN v1, HN launch.

• A separate Go engine plus Postgres (RabbitMQ was optional and removed as a requirement in v1). Workers connect over gRPC, so the engine polls Postgres on behalf of workers and individual workers don't hammer the DB. MIT licensed, with a cloud offering.
• Defaults: schedule timeout 5 min, execution timeout 60 s (refreshable and additive), retries 0. Backoff via backoff_factor and backoff_max_seconds. NonRetryableException.
• Python API: @hatchet.task(name=..., input_validator=PydanticModel), def fn(input, ctx: Context). DAG workflows, durable tasks, concurrency keys (fairness per tenant), rate limits, priority.
• Scaling notes from the authors: "a simple Postgres queue utilizing FOR UPDATE SKIP LOCKED doesn't cut it at this scale" (5k+ tasks/s, 25k TPS). They fixed it with range partitioning of time-series tables, hash partitioning for event updates, separate monitoring and queue tables, buffered reads/writes flushed every 10 ms, and identity columns instead of UUIDs ("UUIDs caused some headaches… index bloat").
• Downsides: heavy (a separate service), and it is an orchestration platform rather than a drop-in for Celery.

10. django-q2¶

Links: configure.

• Defaults: workers = CPU count; recycle=500 tasks; timeout=None; retry=60 s (re-delivery for unacked tasks); max_attempts=0 (infinite); ack_failures=False; save_limit=250 successful results; queue_limit=workers**2; ORM broker poll=0.2 s; bulk=1. Brokers: ORM, Redis, SQS, MongoDB.
• Main footgun: timeout must be less than retry, otherwise a long task is re-delivered while still running and executes twice (docs, django-q #183). The scheduler also duplicates tasks across clusters. Lesson: the visibility timeout (lease) must derive from the task time limit automatically and never be set independently.

11. Django 6.0/6.1 Tasks framework (DEP 0014): the compatibility target¶

Links: topic guide 6.0, reference 6.1, 6.1 release notes, DEP 14 announcement, SC vote thread, django-tasks backport, django-tasks-db, critique.

API (verbatim signatures)¶

from django.tasks import task, task_backends, default_task_backend

task(*, priority=0, queue_name="default", backend="default", takes_context=False, **kwargs)  # **kwargs new in 6.1 -> backend.task_class
Task.using(*, priority=None, backend=None, queue_name=None, run_after=None)  # returns a new Task
Task.enqueue(*args, **kwargs) -> TaskResult        # args must be JSON-serializable
Task.aenqueue(*args, **kwargs)
Task.get_result(result_id) / Task.aget_result(result_id)
Task.call(...) / Task.acall(...)                   # run inline

TaskResult: task, id (str, < 64 chars), status, enqueued_at, started_at, last_attempted_at,
            finished_at, backend, errors (list[TaskError]), worker_ids, return_value
            (ValueError unless SUCCESSFUL), attempts, is_finished, refresh()/arefresh()
TaskResultStatus: READY, RUNNING, SUCCESSFUL, FAILED
TaskError: exception_class, traceback
TaskContext (takes_context=True, first arg): task_result, attempt (starts at 1)

BaseTaskBackend: options, task_class (6.1), supports_defer, supports_async_task,
                 supports_get_result, supports_priority;
                 enqueue(task, args, kwargs) / aenqueue / get_result / aget_result / validate_task(task)
Exceptions: InvalidTask, InvalidTaskBackend, TaskResultDoesNotExist, TaskResultMismatch
Priority range: -100..100 (higher = sooner), default 0. Queue default "default".
Settings: TASKS = {"default": {"BACKEND": "...", "QUEUES": [...], "OPTIONS": {...}}}
Built-ins: ImmediateBackend (default; runs inline), DummyBackend (stores, never runs; .results, .clear())
Signals: task_enqueued, task_started, task_finished

Key facts for Potatoq¶

• There is no worker and no production backend in core. There are also no retries, scheduling or periodic tasks. DEP 14 deliberately defines "a shared API contract between worker libraries and developers". Potatoq should ship a first-class TASKS backend.
• ENQUEUE_ON_COMMIT was removed from core before 6.0 because a global setting couldn't support multi-database setups. The docs now recommend transaction.on_commit(partial(my_task.enqueue, ...)) explicitly. Adam Johnson stressed during the DEP vote that on_commit is essential, and he also warned about workers holding many future-dated tasks in memory (forum).
• Django 6.1 (Aug 2026) forwards @task(**kwargs) to backend.task_class, so Potatoq can define PotatoqTask(Task) with max_retries, time_limit, unique_key, max_age and so on, and users keep writing @task(...). 6.1 also makes Task and TaskResult picklable.
• JSON round-trip semantics are strict: a tuple comes back as a list and the result is FAILED in the docs example. Potatoq's Django backend must use the same JSON rules so that switching backends doesn't change behaviour.
• django-tasks 0.12 split DB and RQ backends into django-tasks-db and django-tasks-rq.
• Other django.tasks backends in the wild or planned: Chancy (planned), Steady Queue (a Solid Queue port for Django, internals).

12. Oban (Elixir, Postgres/SQLite/MySQL): design insights¶

Links: Oban, Worker, unique jobs, Pruner, Lifeline, 1M jobs/min, Postgres notifier.

• States: available, scheduled, executing, retryable, completed, cancelled, discarded.
• Worker defaults: max_attempts=20; backoff is exponential with a 15 s minimum plus jitter; timeout=:infinity; priority=0 (0–9, 0 highest); queue=:default.
• Return values as control flow: :ok, {:error, reason} (retry), {:cancel, reason} (no retry), {:snooze, seconds} (reschedule without consuming an attempt). Potatoq should expose raise Snooze(60), raise Cancel(...) and raise Retry(in_=...).
• Pruner: interval 30 s, max_age 60 s, limit 10,000 per run. Completed, cancelled and discarded jobs are deleted almost immediately, which keeps the table small.
• Lifeline: rescues executing jobs after 1 h purely by time. The docs admit it can duplicate genuinely running jobs; Pro uses producer heartbeats instead.
• Stager: every 1 s, promotes scheduled/retryable jobs to available (limit 5,000) and notifies queues.
• Unique jobs: defaults are period 60 s, fields [:worker, :queue, :args], states :successful. OSS Oban enforces this with transactional locks plus queries, not a unique constraint, so it can race; Pro's Smart engine uses real constraints. Duplicates return {:ok, job} with conflict?: true, and replace: can update fields on conflict.
• Notifier: Postgres LISTEN/NOTIFY by default, or the Erlang PG notifier. Insert notifications moved from DB triggers into application code, are debounced per queue, and can be disabled (insert_trigger: false) because triggers added overhead on bulk inserts and conflicted with poolers.
• Leadership: a DB peer (an oban_peers row with expiry) elects one node for cron, pruning and staging.
• Engines: Basic (Postgres), Lite (SQLite ≥ 3.37), Dolphin (MySQL 8.4). This shows that one abstraction can cover PG, SQLite and MySQL.
• Performance: index-only scans, compound indexes and debounced fetching (cooldown). Async (batched) acking gave +200–210% throughput. Peak 32k jobs/s on one queue and node.
• shutdown_grace_period 15 s.

13. River (Go, Postgres): design insights¶

Links: brandur.org/river, HN, unique jobs, maintenance services, retries, client.go, notifier pattern.

• Transactional enqueue is the main selling point. "When used well, transactions and background jobs are a match made in heaven and completely sidestep a whole host of distributed systems problems." A job inserted in the app's transaction becomes visible exactly when the data commits, and a rollback discards it.
• Defaults (client.go): JobTimeout=1 min, JobStuckThreshold=10 s, FetchCooldown=100 ms, FetchPollInterval=1 s, MaxAttempts=25. Retry backoff is attempt^4 + ±10% jitter (1 s, 16 s, … about 3 weeks in total). Retention is completed 24 h, cancelled 24 h, discarded 7 days. SoftStopTimeout is unset (wait indefinitely).
• Maintenance services run only on the elected leader: Cleaner (retention above); Rescuer (stuck running jobs after 1 h, or JobTimeout + 1 h); Scheduler (every 5 s, promotes scheduled to available); Periodic Enqueuer; Queue Cleaner (24 h); Reindexer (daily REINDEX INDEX CONCURRENTLY, one index at a time).
• Unique jobs: ByArgs, ByPeriod, ByQueue, ByState (default: all states except cancelled/discarded). Enforced by a unique_key column with a partial unique index filtered by a state bitmask, so uniqueness is real and race-free. Caveat: uniqueness covers insertion only; execution is still at-least-once.
• Snooze: JobSnooze(d) doesn't consume attempts.
• Why Postgres is fine now (vs. 2017): SKIP LOCKED (9.5), REINDEX CONCURRENTLY (12), B-tree dedup (13), bottom-up B-tree deletion on insert (14), which removes most of the index bloat from high-churn queue tables. Also batch fetch and update, COPY FROM for bulk insert, and the binary protocol.
• Notifier pattern: one LISTEN connection per process, with fan-out to subscribers over buffered non-blocking channels. That connection needs session pooling; everything else can use PgBouncer in transaction mode.

14. pgmq¶

Link: GitHub.

• An SQS-like extension: send (with delay), read(vt) (visibility timeout), pop, archive, delete, set_vt, read_with_poll. One table per queue (pgmq.q_<name>) plus an archive table (a_<name>). Optional unlogged queues and partitioned queues via pg_partman. FIFO/grouping and topic routing. PG 14–18; used by Supabase and Tembo.
• Lesson: the visibility-timeout lease (vt) is simpler than "running plus heartbeat plus rescuer". It needs an extension, though, which managed providers may not offer, so Potatoq should use plain SQL.

15. Postgres-as-queue literature¶

• Brandur 2015/2017, "Postgres Job Queues & Failure By MVCC" (post, HN): a long-running transaction anywhere in the DB pins the MVCC horizon. Dead tuples pile up and every lock attempt walks thousands of invisible index entries. Lock time went from under 10 ms to over 100 ms, and the queue grew to 60k jobs in an hour. Mitigations: more specific predicates, terminating long-lived transactions, and not sharing the DB across teams.
• Brandur, "Transactionally Staged Job Drains" (post): insert into a staged_jobs table in the app transaction, then an enqueuer process (repeatable read) moves jobs to Redis or Sidekiq and deletes them. You get transactional enqueue semantics with a non-DB broker, and never "job runs before commit". This is the outbox pattern; Potatoq can offer it for Redis/RabbitMQ when the app DB is Postgres or SQLite.
• PlanetScale 2026, "Keeping a Postgres queue healthy" (post): reproduced the death spiral on PG 18. Concurrent analytics pinned the horizon and produced 383k dead tuples and over 300 ms lock time at 800 jobs/s. Batching (10 per transaction) helped the baseline (1.3–3 ms) but not the horizon problem. statement_timeout, idle_in_transaction_session_timeout and transaction_timeout (PG17) are "blunt instruments". The fix is to protect VACUUM by limiting competing long queries.
• Rich Yen 2026 (post): MultiXact SLRU contention under many workers, queue tables "tens of gigabytes when the actual 'live' data was only a few megabytes", and "every lock/unlock is a full WAL-logged transaction". Fine below about 100 concurrent workers.
• DBOS 2026 (post): 30k+/s with SKIP LOCKED, READ COMMITTED (REPEATABLE READ caused serialization failures), and partial indexes on the enqueued state that include the sort columns, so the dequeue needs no sort step.
• Recall.ai 2025, "LISTEN/NOTIFY does not scale" (post, Simon Willison): a transaction that issued NOTIFY takes a global lock during commit (AccessExclusiveLock on object 0 of class 1262) that is held through fsync, which serializes all NOTIFY-ing commits. Under many writers this caused outages. DBOS published a rebuttal, "LISTEN/NOTIFY actually scales", arguing it works if used carefully.
• PgQ/pgque (thebuild.com): snapshot-diff batches, consumers that only read, and 3 rotating tables with TRUNCATE, giving zero dead tuples at the cost of about 1–2 s latency. A good fit for event fan-out, not for job dispatch.
• Adriano, "Choose Postgres queue technology" (post): LISTEN/NOTIFY plus FOR UPDATE SKIP LOCKED is enough for most apps; against the "cargo cult of scale"; build queue-agnostic abstractions (his Neoq).
• SKIP LOCKED (Craig Ringer, 2ndQuadrant, "What is SELECT … SKIP LOCKED for in PostgreSQL 9.5?"; the original URL now redirects to EDB's homepage. See also depesz and Paquier). The canonical pattern is DELETE FROM queue WHERE id = (SELECT id FROM queue ORDER BY id FOR UPDATE SKIP LOCKED LIMIT 1) RETURNING *, done inside the transaction that does the work, so a rollback automatically returns the job. Caveats: it gives an inconsistent view (fine for queues, not for general use), and holding a transaction open while a long job runs is exactly what pins the MVCC horizon. For long jobs, claim (UPDATE state=running with a lease), commit, then work.
• Rails Solid Queue (GitHub) and its Django port Steady Queue (internals) split the hot "ready" table from the cold "jobs" table (ready, scheduled, claimed, blocked, failed executions) so the polled table stays tiny. Defaults: worker poll 0.1 s, dispatcher 1 s / batch 500, process heartbeat 60 s, alive threshold 5 min, clear_finished_jobs_after 1 day. Concurrency limits use a semaphores table. They recommend a separate DB, which gives up transactional enqueue.

16. Redis queue reliability¶

• Lists: RPOP/BRPOP loses messages if the consumer crashes after receiving one. The documented reliable queue pattern is LMOVE/BLMOVE into a processing list, LREM on completion, and a monitor that re-pushes stale items (LMOVE docs). Used by SAQ, Sidekiq Pro super_fetch and RQ (in spirit). The weaknesses are that there is no per-item claim timestamp (you need a side ZSET or hash), LREM is O(N), and blocking commands can't be called inside Lua.
• Streams plus consumer groups: XREADGROUP > gives at-least-once delivery through the PEL, with XACK, XPENDING delivery counts and XAUTOCLAIM key group consumer min-idle start COUNT 100 (since 6.2; scans up to COUNT×10 PEL entries; since 7.0 it removes trimmed or deleted IDs from the PEL; it increments the delivery count unless JUSTID is used) (XAUTOCLAIM, streams). Redis 8.2 adds XACKDEL/XDELEX (ack plus delete); 8.8 adds XNACK (release a message for redelivery). Weaknesses:
• No delayed delivery and no priorities; you still need a ZSET for ETA/countdown.
• Idle-time-based reclaim conflates "slow job" with "dead consumer". Long jobs must periodically XCLAIM … JUSTID themselves to reset idle time (a lease extension).
• Valkey lacks the 8.x additions.
• Taskiq's RedisStreamBroker is the main Python user.
• Lua-script designs: Dramatiq (lists, an ack set per worker, a heartbeat ZSET, a DLQ ZSET), arq (a ZSET queue with optimistic WATCH, which contends at scale), and BullMQ (wait list, active list, delayed ZSET, a priority ZSET, a lock key with TTL per active job, a stalled-job checker, and a "marker" key that workers block on with BZPOPMIN, which gets around the "no blocking in Lua" limit; BullMQ docs, BullMQ Python vs RQ benchmark).
• Durability: Redis with AOF everysec can lose about 1 s of acknowledged writes. WAIT/WAITAOF (7.2) can make critical enqueues synchronous. Redis Cluster needs a hash tag such as {queue} so that all of a queue's keys live in one slot for Lua.

17. RabbitMQ best practices (2025–2026)¶

• Quorum queues are the default choice for durability and replication (Raft) (quorum queues). Classic mirrored queues were removed in 4.0.
• Delivery limit defaults to 20 in 4.0+. When it is exceeded the message is dropped or dead-lettered. Explicit nack/modify returns don't count.
• At-least-once dead-lettering needs x-dead-letter-strategy=at-least-once and x-overflow=reject-publish.
• Priorities: strict priority with 32 levels as of 4.3 (earlier 4.x: 2 levels).
• Not suited to backlogs over 5M messages or to temporary queues. Each message costs about 32 B of metadata in memory.
• consumer_timeout defaults to 30 min. If a delivery isn't acked in time, the channel is closed with PRECONDITION_FAILED and all deliveries on that channel are requeued. It can be overridden per queue with x-consumer-timeout or a policy (consumers). Potatoq must set it to at least the queue's time limit, or ack long tasks through a different strategy.
• Publisher confirms: ack arrives after the message is persisted for durable queues, with possibly hundreds of ms latency under load. Batch or pipeline confirms. Prefetch: "100 through 300 range usually offer optimal throughput", which suits short messages, but for tasks prefetch should be about concurrency. Requeue loops: track redeliveries (confirms).
• Delayed messages without the plugin: rabbitmq-delayed-message-exchange is no longer maintained. It relied on Mnesia, which was removed in 4.3; it kept a single non-replicated copy, lost timers on restart, and doesn't scale to hundreds of thousands of delayed messages (plugin).
• Per-message TTL only expires at the head of the queue, so a single "delay" queue with per-message TTL blocks shorter delays behind longer ones (TTL).
• The fix is the NServiceBus / Kombu 28-level binary TTL+DLX topology: exchanges celery_delayed_{0..27}, each with a quorum queue whose x-message-ttl=2^level s dead-letters to level−1. The routing key is the delay's 28-bit binary with dots plus the destination routing key, giving a maximum of about 149 days (kombu native_delayed_delivery, NServiceBus). Celery 5.5 adopted this for quorum queues.
• Single active consumer (x-single-active-consumer) gives ordered or serialized queues with failover.
• AMQP heartbeats plus long-running tasks on the same thread cause "Couldn't ack, BrokenPipe" (Celery #3773). Keep the AMQP I/O loop on a dedicated thread or in the Rust core, never on the task thread.

18. SQLite as a queue¶

• Required settings: PRAGMA journal_mode=WAL; PRAGMA busy_timeout=5000; PRAGMA synchronous=NORMAL; (dev.to, goqite).
• Use BEGIN IMMEDIATE for anything that writes. A deferred transaction that reads first and then tries to write gets SQLITE_BUSY immediately, ignoring busy_timeout, because SQLite can't upgrade the snapshot (Bert Hubert, Simon Willison). In Python's sqlite3, legacy transaction control issues BEGIN (deferred) implicitly. Use isolation_level=None (or autocommit=True on 3.12+) and issue BEGIN IMMEDIATE yourself. Django 5.1+ supports OPTIONS={"transaction_mode": "IMMEDIATE", "init_command": "PRAGMA ..."} for SQLite (from memory of the Django 5.1 release notes; verify).
• Claim query (no SKIP LOCKED in SQLite; single-writer serialization makes it unnecessary):

  UPDATE jobs SET state='running', attempts=attempts+1, lease_until=:now+:lease, worker_id=:w
  WHERE id IN (SELECT id FROM jobs WHERE queue=:q AND state='ready' AND run_at<=:now
               ORDER BY priority DESC, run_at, id LIMIT :n)
  RETURNING id, payload;   -- RETURNING needs SQLite >= 3.35
  

  Back it with a partial index ON jobs(queue, priority DESC, run_at, id) WHERE state='ready'.
• Throughput: goqite reports about 18.5k msg/s with 1 producer and 1 consumer, and about 12.5k with 16 parallel participants (M3 Max). Fine for single-host deployments.
• litequeue (GitHub): READY/LOCKED/DONE/FAILED states, one write connection plus a pool of query-only connections, and a claim_id fencing token so stale workers can't ack newer deliveries (a good idea for every backend). Includes retry_expired() and prune().
• No NOTIFY. Wake-ups come from in-process signalling, polling with backoff (Huey: 0.1 s rising to 10 s), or a cheap PRAGMA data_version check (it changes when another connection commits). Oban Lite uses the Erlang PG notifier.
• Caveats:
• Single host only, never over NFS.
• Long readers starve WAL checkpoints and the -wal file grows; run wal_checkpoint(TRUNCATE) periodically.
• Keep claim transactions to microseconds and never run a task inside the write transaction.
• Solid Queue notes that SQLite lacks SKIP LOCKED, so workers queue up on the write lock. Batch claims (claim N, process, batch-ack) to amortize.

19. Comparison matrix (defaults)¶

20. Synthesis: recommended Potatoq defaults¶

The principle is to be safe by default (at-least-once delivery, no hoarding, bounded everything) while staying compatible with Celery's call-site API (@app.task, .delay(), .apply_async(countdown=, eta=)) and the Django Tasks API.

21. Per-backend implementation strategy¶

Postgres (the flagship backend; Oban/River-class design)¶

1. Schema: potatoq_job(id bigint identity, queue text, task text, args jsonb, state smallint/enum, priority smallint, run_at timestamptz, attempt int, max_attempts int, lease_until timestamptz, worker_id, unique_key bytea NULL, unique_states bit(8), errors jsonb[], result jsonb, created/started/finished_at, meta jsonb). Use identity columns, not UUIDs, for the PK (Hatchet's bloat lesson); expose a string ID for Django.
2. Indexes: a partial fetch index (queue, priority DESC, run_at, id) WHERE state='available' so the dequeue needs no sort (DBOS); WHERE state='scheduled' on run_at for the stager; a partial unique index on unique_key filtered by a state bitmask (River).
3. Fetch: UPDATE … SET state='running', attempt=attempt+1, lease_until=now()+lease, worker_id=$w WHERE id IN (SELECT id … FOR UPDATE SKIP LOCKED LIMIT $free_slots) RETURNING *, under READ COMMITTED. Commit immediately and never hold a transaction during task execution.
4. Ack: batched async completion (flush every ~10–50 ms, or N rows) in a single UPDATE … FROM unnest(...). Oban saw +200% throughput from this, and Hatchet buffers at 10 ms.
5. Wake-ups: one LISTEN connection per worker process. Send NOTIFY from application code after commit (on_commit), not from triggers, and debounce it per queue per transaction. Fall back to polling every 1 s (River). Allow listen_notify=False for PgBouncer in transaction mode and for high-write databases (Recall.ai's global commit lock).
6. Leader election: a potatoq_leader row with expiry (or pg_try_advisory_lock on a session connection). The leader runs the stager (scheduled/retryable to available, every 1 s), the rescuer (expired leases to retryable or discarded), the pruner (batch delete), the cron enqueuer and an optional daily REINDEX INDEX CONCURRENTLY.
7. Bloat hygiene: per-table autovacuum_vacuum_scale_factor=0.01 (or a fixed threshold) and autovacuum_vacuum_cost_delay=0. Document the MVCC-horizon risk (analytics or long transactions on the same DB) and offer the transaction_timeout (PG17) / idle_in_transaction_session_timeout guidance. Support a separate DB or schema for the queue, which costs transactional enqueue. For very high throughput, consider a Solid-Queue-style split of a small hot "ready" table from the cold job/history table, or time-range partitions for history (Hatchet).
8. Bulk insert via COPY/executemany for group()/chunks().

SQLite¶

• One DB file (it can be the Django DB, which gives transactional enqueue; or a separate potatoq.db). Set WAL, synchronous=NORMAL, busy_timeout=5000 and foreign_keys=ON on every connection, and use BEGIN IMMEDIATE for every write path. Keep the same schema as Postgres (minus SKIP LOCKED and bitmask tricks), with lease, attempt and fencing token.
• Claim N rows per UPDATE … RETURNING (≥ 3.35), commit, run tasks outside the transaction, and batch acks.
• Wake-ups: in-process events when producer and worker share a process; otherwise poll PRAGMA data_version every 50–100 ms, backing off to 1 s when idle.
• Maintenance: periodic wal_checkpoint(TRUNCATE), batch pruning, and incremental vacuum.
• Document clearly: single host, no network FS, roughly 10k+/s ceiling. Refuse multi-host setups loudly.

Redis / Valkey¶

• Keep all keys for a queue under a hash tag {potatoq:<queue>} for Cluster.
• Data model (BullMQ/Dramatiq-style lists, not Streams, because priorities and delays matter more for tasks):
• ready:<prio> lists (or one ZSET with a priority+seq score);
• a delayed ZSET scored by ETA;
• an inflight ZSET scored by lease_until (member = job ID);
• a jobs hash (payload plus metadata), or one hash per job;
• a dead ZSET with 7-day TTL;
• a workers heartbeat ZSET;
• a marker key/list for blocking wake-up.
• All state transitions are Lua scripts (EVALSHA, compatible with Valkey):
• enqueue
• fetch(n): promote due delayed jobs, pop up to n, add to inflight with the lease
• ack
• nack/retry(delay)
• extend_lease
• reap: expired leases go back to ready with an attempt increment, or to dead
• Workers block with BZPOPMIN/BLPOP on the marker key with a short timeout (blocking can't happen inside Lua), then call fetch. This gives sub-ms latency without polling (SAQ/BullMQ), unlike arq's 0.5 s poll and WATCH contention.
• Lease-based reclaim (not "worker is dead") correctly handles slow jobs, SIGKILL and network partitions. The reaper runs on every worker every few seconds and is idempotent, rather than Dramatiq's 1-in-1M dice roll.
• Optional WAITAOF for durable enqueue. Results go in separate keys with TTL (or Streams for multi-result, as RQ 2.0 does).
• Streams (XREADGROUP / XAUTOCLAIM / XACKDEL) can be offered later as an alternative mode for very high-throughput FIFO queues without priorities.

RabbitMQ¶

• Quorum queues by default (x-queue-type=quorum) with x-delivery-limit aligned to max_retries + safety margin (or rely on the 4.0 default of 20), DLX to <queue>.dead with x-dead-letter-strategy=at-least-once and x-overflow=reject-publish. Set a per-queue x-consumer-timeout ≥ the time limit plus grace (the default of 30 min would kill long tasks and requeue the whole channel).
• Publisher confirms always on (pipelined, async). Manual acks. basic.qos(prefetch_count=free slots) per consumer; one channel per worker process, with the AMQP I/O on a dedicated thread or the Rust core (heartbeats must not depend on task execution).
• Delays / countdown / retry backoff: the 28-level binary TTL+DLX quorum ladder (potatoq.delay.{0..27}, compatible in spirit with Kombu's), which supports up to about 149 days. Never consume-and-hold in worker memory (Dramatiq's .DQ approach). Long-dated ETAs (> N days) could alternatively be stored in a DB or Redis scheduler if one is configured.
• Priorities: quorum queues in 4.3+ have 32 levels, so map Django's -100..100 range into buckets. On older brokers, use classic queues with x-max-priority only on request.
• Ordered or serialized queues via x-single-active-consumer.
• Results/status: RabbitMQ is a bad results store (avoid the Celery rpc:// backend). Require or default to a Redis or SQL result backend, or a DB-backed status table when used with Django.
• Retries: re-publish to the delay ladder with an attempt header and ack the original, so retries aren't counted by the delivery limit. The delivery limit then only catches crash loops (poison messages).

Django integration¶

• TASKS = {"default": {"BACKEND": "potatoq.django.Backend", "QUEUES": [...], "OPTIONS": {"broker": "...", ...}}} with supports_defer = supports_priority = supports_get_result = supports_async_task = True.
• task_class = PotatoqTask (Django 6.1 **kwargs) to accept max_retries, time_limit, unique_key, rate_limit, enqueue_on_commit and so on. Map TaskResultStatus READY/RUNNING/SUCCESSFUL/FAILED to the internal states (scheduled/available/retryable map to READY). Fill attempts, errors (exception class plus traceback string), worker_ids and last_attempted_at. Pass TaskContext.attempt.
• Also support the backport django-tasks for Django < 6.
• The Postgres/SQLite backends using the Django connection give true transactional enqueue (the job row commits with the data). Redis and RabbitMQ use transaction.on_commit(using=<alias>).
• Ship migrations, read-only admin models, manage.py potatoq worker, and Huey-3.2-style admin stats.

22. Key takeaways (short list)¶

1. The biggest differentiators are ack-late, no prefetch hoarding, and leases derived from time limits. Those three fix most of the Celery horror stories.
2. Delayed jobs must live in the broker, not in worker RAM. Dramatiq's RabbitMQ and Redis implementations still get this wrong (#431).
3. Postgres backend = River/Oban design: partial indexes, SKIP LOCKED claim-and-commit, batched acks, leader-run maintenance, optional debounced NOTIFY plus a 1 s poll, transactional enqueue, real unique constraints. Watch out for the MVCC horizon, the NOTIFY commit lock and MultiXact under heavy concurrency.
4. Redis = Lua plus leases plus marker-key blocking (BullMQ-style), not BRPOP (Huey/Taskiq List lose jobs) and not WATCH/MULTI polling (arq).
5. RabbitMQ = quorum queues, confirms, prefetch equal to slots, a TTL+DLX delay ladder, and per-queue consumer_timeout.
6. SQLite = WAL plus BEGIN IMMEDIATE plus UPDATE…RETURNING plus short transactions plus data_version polling.
7. Django 6.x Tasks is an API contract with no worker. Potatoq can be the de-facto production backend. Use 6.1's task_class kwargs for Potatoq-specific options, and handle on-commit per DB alias.
8. arq is in maintenance mode while 3.5M downloads/month wait for a home, and Dramatiq has a bus-factor history. There is room for a well-maintained, async-and-sync, typed, Celery-compatible library.