Stop Cargo-Culting Celery for Simple FastAPI Background Jobs

You just need to send a confirmation email after signup. Maybe fire a webhook. Maybe resize an image. Somehow youre now three hours deep into Celery workers, RabbitMQ configs, and Flower dashboards — for a task that shouldve taken 20 minutes. FastAPI background tasks have a whole spectrum of solutions, and Celery sits at the nuclear end of that spectrum. This guide walks through the full range: from the built-in BackgroundTasks to production-ready async queues with ARQ and Redis — so you pick the right tool without torching your stack.

TL;DR: Quick Takeaways

• FastAPIs built-in BackgroundTasks works for fire-and-forget — but has no retries, no status tracking, and no process isolation
• ARQ (Async Redis Queue) is a lightweight Celery alternative built natively for asyncio and FastAPI
• Redis acts as both message broker and result backend — no RabbitMQ, no extra moving parts
• Use Celery only when you need distributed workers, complex task chains, or a Beat scheduler

Why Celery Is Overkill for Most FastAPI Projects

Celery is a battle-tested distributed task queue built for synchronous Django-era Python, designed to scale across thousands of workers on multiple servers. Thats the problem. When you bolt Celery onto a modern async FastAPI app, youre bridging two fundamentally different execution models: Celerys sync workers and FastAPIs asyncio event loop. The result is config hell, thread pool gymnastics, and an ops burden that makes a simple email endpoint feel like deploying Kubernetes. The async Python ecosystem has matured enough that you dont need Celery for 80% of background task patterns anymore — and in 2026, reaching for it by default is legacy thinking.

The hidden tax nobody mentions

Celery workers run in separate processes, which means they dont share memory with your FastAPI app. Any task that needs your apps database session factory, connection pool, or in-memory cache has to re-initialize all of it inside the worker. Thats not a minor inconvenience — its a maintenance trap. A proper async task queue that runs in the same asyncio context lets workers reuse connection pools and app dependencies directly, which is exactly what ARQ gives you.

FastAPIs Built-in BackgroundTasks: Honest Assessment

FastAPI ships with a BackgroundTasks dependency that runs functions after the response is sent, in the same process. No broker, no worker config, no Redis — inject BackgroundTasks into your endpoint, call add_task(), done. For small, non-critical operations this works. The moment you need to check a tasks status, handle failures, or survive a process restart mid-execution — it falls apart completely.

from fastapi import FastAPI, BackgroundTasks

app = FastAPI()

def send_welcome_email(email: str) -> None:
    print(f"Sending email to {email}")

@app.post("/signup")
async def signup(email: str, background_tasks: BackgroundTasks):
    background_tasks.add_task(send_welcome_email, email)
    return {"status": "registered"}

Clean, readable, zero infrastructure. But if your app process crashes after the response and before the task runs, the job is gone. No retry, no trace. For confirmation emails thats probably fine. For payment webhooks or report generation — not a chance.

When built-in BackgroundTasks actually breaks

The real issue with BackgroundTasks isnt async support — FastAPI handles async def tasks fine, running them in the same event loop. The actual danger is heavier tasks. If your background function spikes CPU or hangs on a slow I/O call, it poisons the entire event loop and slows API responses for every concurrent user. Its a fire-and-hope tool, not a queue. Theres no backpressure, no depth limit, no worker isolation. FastAPIs own docs nudge you toward external queues for anything heavy — which is a diplomatic way of saying this will hurt you in production.

Real talk: BackgroundTasks shares the same resources as your API logic. If your task spikes CPU or hangs, your entire API hangs with it.

ARQ: The Async Redis Queue That Fits FastAPI

ARQ (Async Redis Queue) is a lightweight job queue built from the ground up for asyncio. Unlike Celery, async isnt an afterthought — every worker function is a native async def, running in its own event loop. It uses Redis as both the message broker and result backend, keeps its API surface small, and integrates with FastAPI so naturally it feels purpose-built for it. The trade-off: no complex task chains, no Beat scheduler out of the box. For the vast majority of FastAPI background task use cases, thats not a trade-off — its a feature. If you want FastAPI-style Dependency Injection in your workers, look at Taskiq. If you want simplicity and reliability, stick with ARQ.

pip install arq redis

Defining worker functions with ARQ

ARQ worker functions receive a ctx dict as their first argument — this is where you pass shared resources like database connection pools or HTTP clients. Initialize them once in on_startup, attach to ctx, and every worker task reuses them without re-creating connections on each job. The function is a plain async def — no magic decorators, no Celery-style boilerplate, fully testable like any other async function.

import asyncio
import httpx

async def startup(ctx: dict):
    ctx["http"] = httpx.AsyncClient()

async def shutdown(ctx: dict):
    await ctx["http"].aclose()

async def send_email(ctx: dict, recipient: str, subject: str) -> dict:
    # reuse shared httpx client from ctx
    await asyncio.sleep(0.5)  # simulate SMTP call
    print(f"Email sent → {recipient}: {subject}")
    return {"status": "sent", "recipient": recipient}

class WorkerSettings:
    functions = [send_email]
    on_startup = startup
    on_shutdown = shutdown
    redis_settings = {"host": "localhost", "port": 6379}

The WorkerSettings class is all ARQ needs to boot a worker. Run it with arq tasks.WorkerSettings in a separate terminal and it starts polling Redis for jobs immediately.

Enqueueing jobs from a FastAPI endpoint

On the FastAPI side, create a Redis connection pool at startup via lifespan events and inject it wherever you need to enqueue a job. Your endpoint stays thin, returns immediately, and the worker picks up the job independently — completely decoupled from the request lifecycle. This is the core pattern behind a proper async Redis queue setup with FastAPI.

from contextlib import asynccontextmanager
from fastapi import FastAPI
from arq import create_pool
from arq.connections import RedisSettings, ArqRedis

redis_pool: ArqRedis = None

@asynccontextmanager
async def lifespan(app: FastAPI):
    global redis_pool
    redis_pool = await create_pool(RedisSettings(host="localhost", port=6379))
    yield
    await redis_pool.close()

app = FastAPI(lifespan=lifespan)

@app.post("/send-notification")
async def send_notification(email: str, subject: str):
    job = await redis_pool.enqueue_job("send_email", email, subject)
    return {"status": "queued", "job_id": job.job_id}

@app.get("/job-status/{job_id}")
async def job_status(job_id: str):
    job = await redis_pool.get_job_result(job_id)
    return {"result": job}

The /job-status/{job_id} endpoint is something BackgroundTasks simply cannot give you. The job ID returned at enqueue time lets you poll for status, surface progress to the frontend, or retry on failure — that alone covers 90% of the reasons people reach for Celery.

Running the Full Stack Locally

The dev setup for this pattern is three terminals: your FastAPI app, your ARQ worker, and Redis. No Docker Compose required to get started. The FastAPI background tasks without Celery workflow is genuinely lightweight — Redis boots in under a second, the ARQ worker starts in milliseconds, and your API stays completely decoupled from job execution.

# terminal 1 — start Redis
redis-server

# terminal 2 — start FastAPI
uvicorn main:app --reload

# terminal 3 — start ARQ worker
arq tasks.WorkerSettings

Adding retries and timeouts

ARQ supports retries natively — raise Retry inside your task to reschedule it, with an optional delay. The ctx["job_try"] counter tracks which attempt youre on, so you can implement exponential backoff without any extra libraries. This is the kind of thing Celery makes surprisingly verbose for what it is.

from arq.worker import Retry

async def send_email(ctx: dict, recipient: str, subject: str) -> dict:
    try:
        await call_smtp_api(recipient, subject)
        return {"status": "sent"}
    except TimeoutError:
        if ctx["job_try"] < 3:
            raise Retry(defer=ctx["job_try"] * 5)  # backoff: 5s, 10s, 15s
        return {"status": "failed", "reason": "smtp_timeout"}

ARQ vs Celery vs Built-in: Decision Table

Picking between these three comes down to a few questions: Do you need job status tracking? Process isolation? Distributed workers across multiple machines? Heres the comparison for FastAPI async background tasks in 2026:

When Celery Is Still the Right Answer

Celery earns its complexity when requirements genuinely demand it: task DAGs where Task B depends on Task As result, a Beat-scheduled cron replacement running dozens of periodic jobs, or horizontal scaling across multiple physical servers with specialized worker pools. If youre running a large Django monolith with a sync ORM and years of Celery tasks in production — ripping it out for ARQ is masochism. The point isnt that every Celery alternative is always better. The point is that Celery gets chosen by default when its massively oversized for the job, and that default needs to stop.

Production Checklist Before You Ship

Running async background tasks in production has a short list of things that will bite you if ignored. None of these are exotic — theyre the boring table stakes that separate a weekend hack from something that runs quietly at 3am without waking you up.

• Redis connection pooling: create the pool once at startup via lifespan, not per-request
• Worker crash handling: ARQ uses pessimistic locking — a job grabbed by a crashed worker re-enters the queue after its timeout expires
• Graceful shutdown: ARQ workers finish their current job before stopping on SIGTERM — dont SIGKILL them
• Task idempotency: design tasks so re-running them on retry doesnt cause duplicate side effects — double emails, double charges
• Monitoring: ARQ has no Flower equivalent, but job results land in Redis — KEYS arq:result:* gives quick visibility, or instrument with your existing observability stack
• Redis persistence: enable AOF (appendonly yes) if you cant afford to lose queued jobs on Redis restart

FAQ

Whats the main difference between ARQ and FastAPIs built-in BackgroundTasks?

BackgroundTasks runs functions in the same process after the response is sent — no broker, no worker, no status tracking. ARQ runs tasks in a separate worker process via Redis, giving you retries, job status, and full isolation from your API. Use built-in for fire-and-forget microtasks; use ARQ when you need reliability and visibility over your FastAPI background tasks.

Do I need Redis to run async background tasks in FastAPI?

If youre using ARQ or a similar queue-based approach, yes — Redis is the message broker decoupling your API from your workers. For simple tasks that dont need a separate process, asyncio.create_task() works with zero infrastructure. The right choice depends on whether you need persistence, retries, and task tracking.

Is ARQ production-ready or a toy library?

ARQ is production-ready for I/O-bound workloads — actively maintained and used in real production systems. Its not Celery in terms of feature breadth, and it doesnt try to be. For the most common async background task patterns — email, webhooks, external API calls, report generation — it handles them cleanly without the ops overhead.

How do I check the status of a background job in FastAPI with ARQ?

enqueue_job() returns a Job object with a job_id. Return that ID to the client or store it in your DB, then poll redis_pool.get_job_result(job_id) to check status. ARQ stores results in Redis with a configurable TTL. This is the key advantage over built-in BackgroundTasks — you know what happened to your job.

Can I use Taskiq instead of ARQ with FastAPI?

Yes — Taskiq is a newer async task queue with tighter FastAPI integration and Dependency Injection support inside workers. Worth evaluating for new projects in 2026. ARQ is more battle-tested and simpler; Taskiq is more ambitious. Both are solid Celery alternatives for async codebases and miles ahead of Celery for pure-async FastAPI apps.

Whats the best way to handle a failed background task without Celery?

With ARQ, raise Retry inside your task with an optional defer time. Use ctx["job_try"] to implement backoff. For truly critical jobs, combine ARQ retries with an idempotency key stored in your database — so you can detect and recover from partial failures independent of the queue state entirely.

Krun Dev <code/>