Python Async/Await: Writing Concurrent Code That Actually Works

I rewrote a metadata ingestion service from threaded Python to asyncio three years ago and watched its P99 latency drop from 4.2 seconds to 380ms without adding a single machine. Two weeks later, the same service locked up completely for 90 seconds because one coroutine called a synchronous JSON schema validation that blocked the event loop. That duality defines working with Python async: you get remarkable efficiency when you respect the execution model, and bewildering hangs when you don't. In my experience, most teams adopt async and await as faster syntax for concurrency without understanding that it's a fundamentally cooperative model, and that misunderstanding is where production incidents come from.

The Cooperative Illusion: How Python's Event Loop Actually Executes Your Code

Unlike threads where the OS preemptively switches context, asyncio relies on your coroutines voluntarily yielding control. When you write await something, you are explicitly telling the event loop: "I'm stuck waiting for I/O, run something else until this is ready." If you never await, nothing else runs. This single rule explains 80% of the async bugs I've debugged.

The event loop, as documented in the Python Documentation, is a single-threaded scheduler running in a tight loop: check for ready I/O, run the next callback, repeat. It maintains a queue of ready tasks and a selector for network and file descriptor events. A coroutine is just a state machine that the loop drives forward one await at a time. When your code does a CPU-bound calculation or calls a blocking library like requests.get() without yielding, the entire loop stalls. No other coroutine, timeout, or cancellation can make progress until that function returns.

What "await" Actually Suspends

It's a common misconception that async def makes a function run concurrently. It doesn't. Calling an async function without awaiting it does almost nothing - it returns a coroutine object that hasn't started. Execution only begins when the event loop schedules it as a Task. I've found that mentally replacing await with yield control until this future is done helps teams write more correct code. You are not waiting; you are surrendering the thread.

The Single-Threaded Trap That Looks Like Parallelism

I audited a service that fetched data from three external APIs concurrently with asyncio.gather. In staging it was fast. In production with larger payloads, it was slower than the synchronous version. The culprit was response.json() parsing and subsequent Pydantic validation running synchronously inside each coroutine. With 500KB JSON bodies, the parsing blocked the loop long enough to serialize what should have been concurrent I/O. The network waits were concurrent, but the CPU work was not. Any time you see high event loop utilization without high CPU, look for this pattern.

# Blocking the event loop without realizing it
import asyncio
import requests  # Synchronous library - will block

async def fetch_bad(url):
    # This blocks the ENTIRE event loop until the request completes
    # No other coroutine can run during the socket wait
    response = requests.get(url)
    return response.json()

# Correct approach - use an async-native library
import aiohttp

async def fetch_good(session, url):
    async with session.get(url) as response:
        # await yields control while waiting for bytes on the wire
        data = await response.json()
        return data

async def main():
    async with aiohttp.ClientSession() as session:
        results = await asyncio.gather(
            fetch_good(session, "https://api.example.com/users"),
            fetch_good(session, "https://api.example.com/orders"),
            fetch_good(session, "https://api.example.com/inventory")
        )

Task Creation and Lifetime: When to Use gather, TaskGroup, and create_task

For years asyncio.gather was the default way to run things concurrently, and most codebases still use it. Since Python 3.11, asyncio.TaskGroup is the better primitive for almost all structured concurrency, and I've migrated every new service to it. The difference is not just style - it's about error propagation and lifetime guarantees.

gather returns an aggregate future. If one child coroutine raises, the others continue running in the background unless you explicitly cancel them. I've seen this cause subtle resource leaks where a failed database write left orphaned tasks still holding connections from the pool. TaskGroup enforces structured concurrency: if any task fails, all other tasks in the group are cancelled, and the exception is propagated cleanly. It also prevents the classic mistake of creating a task with asyncio.create_task() and forgetting to await it, which silences exceptions entirely.

Structured Lifetime With TaskGroup

In my experience, TaskGroup makes cancellation and error handling explicit. You cannot leave the async with TaskGroup() as tg: block until every task inside has completed or been cancelled. This makes it ideal for request-scoped concurrency like fan-out calls to microservices.

Fire-and-Forget Without Losing Errors

Sometimes you do need background work, like emitting metrics or invalidating a cache after responding. Don't use bare create_task for this. I wrap these in a helper that logs exceptions via a done callback, or better, I tie them to a longer-lived TaskGroup owned by the application lifecycle. An unawaited task that raises will trigger Python's "Task exception was never retrieved" warning, which is easy to miss in logs until it causes state corruption. If you are building these patterns, using Python Type Hints: A Practical Guide for Production Code with Coroutine and Task types helps mypy catch when you're passing a coroutine where a Task is expected.

import asyncio

async def call_service(name, delay):
    await asyncio.sleep(delay)
    if name == "inventory":
        raise RuntimeError("inventory service unavailable")
    return f"{name} ok"

async def with_gather():
    # If inventory fails, users and orders tasks keep running in background
    results = await asyncio.gather(
        call_service("users", 0.5),
        call_service("inventory", 0.3),
        call_service("orders", 0.5),
        return_exceptions=False
    )

async def with_taskgroup():
    # Structured: failure in one task cancels siblings and propagates
    try:
        async with asyncio.TaskGroup() as tg:
            t1 = tg.create_task(call_service("users", 0.5))
            t2 = tg.create_task(call_service("inventory", 0.3))
            t3 = tg.create_task(call_service("orders", 0.5))
    except* RuntimeError as eg:
        # ExceptionGroup contains the inventory failure
        # t1 and t3 were automatically cancelled
        print(f"Group failed: {eg.exceptions}")

    # Correct fire-and-forget with error handling
    def log_task_failure(task: asyncio.Task):
        if task.cancelled():
            return
        if exc := task.exception():
            print(f"Background task failed: {exc}")

    task = asyncio.create_task(call_service("metrics", 0.1))
    task.add_done_callback(log_task_failure)

Backpressure and Bounded Concurrency: Taming the Thundering Herd With Semaphores and Queues

The most dangerous async code I've seen is unbounded concurrency: asyncio.gather(*[fetch(url) for url in ten_thousand_urls]). It works in a test with 10 URLs and then exhausts file descriptors, memory, or the downstream service in production. Async makes it trivial to launch 10,000 concurrent operations, which is precisely why you need explicit limits.

There are two primary patterns I use. For simple fan-out where you just need to cap parallelism, asyncio.Semaphore is perfect. For producer-consumer pipelines where work arrives continuously, asyncio.Queue with a fixed pool of workers gives you natural backpressure. The choice depends on whether your workload is a known batch or a stream.

A semaphore limits how many coroutines can enter a critical section at once. I typically set it to 20-50 for outbound HTTP calls, tuned against the downstream service's rate limits and our connection pool size. A queue allows the producer to await queue.put() when the queue is full (if you set maxsize), which automatically slows down ingestion when consumers can't keep up. Without that, an async producer reading from Kafka or S3 can enqueue millions of items and OOM the process.

One nuance teams miss: Semaphore does not preserve ordering, and gather returns results in input order regardless of completion order. If you need results as they complete to start downstream processing earlier, use asyncio.as_completed instead of gathering and then iterating.

import asyncio
import aiohttp

async def bounded_fetch(urls, concurrency=25):
    sem = asyncio.Semaphore(concurrency)
    # Timeouts are per-request, not per-batch
    timeout = aiohttp.ClientTimeout(total=10)

    async def fetch_one(session, url):
        async with sem:  # Only 25 can be inside this block at once
            async with session.get(url, timeout=timeout) as resp:
                resp.raise_for_status()
                return await resp.json()

    async with aiohttp.ClientSession(timeout=timeout) as session:
        async with asyncio.TaskGroup() as tg:
            tasks = [tg.create_task(fetch_one(session, u)) for u in urls]
        # TaskGroup ensures all tasks are done; collect results
        return [t.result() for t in tasks]

# Producer-consumer with backpressure via bounded Queue
async def worker(name, queue: asyncio.Queue):
    while True:
        item = await queue.get()
        if item is None:  # Sentinel for shutdown
            queue.task_done()
            break
        try:
            await process_item(item)
        finally:
            queue.task_done()

async def run_pipeline(items):
    queue: asyncio.Queue = asyncio.Queue(maxsize=200)
    workers = [asyncio.create_task(worker(f"w{i}", queue)) for i in range(5)]

    for item in items:
        await queue.put(item)  # Backpressure: waits if queue is full

    await queue.join()  # Wait until all items processed
    for _ in workers:
        await queue.put(None)  # Send shutdown sentinel
    await asyncio.gather(*workers)

async def process_item(item):
    await asyncio.sleep(0.01)

Cancelling Without Corrupting State: Timeouts, Shielding, and Cleanup Guarantees

Cancellation in asyncio is cooperative and exception-based. When you cancel a task, Python injects a CancelledError at the next await point. That means cancellation does not preempt CPU-bound code, and it can leave shared state inconsistent if you don't handle it carefully. I've debugged corrupted in-memory caches where a cancellation arrived between a dict read and write.

The modern API is asyncio.timeout() (Python 3.11+) and asyncio.wait_for for older code. I prefer timeout because it composes correctly with TaskGroups and doesn't create an extra task. Critically, timeout cancellation is not instant - your coroutine must hit an await to receive it. If you wrap a blocking call with wait_for, the timeout will appear to not work because the event loop is blocked and cannot deliver the cancellation.

Where Shielding Is Actually Needed

asyncio.shield protects an inner await from being cancelled when the outer task is cancelled. I use it sparingly, only for operations that must not be interrupted, like committing a database transaction or releasing a distributed lock. Overusing shield makes shutdown logic opaque. In most cases, you should let cancellation propagate and use try/finally for cleanup. Every await that acquires a resource should be paired with a finally block that releases it, because that finally will run even when CancelledError is raised.

Graceful Shutdown on SIGTERM

For long-running services, I install signal handlers that cancel the main task. The pattern is: catch CancelledError at the top level, cancel all child tasks, and await them with return_exceptions=True so they can run their finally blocks. Without this, Kubernetes sending SIGTERM can leave half-written files or un-acked queue messages.

Bridging Sync and Async: Thread Pools, Process Pools, and Blocking Libraries

You will inevitably need to call synchronous code from an async context. Database drivers, ORMs like SQLAlchemy in sync mode, and CPU-heavy libraries like Pillow have no async equivalent or require significant refactoring. The answer is not to make everything async - it's to run blocking code in an executor so the event loop stays responsive. This is also where isolating workloads matters; I keep async service dependencies clean by using Python Virtual Environments: Why They Matter and How to Use Them to pin separate async and sync driver versions when migrating incrementally.

loop.run_in_executor with the default thread pool is the right tool for I/O-bound blocking calls (like boto3 or psycopg2). For CPU-bound work like image processing or JSON validation that would otherwise block the loop, use ProcessPoolExecutor to avoid GIL contention. I learned this the hard way after moving CPU-heavy Pydantic validation off the event loop into a thread pool and seeing no improvement - threads in Python don't help with CPU work due to the GIL. The Python Documentation on running in threads explicitly warns about this distinction.

When mixing data models across sync and async boundaries, keep your validation layer consistent. Choosing between Dataclasses vs Pydantic: When to Use Which in Python Projects early helps, because passing unvalidated dicts between executors and coroutines creates serialization bugs that only appear under cancellation.

Workload Type Correct Executor Example Libraries Risk If Run Directly on Loop
Blocking I/O (sync network, DB) ThreadPoolExecutor via run_in_executor requests, boto3, psycopg2, sqlalchemy sync Event loop stalls, all coroutines freeze, timeouts stop firing
CPU-Bound Computation ProcessPoolExecutor Pillow, numpy ops, jsonschema, crypto hashing High latency and missed I/O events; GIL prevents thread benefit
Non-Blocking Async I/O None (native await) aiohttp, asyncpg, httpx, aioboto3 No risk - yields control correctly
Small Fast Sync Call (<1ms) None (run directly) Logging, simple dict ops, datetime parsing Negligible; context switch overhead would be worse

Tracing Hangs and Latency Ghosts: Debugging Async Code That Stalls in Production

Async bugs rarely produce stack traces. They produce silence: a request that never returns, a loop whose latency suddenly spikes, a worker that appears idle but consumes no CPU. My debugging toolkit for this is specific.

First, enable asyncio.Runner(debug=True) or PYTHONASYNCIODEBUG=1 and logging for asyncio in staging. Debug mode tracks when coroutines block the loop longer than 100ms and warns about never-awaited coroutines. It has overhead, so I don't run it in production, but it catches the synchronous time.sleep(1) that someone accidentally left in an async path.

Second, for live production hangs, I use asyncio.all_tasks() and inspect their stack traces. I expose an admin endpoint that returns the output of task.get_stack() for every running task. When the service stalls, that endpoint shows exactly which coroutine is stuck on which await - or worse, which coroutine is stuck not awaiting at all. I've found that adding this endpoint pays for itself the first time you need it at 2 AM.

Third, measure event loop lag directly. I run a tiny coroutine that sleeps for 0.1 seconds in a loop and measures how late it wakes up. If the loop is healthy, the lag is <5ms. If something is blocking, the lag jumps to hundreds of milliseconds. Exporting this as a Prometheus metric gives you an early warning before timeouts cascade. Combine this with structured tests; patterns from Pytest Patterns That Scale: Fixtures, Mocks and Parametrize make it straightforward to write deterministic tests for timeout and cancellation paths using pytest-asyncio and fake clocks.

Common Hang Signatures I Look For

A queue that never drains often means a worker crashed without calling task_done(), leaving queue.join() waiting forever. A gather that never returns often means one child is blocked on synchronous I/O while the others completed. A service that degrades over hours often means file descriptors or connections are leaking because cancelled tasks skipped cleanup. In each case, the fix is not more concurrency but stricter lifecycle guarantees with TaskGroup, timeouts, and finally blocks.

Frequently Asked Questions

When should I use asyncio instead of threads or multiprocessing?

Use asyncio when your workload is I/O-bound with many concurrent waits - typically outbound HTTP calls, database queries, or message queue consumers where you wait on the network more than you compute. It scales to thousands of connections with low memory overhead because tasks are lightweight coroutines, not OS threads. Use threads when you need to run blocking synchronous libraries without rewriting them, and multiprocessing when you need true CPU parallelism that bypasses the GIL. In my experience, if your concurrency is under ~50 and mostly CPU work, threads or processes are simpler to reason about than async.

Why does asyncio.gather silently swallow exceptions when I use return_exceptions=True?

With return_exceptions=True, gather does not raise - it returns exception objects as if they were results, in the same order as your inputs. You must iterate the results and check isinstance(r, Exception) for each one. Forgetting this check makes failures look like successful values. TaskGroup avoids this pitfall by raising an ExceptionGroup that you handle with except*, forcing you to confront errors. I only use return_exceptions=True for batch jobs where partial failure is expected and I log each exception individually.

Can I call synchronous blocking code inside an async function?

You can, but you will block the event loop until that call returns, which stalls all other tasks, timeouts, and cancellations. For fast operations under a millisecond, this is acceptable. For anything that touches the network, disk, or does heavy CPU work, offload it with await loop.run_in_executor(None, blocking_func) for I/O-bound work or a ProcessPoolExecutor for CPU-bound work. As noted in the Python Documentation, asyncio.to_thread (Python 3.9+) is a convenient wrapper around the thread pool executor for this purpose.

How do I correctly handle timeouts and cancellation without leaking resources?

Wrap the await that might hang with async with asyncio.timeout(seconds): and ensure every resource acquisition has a try/finally that releases it. When a timeout fires, it raises TimeoutError (or CancelledError inside the block) at the next await, and your finally block will run to close connections, release semaphores, or call queue.task_done(). Avoid catching CancelledError broadly - let it propagate - and use asyncio.shield only for the narrow section that must not be interrupted, like a final commit.

Related Articles

References & Standards: Python Documentation · MDN Web Docs · IETF RFC Standards