concurrency
Concurrency in Python encompasses the techniques for executing multiple tasks simultaneously or in overlapping time periods — including threading (for I/O-bound tasks), asyncio (for high-concurrency I/O with cooperative scheduling), and multiprocessing (for CPU-bound tasks that bypass the GIL), with the choice between these approaches determined by whether the workload is I/O-bound or CPU-bound and the specific requirements for parallelism, memory sharing, and integration with async frameworks like those used in LLM API clients.
What Is Concurrency in Python?
- Definition: The ability to manage multiple tasks that make progress within overlapping time periods — concurrency (tasks interleave on one core) differs from parallelism (tasks execute simultaneously on multiple cores), though Python supports both through different mechanisms.
- GIL (Global Interpreter Lock): CPython's GIL allows only one thread to execute Python bytecode at a time — this means threading does NOT provide true parallelism for CPU-bound Python code, but it DOES allow parallel I/O operations because the GIL is released during I/O waits.
- Choosing the Right Tool: I/O-bound tasks (API calls, database queries, file I/O) benefit from threading or asyncio — CPU-bound tasks (data processing, model inference) require multiprocessing or external libraries (NumPy, PyTorch) that release the GIL during computation.
Concurrency Models
| Model | Best For | Python Module | True Parallelism | Memory |
|---|---|---|---|---|
| Threading | I/O-bound, simple | threading | No (GIL) | Shared |
| Asyncio | I/O-bound, many connections | asyncio | No (single thread) | Shared |
| Multiprocessing | CPU-bound | multiprocessing | Yes (separate processes) | Separate |
| ProcessPoolExecutor | CPU-bound, simple API | concurrent.futures | Yes | Separate |
| ThreadPoolExecutor | I/O-bound, simple API | concurrent.futures | No (GIL) | Shared |
Async for LLM APIs
- Why Async: LLM API calls take 500ms-30s — async allows hundreds of concurrent requests on a single thread, maximizing throughput when calling OpenAI, Anthropic, or self-hosted models.
- AsyncOpenAI: The OpenAI Python client provides an async interface — `await client.chat.completions.create()` enables non-blocking API calls.
- asyncio.gather: Run multiple async calls concurrently — `results = await asyncio.gather(*[call_api(p) for p in prompts])` processes all prompts in parallel.
- Rate Limiting: Use `asyncio.Semaphore` to limit concurrent requests — preventing API rate limit errors while maintaining high throughput.
- Streaming: Async streaming (`async for chunk in response`) enables real-time token delivery to users while other requests are processed concurrently.
When to Use Each Approach
- Threading: Simple I/O parallelism (downloading files, making a few API calls) — easy to use but limited scalability for thousands of connections.
- Asyncio: High-concurrency I/O (web servers, LLM API batching, websockets) — scales to thousands of concurrent connections on a single thread but requires async-compatible libraries.
- Multiprocessing: CPU-intensive work (data preprocessing, model inference without GPU) — true parallelism but higher memory overhead (each process gets its own memory space).
- External Libraries: NumPy, PyTorch, and other C-extension libraries release the GIL during computation — enabling true parallelism within threads for numerical workloads.
Concurrency in Python is the essential skill for building performant ML applications — choosing between threading, asyncio, and multiprocessing based on whether workloads are I/O-bound or CPU-bound, with async programming particularly critical for LLM applications that must efficiently manage hundreds of concurrent API calls and streaming responses.