When your Celery tasks are too slow, and you want them to run faster, you usually want to find and then fix the performance bottleneck. It’s true, you can architect a solution where slow tasks don’t impact faster ones, and you may sometimes need to. But if you can manage to make all your tasks fast, that is ideal.
To speed up your code, you need to identify the bottlenecks: which task is slow, and why it’s slow. In this article we’ll go through the process for finding the performance bottleneck in your Celery tasks, explaining some of the tradeoffs and options along the way:
- Identifying slow tasks.
- Finding predictable-in-advance performance bottlenecks with code instrumentation.
- Profiling to find harder-to-predict bottlenecks, either:
- On a development machine.
- Or, in production.