When your software is too slow in production, how can you figure out the source of the problem?
One common starting point to improving production observability is logging, and ideally trace-based logging (tracing for short). For example, the OpenTelemetry standard and the libraries and backend services that work with it can help you collect metrics, logs, and traces. Tracing—both within and across processes—is the most general of these, immensely useful for identifying and debugging problems, including performance problems.
But there’s another approach to finding performance problems, using sampling-based profiling in production. While profiling is typically used offline during development, you can also use it in production to find performance bottlenecks, with a new generation of continuous profiling tools.
So what’s the difference between trace-based logging and profiling? Which should you use—do you need only one, or both? What’s the fastest way to find performance problems in production?
To jump ahead to our conclusion: trace-based logging and profiling actually give you different information. Once your software is complex enough, you will likely want both.
In this article you’ll learn:
- What exactly trace-based logging and profiling mean.
- The strengths of weaknesses of each.
- Why you probably want both running in production.