As you code, you might have a coworker, or friend, or a little voice in your head, reminding you of Knuth’s famous saying: “premature optimization is the root of all evil.” But what makes an optimization premature, anyway?
The short answer is that this aphorism is a tautology. “Premature” means “too early,” so we can rephrase the point as “doing things at the wrong time isn’t ideal.” Can’t argue with that!
But what is the correct time? When should you make your software fast? When is optimization good, and when is it bad?
In order to reduce the scope a bit, I’m going to focus on one particular problem domain: data processing pipelines or batch jobs. This is the kind of software you often write when doing data science, or scientific computing: you load in some data, process them, spit out a result.
In this particular context, at least, I’m going to argue that:
- Having sufficiently fast software from the start is a huge boost to productivity.
- Writing fast software doesn’t necessarily take any more work than writing slow software.
- Fast software is often just as good as slow software in terms of maintainability and correctness.