Itamar Turner Trauring: Stabbing yourself with a fork() in a multiprocessing.Pool full of sharks

It’s time for another deep-dive into Python brokenness and the pain that is POSIX system programming, this time with exciting and not very convincing shark-themed metaphors! Most of what you’ll learn isn’t really Python-specific, so stick around regardless and enjoy the sharks.

Let’s set the metaphorical scene: you’re swimming in a pool full of sharks. (The sharks are a metaphor for processes.)

Next, you take a fork. (The fork is a metaphor for fork().)

You stab yourself with the fork. Stab stab stab. Blood starts seeping out, the sharks start circling, and pretty soon you find yourself—dead(locked) in the water!

In this journey through space and time you will encounter:

• A mysterious failure wherein Python’s multiprocessing.Pool deadlocks, mysteriously.
• The root of the mystery: fork().
• A conundrum wherein fork() copying everything is a problem, and fork() not copying everything is also a problem.
• Some bandaids that won’t stop the bleeding.
• The solution that will keep your code from being eaten by sharks.

Let’s begin!