David Amos: My Goldilocks Python Setup: pyenv, pipx, and pip-tools

This article contains affiliate links. See my affiliate disclosure for more information.

Beginners should install Python with the official python.org installers. But the installer workflow isn&apost very efficient for professional developers working on multiple projects — or any project that supports multiple Python versions.

In this guide, I&aposll show you how to:

• Install multiple versions of Python with pyenv.
• Install global utilities like black and flake8 with pipx.
• Manage project dependencies with pip-tools.

Install Python With pyenv

pyenv allows you to install multiple versions of Python and switch between them as needed. To install pyenv, run the command below that corresponds to your OS.

macOS/Linux
Use the automatic installer below, or see the README for more options:

curl https://pyenv.run | bash

You&aposll need to restart your shell when you&aposre done:

exec "$SHELL"

Windows
Install with PowerShell below, or see the README for more options:

Invoke-WebRequest -UseBasicParsing -Uri "https://raw.githubusercontent.com/pyenv-win/pyenv-win/master/pyenv-win/install-pyenv-win.ps1" -OutFile "./install-pyenv-win.ps1"; &"./install-pyenv-win.ps1"

Once pyenv or pyenv-win is installed, run pyenv install 3.11 to install the latest version of Python 3.11:

pyenv install 3.11

At the time of the writing, the latest version is 3.11.1.

Use a different version prefix or a specific version number to install other versions of Python. For example, the following command installs the latest version of Python 3.10:

pyenv install 3.10

To use a specific Python version that you&aposve installed, run python global:

pyenv global 3.11.1

Now your python3 command points to Python 3.11.1.

Install Global Utilities With pipx

pipx is a Python package installer that installs applications and command-line utilities in isolated environments and makes them available globally.

This is particularly helpful for tools like black and flake8, as they can be installed once and used across multiple projects while still taking into account project-specific configuration files.

You can install pipx with pip:

python3 -m pip install pipx

Before you can use pipx, you need to add it to PATH:

python3 -m pipx ensurepath

Some of the tools I install with pipx are:

• black: My preferred Python auto-formatter.
• flake8: My preferred Python code linter.
• ipython: My preferred Python REPL.

The following commands install all three tools in three separate environments:

pipx install black
pipx install flake8
pipx install ipython

Now you can use these tools as you normally would across all your projects without installing them in every project&aposs environment.

A Note For VS Code Users

If you use Python for VS Code, you&aposll need to point your flake8 path to the right location. If you don&apost, you&aposll see a pop-up that says "flake8 is not installed" every time you open VS Code.

Run pipx list to list every app installed by pipx. The application install path is displayed on the second line of output:

$ pipx list
venvs are in /Users/damos/.local/pipx/venvs
apps are exposed on your $PATH at /Users/damos/.local/bin
   package black 22.12.0, installed using Python 3.11.1
    - black
    - blackd
   package flake8 6.0.0, installed using Python 3.11.1
    - flake8
   package ipython 8.9.0, installed using Python 3.11.1
    - ipython
    - ipython3

In my case, apps are installed in /Users/damos/.local/bin. Now run which flake8 to see which executable the flake8 command points to:

$ which flake8
/Users/damos/.local/bin/flake8

Copy this path to your clipboard.

Open the VS Code settings explorer by going to File > Settings menu or pressing Cmd+, on macOS or Ctrl+, on Linux/Windows.

Type flake8 into the search bar and press Enter. Then paste the path to flake8 into the input box for the Python > Linting: Flake8 Path setting:

Manage Project Dependencies With pip-tools

I only had to use pip-tools for about 5 minutes to know I needed it all the time. pip-tools helps you where you need it to and gets out of the way everywhere else. The basic usage goes like this.

Start a new Python project in a new folder:

# Create a new folder and change directories to it
mkdir ~/my-project && cd ~/my-project

# Create a new virtual environment
python3 -m venv .venv --prompt my-project

# Activate the virtual environment
source .venv/bin/activate

# Update pip
python -m pip install -U pip

Then install pip-tools into the project&aposs virtual environment using pip:

python -m pip install pip-tools

Your project&aposs dependencies go in a file called requirements.in. For example, the requirements.in file for a Django 3 project might look like this:

# ~/my-project/requirements.in

django<4.0

You can generate a requirements.txt with fully resolved dependencies using the pip-compile command:

pip-compile --allow-unsafe --resolver=backtracking requirements.in

The requirements.txt file generated by pip-compile will look something like this:

#
# This file is autogenerated by pip-compile with Python 3.11
# by the following command:
#
#    pip-compile --allow-unsafe --resolver=backtracking requirements.in
#
asgiref==3.6.0
    # via django
django==3.2.16
    # via -r requirements.in
pytz==2022.7.1
    # via django
sqlparse==0.4.3
    # via django

Notice that there is a comment at the top of the file indicating that it was autogenerated by pip-compile (it even includes the command!) and that every version number is pinned.

I usually create a second file called dev-requirements.in for development dependencies, such as pytest and django-debug-toolbar, and even pip-tools itself:

# ~/my-project/dev-requirements.in

# Use requrements.txt as a constraint file
-c requirements.txt

django-debug-toolbar
pip-tools
pytest

Using requirements.txt as a constraint file ensures that any dependencies installed for the packages in dev-requirements.in are compatible with the package versions specified in requirements.txt.

Run pip-compile a second time to generate a dev-requirements.txt file:

pip-compile --allow-unsafe --resolver=backtracking dev-requirements.in

Now for the real magic. Run pip-sync to synchronize your virtual environment with the packages in the requirements.txt and dev-requirements.txt files:

pip-sync requirements.txt dev-requirements.txt

To update or add a package, edit the appropriate .in file and then re-compile and re-sync everything.

I often wrap all of this into a Makefile:

install:
	@pip install -r requirements.txt -r dev-requirements.txt

compile:
	@pip-compile --allow-unsafe --resolver=backtracking requirements.in
	@pip-compile --allow-unsafe --resolver=backtracking dev-requirements.in

sync:
	@pip-sync requirements.txt dev-requirements.txt

Collaborators can run make install after cloning the repository to set up their environment, then run make compile && make sync as needed to keep everything synchronized.

Python dependency management tools abound.

I know many Python devs that swear by poetry or pipenv. And I know plenty that still use good ol&apos pip freeze. Personally, pipenv and poetry are too heavy-handed for my taste. pip freeze feels tedious to me. And the new pdm project looks interesting but still has a long way to go.

Call me Goldilocks, but pip-tools is the porridge that&aposs just right.

Other pipx Apps To Consider

Depending on the kind of development you do, the following packages may be useful tools to install as applications with pipx:

• cookiecutter for scaffolding projects.
• pre-commit for managing pre-commit hooks.
• tox for testing and task automation.
• build frontend for building Python packages.

Dig Deeper

If you write and distribute Python packages, then you need to read Dan Hillard&aposs book Publishing Python Packages.

In addition to covering how to set up, publish, maintain, and scale your package, Dane discusses setting up a professional development environment specifically for package development. It is far more versatile than the environment described in this guide.

Get instant access from Manning, or buy a print version from Amazon.