Python Linux users would have noticed that python is now an "externally managed environment" on newer releases. I suspect that it has caused many frustrations. It certainly did for me - at least initially. Marcos, a long term friend of TurnKey recently reached out to me to ask about the best way to work around this when developing on our Odoo appliance.
The issue
Before I go on, for those of you who are not sure what I'm talking about, try installing or updating a python package via pip. It will fail with an error message:
error: externally-managed-environment
× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
python3-xyz, where xyz is the package you are trying to
install.
If you wish to install a non-Debian-packaged Python package,
create a virtual environment using python3 -m venv path/to/venv.
Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
sure you have python3-full installed.
If you wish to install a non-Debian packaged Python application,
it may be easiest to use pipx install xyz, which will manage a
virtual environment for you. Make sure you have pipx installed.
See /usr/share/doc/python3.11/README.venv for more information.
note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.Whilst this does fix a legitmate issue, it's also a big PITA for developers. To read more about the rationale, please see PEP 668.
Resolution options
As per the message, installing Python packages via apt is preferable. But what about if you need a python library not available in Debian? Or a newer version that what Debian provides? As noted by the error message, using a Python venv is the next best option. But that means duplication of any apt packages you may already have installed therefore bloat. It also means that you miss out on the automated security updates that TurnKey provides for Debian packages. The only remaining option is to "break system packages". That doesn't sound good! It will revert your system to the behavior before the application of PEP 668 - thus making life more complicated in the future...
Pros and Cons of each approach.
Assuming that you want/need versions and/or packages not available in Debian, what is the best path? Obviously each option note has pros and cons, so which way should you go? In his message to me, Marcos nicely laid out the pros and cons of each of the 2 suggested approaches, so I'll share them here:
Virtual Environment
Pros:- Isolates application dependencies, avoiding conflicts with system packages.
- Allows for more flexible and up-to-date package management.
- Adds complexity to the setup and maintenance process.
- Increases the overall footprint and resource requirements of the deployment.
System-Wide Installation
Pros:- Simpler setup and integration with the system.
- Utilizes the standard Turnkey Linux deployment model.
- Potential conflicts with system-managed packages.
- Limited by the constraints imposed by Debian 12.
Another - perhaps better - option
Another option not noted in the pip error message is to create a virtual environment, with the system python passed through. Whilst it's still not perfect, in my option it is by far the best option - unless of course you can get by just using system packages alone. TBH, I'm a bit surprised that it's not noted in the error message. It's pretty easy to set up, just include adding the '--system-site-packages' switch when creating your virtual environment. I.e.:
python3 -m venv --system-site-packages /path/to/venv
What you get with this approach
Let's have a look at what you get when using '--system-site-packages'. First let's create an example venv. Note that all of this is running as root from root's home (/root). Although for any AWS Marketplace users (or non TurnKey users) most of these commands should work fine as a "sudo" user (for apt installs).
root@core ~# mkdir ~/test_venv root@core ~# python3 -m venv --system-site-packages ~/test_venv root@core ~# source ~/test_venv/bin/activate (test_venv) root@core ~#
Now for a demonstration of what happens when you use it.
I'll use a couple of apt packages with my examples:
- python3-pygments (initially installed)
- python3-socks (initially not-installed)
Continuing on from creating the venv above, let's confirm the package versions and status:
(test_venv) root@core ~# apt list python3-pygments python3-socks Listing... Done python3-pygments/stable,now 2.14.0+dfsg-1 all [installed,automatic] python3-socks/stable 1.7.1+dfsg-1 all
So we have python3-pygments installed and it's version 2.14.0. python3-socks is not installed, but the available version is 1.7.1. Now let's check that the installed package (pygments) is available in the venv and that it's the system version. For those not familiar with grep, the grep command does a case-insensitive search for lines that include socks or pygments.
(test_venv) root@core ~# pip list | grep -i 'socks\|pygments' Pygments 2.14.0
Let's install python3-socks and check the status again:
(test_venv) root@core ~# apt install -y python3-socks [...] (test_venv) root@core ~# apt list python3-pygments python3-socks Listing... Done python3-pygments/stable,now 2.14.0+dfsg-1 all [installed,automatic] python3-socks/stable,now 1.7.1+dfsg-1 all [installed]
Ok so python3-socks is installed now. And it's instantly available in our venv:
(test_venv) root@core ~# pip list | grep -i 'socks\|pygments' Pygments 2.14.0 PySocks 1.7.1
Woohoo! :) And we can still install and/or update packages in our venv with pip?:
(test_venv2) root@core ~# pip install --upgrade pygments
Requirement already satisfied: pygments in /usr/lib/python3/dist-packages (2.14.0)
Collecting pygments
Downloading pygments-2.18.0-py3-none-any.whl (1.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 6.5 MB/s eta 0:00:00
Installing collected packages: pygments
Attempting uninstall: pygments
Found existing installation: Pygments 2.14.0
Not uninstalling pygments at /usr/lib/python3/dist-packages, outside environment /root/test_venv2
Can't uninstall 'Pygments'. No files were found to uninstall.
Successfully installed pygments-2.18.0
Yes! When using pip install in our venv there are some extra lines related to the system package, but otherwise it's the same as using a standalone venv. Let's double check the new version:
(test_venv) root@core ~# pip list | grep -i 'socks\|pygments' Pygments 2.18.0 PySocks 1.7.1
So we've updated from the system version of Pygments to 2.18.0, but the system version still exists - and is still 2.14.0:
(test_venv) root@core ~# apt list python3-pygments Listing... Done python3-pygments/stable,now 2.14.0+dfsg-1 all [installed,automatic]
So what happens if we remove the pip installed version?:
(test_venv) root@core ~# pip uninstall pygments
Found existing installation: Pygments 2.18.0
Uninstalling Pygments-2.18.0:
Would remove:
/root/test_venv2/bin/pygmentize
/root/test_venv2/lib/python3.11/site-packages/pygments-2.18.0.dist-info/*
/root/test_venv2/lib/python3.11/site-packages/pygments/*
Proceed (Y/n)? y
Successfully uninstalled Pygments-2.18.0
This time there is no mention of the system package. Let's double check the system and the venv:
(test_venv) root@core ~# apt list python3-pygments Listing... Done python3-pygments/stable,now 2.14.0+dfsg-1 all [installed,automatic] (test_venv) root@core ~# pip list | grep -i 'pygments' Pygments 2.14.0
Yep, the system package is still there and it's still in the venv!
The best of both worlds
So using '--system-site-packages' is essentially the best of both worlds. Where possible you can use system packages via apt, but you still have all the advantages of a virtual environment. In my opinion it's the best option by far! What do you think? Feel free to share your thoughts and feedback below.