import sys
from PIL import Image, ImageSequence
try:
        img = Image.open(sys.argv[1])
except IOError:
        print "Cant load", infile
        sys.exit(1)

pal = img.getpalette()
prev = img.convert('RGBA')
prev_dispose = True
for i, frame in enumerate(ImageSequence.Iterator(img)):
    dispose = frame.dispose

    if frame.tile:
        x0, y0, x1, y1 = frame.tile[0][1]
        if not frame.palette.dirty:
            frame.putpalette(pal)
        frame = frame.crop((x0, y0, x1, y1))
        bbox = (x0, y0, x1, y1)
    else:
        bbox = None

    if dispose is None:
        prev.paste(frame, bbox, frame.convert('RGBA'))
        prev.save('result_%03d.png' % i)
        prev_dispose = False
    else:
        if prev_dispose:
            prev = Image.new('RGBA', img.size, (0, 0, 0, 0))
        out = prev.copy()
        out.paste(frame, bbox, frame.convert('RGBA'))
        out.save('result_%03d.png' % i)

C:\Python27>python.exe convert_gif.py 0001.gif

We received 195 proposals for talks at PyGotham this year. Now we have to find the best 50 or so. For the first time, we’re asking the community to vote on their favorite talks. Voting will close August 7th; then I and my comrades on the Program Committee will make a final selection.

Your Mission, If You Choose To Accept It

We need your help judging which proposals are the highest quality and the best fit for our community’s interests. For each talk we’ll ask you one question: “Would you like to see this talk at PyGotham?” Remember, PyGotham isn’t just about Python: it’s an eclectic conference about open source technology, policy, and culture.

You can give each talk one of:

• +1“I would definitely like to see this talk”
•  0“I have no preference on this talk”
• -1“I do not think this talk should be in PyGotham”

You can sign up for an account and begin voting at vote.pygotham.org. The site presents you with talks in random order, omitting the ones you have already voted on. For each talk, you will see this form:

Click “Save Vote” to make sure your vote is recorded. Once you do, a button appears to jump to the next proposal.

Our thanks to Ned Jackson Lovely, who made this possible by sharing the talk voting app “progcom” that was developed for the PyCon US committee.

So far, about 50 people have cast votes. We need to hear from you, too. Please help us shape this October’s PyGotham. Vote today!

Image: Voting in Brisbane, 1937

• What the bylaws used to say

• What the bylaws now say

• Why was this change made?

• What the bylaws used to say

• What the bylaws now say

• Why was this change made?

• What the bylaws used to say

• What the bylaws now say

1. Naomi Ceder (3 yr)
2. Eric Holscher (3 yr)
3. Jackie Kazil (3 yr)
4. Paul Hildebrandt (3 yr)
5. Lorena Mesa (2 yr)
6. Thomas Wouters (2 yr)
7. Kushal Das (2 yr)
8. Marlene Mhangami (2 yr)
9. Kenneth Reitz (1 yr)
10. Trey Hunner (1 yr)
11. Paola Katherine Pacheco (1 yr)

• Why was this change made?

• What the bylaws used to say

• What the bylaws now say

• Why was this change made?

1. Van Lindberg
2. Ewa Jodlowska

• What the bylaws used to say

• What the bylaws now say

• Why was this change made?

We had several outages over the last few days. The problem appears to be fixed now, but investigations into the underlying cause are still underway. This post is a summary of what happened, and what we know so far. Once we've got a better understanding of the issue, we'll post more.

It's worth saying at the outset that while the problems related to the way we manage our users' files, those files themselves were always safe. While availability problems are clearly a big issue, we regard data integrity as more important.

20 July: the system update

On Thursday 20 July, at 05:00 UTC, we released a new system update for PythonAnywhere. This was largely an infrastructural update. In particular, we updated our file servers from Ubuntu 14.04 to 16.04, as part of a general upgrade of all servers in our cluster.

File servers are, of course, the servers that manage the files in your PythonAnywhere disk storage. Each server handles the data for a specific set of users, and serves the files up to the other servers in the cluster that need them -- the "execution" servers where your websites, your scheduled tasks, and your consoles run. The files themselves are stored on network-attached storage (and mirrored in realtime to redundant disks on a separate set of backup servers); the file servers simply act as NFS servers and manage a few simple things like disk quotas.

While the system update took a little longer than we'd planned, once everything was up and running, the system looked stable and all monitoring looked good.

20 July: initial problems

At 12:07 UTC our monitoring system showed a very brief issue. From some of our web servers, it appeared that access to one of our file servers, file-2, had very briefly slowed right down -- it was taking more than 30 seconds to list the specific directory that is monitored. The problem cleared up after about 60 seconds. Other file servers were completely unaffected. We did some investigations, but couldn't find anything, so we chalked it up as a glitch and kept an eye out for further problems.

At 14:12 UTC it happened again, and then over the course of the afternoon, the "glitches" became more frequent and started lasting longer. We discovered that the symptom from the file server's side was that all of the NFS daemons -- the processes that together make up an NFS server -- would all become busy; system load would rise from about 1.5 to 64 or so. They were all waiting uninterruptably on what we think was disk I/O (status "D" in top).

The problem only affected file-2 -- other file servers were all fine. Given that every file server had been upgraded to an identical system image, our initial suspicion was that there might be some kind of hardware problem. At 17:20 UTC we got in touch with AWS to discuss whether this was likely.

By 19:10 our discussions with AWS had revealed nothing of interest. The "glitches" had become a noticeable problem for users, and we decided that while there was no obvious sign of hardware problems, it would be good to at least eliminate that as a possible cause, so we took a snapshot of all disks containing user data (for safety), then migrated the server to new hardware, causing a 20-minute outage for users on that file server (who were already seeing a serious slowdown anyway), and a 5-minute outage for everyone else, the latter because we had to reboot the execution servers

After this move, at 19:57 UTC, everything seemed OK. Our monitoring was clear, and the users we were in touch with confirmed that everything was looking good.

21 July: the problem continues

At 14:31 UTC on 21 July, we saw another glitch on our monitoring. Again, the problem cleared up quickly, but we started looking again into what could possibly be the cause. There were further glitches at 15:17 and 16:51, but then the problem seemed to clear up.

Unfortunately at 22:44 it flared up again. Again, the issues started happening more frequently, and lasting longer each time, until they became very noticeable for our users at around 00:30 UTC. At 00:55 UTC we decided to move the server to different hardware again -- there's no official way to force a move to new hardware on AWS; stopping and starting an instance usually does it, but there's a small chance you'd end up on the same physical host again, so a second attempt seemed worth-while. If nothing else, it would hopefully at least clear things up for another 12 hours or so and buy us time to work out what was really going wrong.

This time, things didn't go according to plan. The file server failed to come up on the new hardware, and trying to move again did not help. We decided that we were going to need to provision a completely fresh file server, and move the disks across. While we have processes in place for replacing file servers as part of a normal system update, and for moving them to new hardware without changing (for example) their IP address, replacing one under these circumstances is not a procedure we've done before. Luckily, it went as well as could be expected under the circumstances. At 01:23 UTC we'd worked out what to do and started the new file server. By 01:50 we'd started the migration, and by 02:20 UTC everything was moved over. There were a few remaining glitches, but these were cleared up by 02:45 UTC.

23 July -- more problems -- and a resolution?

We did not expect the fix we'd put in to be a permanent solution -- though we did have a faint hope that perhaps the problem had been caused by some configuration issue on file-2, which might have been remediated by our having provisioned a new server rather than simply moving the old one. This was never a particularly strong hope, however, and when the problems started again at 12:16 UTC we weren't entirely surprised.

We had generated two new hypotheses about the possible cause of these issues by now:

• The number of NFS daemons running on the machine was either too low or too high. Before our upgrade, we'd been running 8 daemons, and we'd moved to running 64 -- which we believed was the right number given the size of the machines in question, but of course we could have been wrong.
• The upgrade from Ubuntu 14.04 to 16.04 could have introduced some kind of bug at the NFS level.

The problem with both of these hypotheses was that only one of our file servers was affected. All file servers had the same number of workers, and all had been upgraded to 16.04.

Still, it was worth a try, we thought. We decided to try changing the number of daemon processes first, as it we believed it would cause minimal downtime; however, we started up a new file server on 14.04 so that it would be ready just in case.

At 14:41 UTC we reduced the number of workers down to eight. We were happy to see that this was picked up across the cluster without any need to reboot anything, so there was no downtime.

Unfortunately, at 15:04, we saw another problem. We decided to spend more time investigating a few ideas that had occurred to us before taking the system down again. At 19:00 we tried increasing the number of NFS processes to 128, but that didn't help. At 19:23 we decided to go ahead with switching over to the 14.04 server we'd prepared earlier. We kicked off some backup snapshots of the user data, just in case there were problems, and at 19:38 we started the migration over.

This completed at 19:46, but required a restart of all of the execution servers in order to pick up the new file server. We started this process immediately, and web servers came back online at 19:48, consoles at 19:50, and scheduled tasks at 19:55.

By 20:00 we were satisfied that everything looked good, and so we went back to monitoring.

Where we are now

Since the update on Saturday, there were no monitoring glitches at all on Sunday, but we did see one potential problem on Monday at 12:03. However this blip was only noticed from one of our web servers (previous issues affected at least three at a time, and sometimes as many as nine), and the problem has not been followed by any subsequent outages in the 4 hours since, which is somewhat reassuring.

We're continuing to monitor closely, and are brainstorming hypotheses to explain what might have happened (or, perhaps still be happening). Of particular interest is the fact that this issue only affected one of our file servers, despite all of them having been upgraded. One possibility we're considering is that the correlation in timing with the upgrade is simply a red herring -- that instead there's some kind of access pattern, some particular pattern of reads/writes to the storage, which only started at around midday on Thursday after the system update. We're planning possible ways to investigate that should the problem occur again.

Either way, whether the problem is solved now or not, we clearly have much more investigation to do. We'll post again when we have more information.

How to Install and Uninstall Python Packages Using Pip

A step-by-step introduction to basic Python package management skills with the “pip” command. Learn how to install and remove third-party modules from PyPI.

Python is approaching its third decade of good old age, and over the years many people have contributed to the creation of Python packages that perform specific functions and operations.

As of this writing, there are ~112K packages listed on the PyPI website. PyPI is short for “Python Package Index”, a central repository for free third-party Python modules.

This large and convenient module ecosystem is what makes Python so great to work with:

You see, most Python programmers are really assemblers of Python packages, which take care of a big chunk of the programming load required by modern applications.

Chances are that there is more than one Python package ready to be unleashed and help you with your specific programming needs.

For instance, while reading dbader.org, you may notice that the pages on the site render emoji quite nicely. You may wonder…

I’d like to use emoji on my Python app!

Is there a Python package for that?

Let’s find out!

Here’s what we’ll cover in this tutorial:

1. Finding Python Packages
2. What to Look for in a Python Package
3. Installing Python Packages With Pip
4. Capturing Installed Python Packages with Requirements Files
5. Visualizing Installed Packages
6. Installing Python Packages From a requirements.txt File
7. Uninstalling Python Packages With Pip
8. Summary & Conclusion

Finding Python Packages

Let’s use the emoji use case as an example. We find emoji related Python packages by visiting the PyPI website and searching for emoji via the search box on the top right corner of the page.

As of this writing, PyPI lists 94 packages, of which a partial list is shown below.

Notice the “Weight*” header of the middle column. That’s a key piece of information. The weight value is basically a search scoring number, which the site calculates for each package to rank them and list them accordingly.

If we read the footnote it tells us that the number is calculated by “the occurrence of search term weighted by field (name, summary, keywords, description, author, maintainer).”

Does that mean that the top one is the best package?

Not necessarily. Although uncommon, a package maintainer may stuff emoji into every field to try to top rank the package, which could well happen.

Conversely, many developers don’t do their homework and don’t bother filling out all the fields for their packages, which results in those packages being ranked lower.

You still need to research the packages listed, including a consideration for what your specific end use may be. For instance, a key question could be:

Which environment do you want to implement emoji on? A terminal-based app, or perhaps a Django web app?

If you are trying to display emoji on a django web app, you may be better off with the 10th package down the list shown above (package django-emoji 2.2.0).

For our use case, let’s assume that we are interested in emoji for a terminal based Python app.

Let’s check out the first one on our list (package emoji 0.4.5) by clicking on it.

What to Look for in a Python Package

The following are characteristics of a good Python package:

1. Decent documentation: By reading it we can get a clue as to whether the package could meet our need or not;
2. Maturity and stability: It’s been around for some time, proven by both its age and its successive versions;
3. Number of contributors: Healthy packages (especially complex ones) tend to have a healthy number of maintainers;
4. Maintenance: It undergoes maintenance on a regular basis (we live in an ever-evolving world).

Although I would check it out, I wouldn’t rely too much on the development status listed for each package, that is, whether it’s a 4 - Beta or 5 - Production/Stable package. That classification is in the eye of the package creator and not necessarily reliable.

On our emoji example, the documentation seems decent. At the top of the page, we get a graphical indication of the package at work (see snippet below), which shows emoji on a Python interpreter. Yay!

The documentation for our emoji package also tells us about installing it, how to contribute to its development, etc., and points us to a GitHub page for the package, which is a great source of useful information about it.

By visiting its GitHub page, we can glean from it that the package has been around for at least two years, was last maintained in the past couple of months, has been starred 300+ times, has been forked 58 times, and has 10 contributors.

It’s looking good! We have identified a good candidate to incorporate emoji-ing into our Python terminal app.

How do we go about installing it?

Installing Python Packages With Pip

At this time, I am assuming that you already have Python installed on your system. There is plenty of info out there as to how to accomplish that.

Once you install Python, you can check whether pip is installed by running pip --version on a terminal.

I get the following output:

$ pip --version
pip 9.0.1 from /Library/Frameworks/Python.framework/↵
Versions/3.5/lib/python3.5/site-packages (python 3.5)

Since Python 3.4, pip is bundled with the Python installation package. If for some reason it is not installed, go ahead and get it installed.

I highly recommend also that you use a virtual environment (and more specifically, virtualenvwrapper), a set of extensions that…

…include wrappers for creating and deleting virtual environments and otherwise managing your development workflow, making it easier to work on more than one project at a time without introducing conflicts in their dependencies.

For this tutorial, I have created a virtual environment called pip-tutorial, which you will see going forward. My other tutorial walks you through setting up Python and virtualenvwrapper on Windows.

Below you’ll see how package dependencies can bring complexity into our already complex development environments, which is the reason why using virtual environments is a must for Python development.

A great place to start learning about a terminal program is by running it without any options on the terminal. So, on your terminal, run pip. You would get a list of Commands and General Options.

Below is a partial list of the results on my terminal:

From there on you could run pip install --help to read on what the install command does and what you need to specify to run it, for example. Of course, reading the pip documentation is another great place to start.

$ pip install --help

Usage:
  pip install [options] <requirement specifier> [package-index-options] ...
  pip install [options] -r <requirements file> [package-index-options] ...
  pip install [options] [-e] <vcs project url> ...
  pip install [options] [-e] <local project path> ...
  pip install [options] <archive url/path> ...

Description:
  Install packages from:

  - PyPI (and other indexes) using requirement specifiers.
  - VCS project urls.
  - Local project directories.
  - Local or remote source archives.

  pip also supports installing from "requirements files", which provide
  an easy way to specify a whole environment to be installed.

Install Options:
  ...

Let’s take a quick detour and focus on the freeze command next, which will be a key one in dealing with dependencies. Running pip freeze displays a list of all installed Python packages. If I run it with my freshly installed virtual environment active, I should get an empty list, which is the case:

$ pip freeze

Now we can get the Python interpreter going by typing python on our terminal. Once that’s done, let’s try to import the emoji module, upon which python will complain that there isn’t such a module installed, and rightfully so for we haven’t installed it, yet:

$ python
Python 3.5.0 (default)
[GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.38)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import emoji
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'emoji'

To finally install the package, we can go ahead and run pip install emoji on our terminal. I get the following output:

$ pip install emoji==0.4.5
Collecting emoji==0.4.5
Installing collected packages: emoji
Successfully installed emoji-0.4.5

When installing packages with pip, we can constrain pip to install a version of our preference, by using the following operators:

A specific version of the package (==):

$ pip install emoji==0.4.1

A version other than the specified one (!=):

$ pip install emoji!=0.4.1

A version equal to or greater than a specific version (>=):

$ pip install emoji>=0.4.0

A version of the package in the specified range (>=X.Y.T, <=X.Y.Z):

$ pip install emoji>=0.4.0, <=0.4.9

For a full specification of the version specifiers, refer to this page. Generally the most useful specifier here is == to pip install a specific version of a package. If we don’t constrain pip, it will grab the latest version of the package.

You may be wondering why you would want to install an older version of a Python package in the first place:

• One good example of that is if you are following a tutorial which might have used the latest version of a package when it was written but which could be an older version by the time you are reading it. If you want to follow along, you would do well by installing the same version that the author used.

• Another example is that if you start writing code for an app today, chances are that the packages that you use today will evolve, and new versions of it will be released in the future (while your app is “stuck” with the versions that you use today).

Programmers freeze requirements to keep track of the versions of the different packages that are installed on development and production environments. One of the objectives is to be able to replicate the environments as needed. Dan’s course on Python dependency management goes into more detail on that topic.

Let’s continue on and run pip freeze again after installing the emoji package. You should now see it included in the list of all installed Python modules:

$ pip freeze
emoji==0.4.5

As expected, pip freeze now lists the emoji package as an added dependency with a specific version number.

I now go back to my Python interpreter session, and run import emoji, and this time around Python doesn’t complain, which is a good sign. I test it, and get the following output:

Success, at last! We just installed and then imported a third-party Python module. Great job 🙂

It’s typical for an application to have several interdependent packages. For instance, running pip freeze on the virtual environment that I use to develop tumblingprogrammer.com, will output the following list of modules:

appdirs==1.4.3
beautifulsoup4==4.6.0
Django==1.11.1
django-bootstrap3==8.2.3
django-crispy-forms==1.6.1
django-debug-toolbar==1.8
(...)pyparsing==2.2.0
pytz==2017.2
PyYAML==3.12
selenium==3.4.1
six==1.10.0
sqlparse==0.2.3
tornado==4.5.1

That’s a total of 25 Python packages. And it’s a fairly simple application. Later on I’ll describe a way to visualize the interdependency between packages.

Capturing Installed Python Packages with Requirements Files

Developers get in the habit of freezing requirements every time that a package or a dependency gets installed on their projects. We do that by running the following pip command:

$ pip freeze > requirements.txt

This dumps the output of pip freeze into a requirements.txt file on the working directory.

Let’s assume now that for some reason we need to install MarkupSafe version 0.11. Let’s assume also that we have gotten ahead, installed it, tested it, and that our app behaves as we expect it to.

Let’s run pip freeze, which would only output our two packages, as shown below:

$ pip freeze
emoji==0.4.5
MarkupSafe==0.11

To continue with our learning, let’s go ahead and install Flask, a popular web microframework. We’ll grab the latest version of it by running pip install flask.

I get the following output (if you are following along, yours might differ a little bit, for my computer had cached the files from a previous install):

$ pip install flask
Collecting flask
  Using cached Flask-0.12.2-py2.py3-none-any.whl
Collecting itsdangerous>=0.21 (from flask)
Collecting Jinja2>=2.4 (from flask)
  Using cached Jinja2-2.9.6-py2.py3-none-any.whl
Collecting click>=2.0 (from flask)
  Using cached click-6.7-py2.py3-none-any.whl
Collecting Werkzeug>=0.7 (from flask)
  Using cached Werkzeug-0.12.2-py2.py3-none-any.whl
Collecting MarkupSafe>=0.23 (from Jinja2>=2.4->flask)
Installing collected packages: itsdangerous, MarkupSafe, Jinja2, click, Werkzeug, flask
  Found existing installation: MarkupSafe 0.11
    Uninstalling MarkupSafe-0.11:
      Successfully uninstalled MarkupSafe-0.11
Successfully installed Jinja2-2.9.6 MarkupSafe-1.0 Werkzeug-0.12.2 click-6.7 flask-0.12.2 itsdangerous-0.24

Flask, being a more complex package, has some dependencies (Werkzeug, itsdangerous, etc.) which are installed with it automatically through the pip install command.

I want to call your attention to the following lines, extracted from the above listing:

...
  Found existing installation: MarkupSafe 0.11
    Uninstalling MarkupSafe-0.11:
      Successfully uninstalled MarkupSafe-0.11
...

Take a close look…

You’ll see that pip doesn’t have a way of reconciling conflicting dependencies. Without even warning us, it went ahead and replaced version 0.11 with version 1.0 of our MarkupSafe package. And that could be trouble for our application.

At that point in time, we run our app tests (assuming that have them), and dig into our application to make sure that the changes between 0.11 and 1.0 of the MarkupSafe package don’t break it.

If I were to face this situation in real life, I would roll back the changes first by uninstalling Flask and its dependencies and restore the packages that I had before. Then I would upgrade MarkupSafe to 1.0, test to make sure that the application works as expected. And then—and only then—would I re-install Flask.

Assuming that we have gone through rolling back, upgrading, testing, and re-installing Flask, if we run pip freeze now, we get 7 packages in total:

$ pip freeze
click==6.7
emoji==0.4.5
Flask==0.12.2
itsdangerous==0.24
Jinja2==2.9.6
MarkupSafe==1.0
Werkzeug==0.12.2

Let’s go ahead and freeze our requirements into a requirements.txt file by running pip freeze > requirements.txt.

Now we’re going to add another package with dependencies to increase the complexity of our setup. We’ll install version 6.0 of a package called alembic by running:

$ pip install alembic==0.6
Collecting alembic==0.6
Collecting Mako (from alembic==0.6)
Collecting SQLAlchemy>=0.7.3 (from alembic==0.6)
Requirement already satisfied: MarkupSafe>=0.9.2 in /Users/puma/.virtualenvs/pip-tutorial/lib/python3.5/site-packages (from Mako->alembic==0.6)
Installing collected packages: Mako, SQLAlchemy, alembic
Successfully installed Mako-1.0.7 SQLAlchemy-1.1.11 alembic-0.6.0

I now call your attention to the following line from the above listing:

...
Requirement already satisfied: MarkupSafe>=0.9.2 in /Users/puma/.virtualenvs/pip-tutorial/lib/python3.5/site-packages (from Mako->alembic==0.6)
...

Which means that alembic also depends on MarkupSafe. More complexity, huh? Let’s run pip freeze:

$ pip freeze
alembic==0.6.0
click==6.7
emoji==0.4.5
Flask==0.12.2
itsdangerous==0.24
Jinja2==2.9.6
Mako==1.0.7
MarkupSafe==1.0
SQLAlchemy==1.1.11
Werkzeug==0.12.2

The listing above showing all the packages on our emoji application is not very helpful at the moment, for it doesn’t give us info on dependencies (it only lists packages in alphabetical order). Let’s fix that.

Visualizing Installed Packages

One good package to have installed on our environment is pipdeptree, which shows the dependency tree of packages. Let’s go ahead and install the latest version of it by running the following command:

$ pip install pipdeptree

Once it’s done, let’s run pip freeze to see what we get:

$ pip freeze
alembic==0.6.0
click==6.7
emoji==0.4.5
Flask==0.12.2
itsdangerous==0.24
Jinja2==2.9.6
Mako==1.0.7
MarkupSafe==1.0
pipdeptree==0.10.1
SQLAlchemy==1.1.11
Werkzeug==0.12.2

We now get 11 packages, as we have added pipdeptree, which had no dependencies. Let’s run pipdeptree on the terminal to see what it does. Below is the output that I get on my machine:

$ pipdeptree
alembic==0.6.0
  - Mako [required: Any, installed: 1.0.7]
    - MarkupSafe [required: >=0.9.2, installed: 1.0]
  - SQLAlchemy [required: >=0.7.3, installed: 1.1.11]
emoji==0.4.5
Flask==0.12.2
  - click [required: >=2.0, installed: 6.7]
  - itsdangerous [required: >=0.21, installed: 0.24]
  - Jinja2 [required: >=2.4, installed: 2.9.6]
    - MarkupSafe [required: >=0.23, installed: 1.0]
  - Werkzeug [required: >=0.7, installed: 0.12.2]
pipdeptree==0.10.1
  - pip [required: >=6.0.0, installed: 9.0.1]
setuptools==36.2.0
wheel==0.29.0

We notice much more useful information here, including dependencies, and the minimum versions required for dependent packages to work properly.

Notice, once again, how MarkupSafe is listed twice, as both Jinja2 (and Flask) and Mako (and alembic) depend on it. That’s very useful information to troubleshoot things gone ugly.

We also notice other packages here that pip freeze doesn’t list, including pip, setuptools and wheel. The reason is that by default pip freeze doesn’t list packages that pip itself depends on.

We can use the --all flag to show also those packages. Let’s test this by running pip freeze --all, in which case we get:

$ pip freeze --all
alembic==0.6.0
click==6.7
emoji==0.4.5
Flask==0.12.2
itsdangerous==0.24
Jinja2==2.9.6
Mako==1.0.7
MarkupSafe==1.0
pip==9.0.1
pipdeptree==0.10.1
setuptools==36.2.0
SQLAlchemy==1.1.11
Werkzeug==0.12.2
wheel==0.29.0

Another benefit of using pipdeptree is that it warns us about conflicting dependencies, including circular ones (where packages depend on one another), but I have yet to see that in action. So far I couldn’t replicate the functionality on my system. You can find more info about the tool on its PyPI page.

Installing Python Packages From a requirements.txt File

If you have a requirements.txt file, you can install all the packages listed there by running the following command:

$ pip install -r /path/to/the/file/requirements.txt

This is very handy when we want to replicate environments and have access to a requirements.txt that reflects the makeup of it.

Uninstalling Python Packages With Pip

In this section you’ll see how to uninstall individual Python packages from your system or active virtual environment, how you can remove multiple packages at once with a single command, and how you can remove all installed Python packages.

Uninstalling individual packages:

You can do so by running, as an example, pip uninstall alembic. Let’s do that on our setup to see what happens. Here is the output on my end:

$ pip uninstall alembic
Uninstalling alembic-0.6.0:
  /Users/puma/.virtualenvs/pip-tutorial/bin/alembic
  ... a bunch on other files ...
  /Users/puma/.virtualenvs/pip-tutorial/lib/python3.5/site-packages/alembic/util.py
Proceed (y/n)? y
  Successfully uninstalled alembic-0.6.0

Let’s run pipdeptree to see what our setup looks like:

$ pipdeptree
emoji==0.4.5
Flask==0.12.2
  - click [required: >=2.0, installed: 6.7]
  - itsdangerous [required: >=0.21, installed: 0.24]
  - Jinja2 [required: >=2.4, installed: 2.9.6]
    - MarkupSafe [required: >=0.23, installed: 1.0]
  - Werkzeug [required: >=0.7, installed: 0.12.2]
Mako==1.0.7
  - MarkupSafe [required: >=0.9.2, installed: 1.0]
pipdeptree==0.10.1
  - pip [required: >=6.0.0, installed: 9.0.1]
setuptools==36.2.0
SQLAlchemy==1.1.11
wheel==0.29.0

If you look carefully, you may notice that the alembic dependencies are still present, because pip uninstall does not get rid of them, by design.

We have to manually do that (there are other options, which we will cover below). Therefore, it is extremely important that we freeze our requirements and commit changes to our requirements.txt file every time that we install or uninstall packages so we know what our setup should look like if we need to roll back changes.

Uninstalling multiple Python packages at once:

You can also uninstall several packages at once, by using the following command-line syntax:

$ pip uninstall package1 package2 ...

Another option is reading the list of packages to uninstall from a requirements file. Similar to its install counterpart, if you have a requirements.txt file, you can uninstall all the packages listed there like so:

$ pip uninstall -r /path/to/the/file/requirements.txt

Note that we could wipe out all the packages on our setup, which could actually be quite useful. Let’s take a look at an example.

The output below is a list of my git commits log (gl is an alias on my bash profile for a prettified git log):

$ gl
* 40f4f37 - (HEAD -> master) all packages in (37 minutes ago)<Jose Pumar>
* 2d00cf5 - emoji + markupsafe + flask + alembic (56 minutes ago)<Jose Pumar>
* e52002b - emoji + MarkupSafe + Flask (84 minutes ago)<Jose Pumar>
* 9c48895 - emoji + MarkupSafe (86 minutes ago)<Jose Pumar>
* 3a797b3 - emoji + MarkSafe (2 hours ago)<Jose Pumar>
* ... other commits...

If I change my mind and decide that I don’t need alembic any more, I could delete all the packages by running pip uninstall -r requirements.txt while on commit 40f4f37 (the HEAD).

If I do it, it gives me a bunch of warnings and asks me if I want to proceed several times (once for each package), to which I say yes. I could have avoided that by using the flag -y, as in:

$ pip uninstall -y -r requirements.txt

The -y flag tells pip not to ask for confirmation of uninstall deletions. If we run pip freeze after this operation, we’ll get an empty packages list, which is what we want.

We then checkout commit e52002b (the last safe commit before we installed alembic), and run pip install -r requirements.txt to reinstate the packages that we had at that point in time.

Removing all installed Python packages:

Sometimes it can be useful to remove all installed packages in a virtual environment or on your system Python install. It can help you get back to a clean slate.

Running the following command will uninstall all Python packages in the currently active environment:

$ pip freeze | xargs pip uninstall -y

This command works by first listing all installed packages using the freeze command, and then feeding the list of packages into the pip uninstall command to remove them.

Adding the -y flag automatically confirms the uninstallation so you don’t have to stick around hammering the “y” key on your keyboard.

Installing and Uninstalling Python Packages with the “pip” Package Manager – Conclusion

Although we covered a lot of ground and shed light on key commands and major challenges that you may face when dealing with installing and uninstalling Python packages and their dependencies.

In summary, the workflow for the installation of a Python package with pip is as follows:

1. Make sure that you are using a virtual environment.
2. Identify the need for a new package.
3. Research potential candidate packages: Look for the maturity of the package, its documentation, etc. See what you can find regarding its dependencies. For example, other packages that have to be installed so the package works properly. Sometimes the documentation refers to them.
4. Install the package and its dependent packages:pip will do this for you. Look for version upgrades in the pip installation log.
5. Test your application to make sure that the package meets your needs and that the package and or its dependent packages don’t break it.
6. Freeze your requirements: Run pip freeze > requirements.txt if tests show your application is still okay and works as intended.
7. Commit the changes to Git or the version control tool of your choice.
8. Repeat.

There is a lot more to cover, especially when it comes to dependency management, which has long-term implications on how we setup and configure our Python projects.

Such a complexity is one of the factors that make necessary to implement different settings and configurations to account for the distinct needs of our development, staging, and production environments.

Happy pip-ing!

If you're reading this you're probably familiar with how, in django-pipeline, you define bundles of static files to be combined and served. If you're not familiar with django-pipeline it's unlike this'll be of much help.

The Challenge (aka. the pitfall)

So you specify bundles by creating things in your settings.py something like this:

PIPELINE={'STYLESHEETS':{'colors':{'source_filenames':('css/core.css','css/colors/*.css','css/layers.css'),'output_filename':'css/colors.css','extra_context':{'media':'screen,projection',},},},'JAVASCRIPT':{'stats':{'source_filenames':('js/jquery.js','js/d3.js','js/collections/*.js','js/aplication.js',),'output_filename':'js/stats.js',}}}

You do a bit more configuration and now, when you run ./manage.py collectstatic --noinput Django and django-pipeline will gather up all static files from all Django apps installed, then start post processing then and doing things like concatenating them into one file and doing stuff like minification etc.

The problem is, if you look at the example snippet above, there's a typo. Instead of js/application.js it's accidentally js/aplication.js. Oh noes!!

What's sad is it that nobody will notice (running ./manage.py collectstatic will exit with a 0). At least not unless you do some careful manual reviewing. Perhaps you will notice later, when you've pushed the site to prod, that the output file js/stats.js actually doesn't contain the code from js/application.js.

Or, you can automate it!

A Solution (aka. the hack)

I started this work this morning because the error actually happened to us. Thankfully not in production but our staging server produced a rendered HTML page with <link href="/static/css/report.min.cd784b4a5e2d.css" rel="stylesheet" type="text/css" /> which was an actual file but it was 0 bytes.

It wasn't that hard to figure out what the problem was because of the context of recent changes but it would have been nice to catch this during continuous integration.

So what we did was add an extra class to settings.STATICFILES_FINDERS called myproject.base.finders.LeftoverPipelineFinder. So now it looks like this:

# in settings.pySTATICFILES_FINDERS=('django.contrib.staticfiles.finders.FileSystemFinder','django.contrib.staticfiles.finders.AppDirectoriesFinder','pipeline.finders.PipelineFinder','myproject.finders.LeftoverPipelineFinder',# the new hotness!)

And here's the class implementation:

frompipeline.findersimportPipelineFinderfromdjango.confimportsettingsfromdjango.core.exceptionsimportImproperlyConfiguredclassLeftoverPipelineFinder(PipelineFinder):"""This finder is expected to come AFTER     django.contrib.staticfiles.finders.FileSystemFinder and     django.contrib.staticfiles.finders.AppDirectoriesFinder in     settings.STATICFILES_FINDERS.    If a path is looked for here it means it's trying to find a file    that none of the regular staticfiles finders couldn't find."""deffind(self,path,all=False):# Before we raise an error, try to find out where,# in the bundles, this was defined. This will make it easier to correct# the mistake.forconfig_namein'STYLESHEETS','JAVASCRIPT':config=settings.PIPELINE[config_name]forkeyinconfig:ifpathinconfig[key]['source_filenames']:raiseImproperlyConfigured('Static file {!r} can not be found anywhere. Defined in '"PIPELINE[{!r}][{!r}]['source_filenames']".format(path,config_name,key,))# If the file can't be found AND it's not in bundles, there's# got to be something else really wrong.raiseNotImplementedError(path)

Now, if you have a typo or something in your bundles, you'll get a nice error about it as soon as you try to run collectstatic. For example:

▶ ./manage.py collectstatic --noinput
Post-processed 'css/search.min.css' as 'css/search.min.css'
Post-processed 'css/base.min.css' as 'css/base.min.css'
Post-processed 'css/base-dynamic.min.css' as 'css/base-dynamic.min.css'
Post-processed 'js/google-analytics.min.js' as 'js/google-analytics.min.js'
Traceback (most recent call last):
...
django.core.exceptions.ImproperlyConfigured: Static file 'js/aplication.js' can not be found anywhere. Defined in PIPELINE['JAVASCRIPT']['stats']['source_filenames']

Final Thoughts

This was a morning hack. I'm still not entirely sure if this the best approach, but there was none better and the result is pretty good.

We run ./manage.py collectstatic --noinput in our continous integration just before it runs ./manage.py test. So if you make a Pull Request that has a typo in bundles.py it will get caught.

Unfortunately, it won't find missing files if you use foo*.js or something like that. django-pipeline uses glob.glob to convert expressions like that into a list of actual files and that depends on the filesystem and all of that happens before the django.contrib.staticfiles.finders.find function is called.

If you have any better suggestions to solve this, please let me know.

Writing games in Python is fun, so how about using it to teach computer programming? Paul Craven is both a professor and the creator of Arcade, a 2d game library for Python. He’s doing a webinar with us next week, so we talked to him about teaching Python, using Python 3 type hints, why another Python game library, and how IDEs fit into teaching.

Thanks a bunch for doing the webinar next week. First, can you tell us a little bit about yourself?

I worked in the IT industry for 15 years before switching to teaching at Simpson College, a small 4-year college in Iowa. My main interest has been getting first-time programmers to realize that programming can be fun. That moment when a student cheers out loud because they finally figured out how to get sprites to move correctly on the screen? It is a beautiful thing to see.

You teach programming, and you created the Arcade game library to help. Before talking about Arcade, can you explain the motivation behind having a framework that you can teach?

Teaching is like engineering. Each semester you work to improve how you teach students. I had been using the Pygame library. But I wanted a library that I could improve on based on what I saw from students. For example:

Function and parameter names that students intuitively understand. Each year I had to teach them “set_mode” opens a window. Why not just have the function called “open_window”?
Support for functions students which ask for. In Pygame drawing ellipses with thick borders always had a moire pattern because of a bug in the library. And you can’t tilt ellipses when you draw them. Every year I have a student that wants to draw a football. And each year they were frustrated that it looked awful. I wanted a library where it just worked.
Students would download a graphic for their sprite. But it would be too large and there was no easy way to scale the image. That always resulted in hours of wasted time explaining how to do the scaling. What if the library just supported scaling?

After a while I collected such a long list of things like that, I decided to create a game library that where I wouldn’t have to teach “around” these issues.

Beyond using it for teaching, can you talk a bit about Arcade? What is it, how is it different, who might want to use it?

Arcade is great for sprite-based 2D games. It is simple and easy to get started with the library. There is no need to learn a complex framework. If you’ve wanted to create a game for fun, but Kivy, Unity, Unreal would just take more time to learn than what you’ve got, Arcade is a better choice. If you want to quickly create a scientific visualization without a lot of overhead code, Arcade can help you there too.

Arcade uses OpenGL and Pyglet. With OpenGL acceleration, it can draw a lot of sprites fast.

I use Arcade for my PyCharm tutorials and Arcade’s Python 3.6 type hinting is a big reason. Can you talk about your experience, as a library author and teacher, with type hinting?

New programmers often get confused when calling functions and methods. What data does it expect? And when the program doesn’t run, the students aren’t yet experts in reading stack traces. So they are stuck with a 500 line program that doesn’t work and they don’t know why. Frustrating.

Type hinting can sometimes tell students that they are passing unexpected data to the function. It does this before they run the program. Before they’ve even moved to writing the next line of code. It seems trivial, but it’s not. I found students able to create better, more complex programs because PyCharm and type hinting kept them from that error and allowed them to move on.

You also use PyCharm CE with your students. What’s been your experience having beginners start with an IDE?

I’ve taught students with an IDE and without an IDE. The biggest advantage is how the IDE can help students catch errors early. PyCharm’s built-in PEP-8 checking is huge. Also, built in spell-checking! Imagine trying to read program comments written without a spell-checker from today’s student. Students come up with some interesting ways to spell words.