Giampaolo Rodola: psutil 3.0

Back then I didn't know any C (and I still am a terrible C developer) so he's been crucial to develop the initial psutil skeleton including OSX and Windows support. I'm back home now (but not for long ;-)), so I finally have some time to write this blog post and tell you about the new psutil release. Let's see what happened.

net_if_addrs()

In a few words, we're now able to list network interface addresses similarly to "ifconfig" command on UNIX:

>>> import psutil
>>> psutil.net_if_addrs()
{'eth0': [snic(
            family=<AddressFamily.AF_INET: 2>, 
            address='10.0.0.4', 
            netmask='255.0.0.0', 
            broadcast='10.255.255.255'),
          snic(
            family=<AddressFamily.AF_PACKET: 17>, 
            address='9c:eb:e8:0b:05:1f', 
            netmask=None, 
            broadcast='ff:ff:ff:ff:ff:ff')],
 'lo': [snic(
            family=<AddressFamily.AF_INET: 2>, 
            address='127.0.0.1', 
            netmask='255.0.0.0', 
            broadcast='127.0.0.1'),
        snic(
            family=<AddressFamily.AF_PACKET: 17>, 
            address='00:00:00:00:00:00', 
            netmask=None, 
            broadcast='00:00:00:00:00:00')]}

This is limited to AF_INET (IPv4), AF_INET6 (IPv6) and AF_LINK (ETHERNET) address families. If you want something more poweful (e.g. AF_BLUETOOTH) you can take a look at netifaces extension. And here's the code which does these tricks on POSIX and Windows:

• POSIX implementation
• Windows implementation

Also, here's some doc.

net_if_stats()

This will return a bunch of information about network interface cards:

>>> import psutil
>>> psutil.net_if_stats()
{'eth0': snicstats(
    isup=True, 
    duplex=<NicDuplex.NIC_DUPLEX_FULL: 2>, 
    speed=100, 
    mtu=1500),
 'lo': snicstats(
    isup=True, 
    duplex=<NicDuplex.NIC_DUPLEX_UNKNOWN: 0>, 
    speed=0, 
    mtu=65536)}

Again, here's the code:

• Windows implementation
• Linux implementation
• OSX / FreeBSD implementation
• SunOS implementation

...and the doc.

Enums

Enums are a nice new feature introduced in Python 3.4. Very briefly (or at least, this is what I appreciate the most about them), they help you write an API with human-readable constants. If you use Python 2 you'll see something like this:

>>> import psutil
>>> psutil.IOPRIO_CLASS_IDLE
3

On Python 3.4 you'll see a more informative:

>>> import psutil
>>> psutil.IOPRIO_CLASS_IDLE
<IOPriority.IOPRIO_CLASS_IDLE: 3>

They are backward compatible, meaning if you're sending serialized data produced with psutil through the network you can safely use comparison operators and so on. The psutil APIs returning enums (on Python >=3.4) are:

• psutil.net_connections() (the address families):
• psutil.Process.connections() (same as above)
• psutil.net_if_stats() (all NIC_DUPLEX_* constants)
• psutil.Process.nice() on Windows (for all the *_PRIORITY_CLASS constants)
• psutil.Process.ionice() on Linux (for all the IOPRIO_CLASS_* constants)

All the other existing constants remained plain strings (STATUS_*) or integers (CONN_*).

Zombie processes

This is a big one. The full story is here but basically the support for zombie processes on UNIX was broken (except on Linux - Windows doesn't have zombie processes). Up until psutil 2.* we could instantiate a zombie process:

>>> pid = create_zombie()
>>> p = psutil.Process(pid)

...but every time we queried it we got a NoSuchProcess exception:

>>> psutil.name()
  File "psutil/__init__.py", line 374, in _init
    raise NoSuchProcess(pid, None, msg)
psutil.NoSuchProcess: no process found with pid 123

That was misleading though because the PID technically still existed:

>>> psutil.pid_exists(p.pid)
True

Furthermore, depending on what platform you were on, certain process stats could still be queried (instead of raising NoSuchProcess):

>>> psutil.cmdline()
['python']

Also process_iter() did not return zombie processes at all. This was probably the worst aspect because being able to identify them is an important use case, as they signal an issue with process: if a parent process spawns a child, terminates it (via kill()), but doesn't wait() for it it will create a zombie. Long story short, the way this changed in psutil 3.0 is that:

• we now have a new ZombieProcess exception, raised every time we're not able to query a process because it's a zombie
• it is raised instead of NoSuchProcess (which was incorrect and misleading)
• it is still backward compatible (meaning you won't have to change your old code) because it inherits from NoSuchProcess
• process_iter() finally works, meaning you can safely identify zombie processes like this:

import psutil
zombies = []
for p in psutil.process_iter():
    try:
        if p.status() == psutil.STATUS_ZOMBIE:  
            zombies.append(p)
    except NoSuchProcess:
        pass

Removal of deprecated APIs

This is another big one, probably the biggest. In a previous blog post I already talked about deprecated APIs. What I did back then (January 2014) was to rename and officially deprecate different APIs and provide aliases for them so that people wouldn't yell at me because I broke their existent code. The most interesting deprecation was certainly the one affecting module constants and the hack which was used in order to provide "module properties". With this new release I decided to get rid of all those aliases. I'm sure this will cause problems but hey! This is a new major release, right? =). Plus the amount of crap which was removed is impressive (see the commit). Here's the old aliases which are now gone for good (or bad, depending on how much headache they will cause you):

Removed module functions and constants

Already deprecated name	New name
psutil.BOOT_TIME()	psutil.boot_time()
psutil.NUM_CPUS()	psutil.cpu_count()
psutil.TOTAL_PHYMEM()	psutil.virtual_memory().total
psutil.avail_phymem()	psutil.virtual_memory().free
psutil.avail_virtmem()	psutil.swap_memory().free
psutil.cached_phymem() (Linux only)	psutil.virtual_memory().cached
psutil.get_pid_list()	psutil.pids().cached
psutil.get_process_list()	-
psutil.get_users()	psutil.users()
psutil.network_io_counters()	psutil.net_io_counters()
psutil.phymem_buffers() (Linux only)	psutil.virtual_memory().buffers
psutil.phymem_usage()	psutil.virtual_memory()
psutil.total_virtmem()	psutil.swap_memory().total
psutil.used_virtmem()	psutil.swap_memory().used
psutil.used_phymem()	psutil.virtual_memory().used
psutil.virtmem_usage()	psutil.swap_memory()

Process methods (assuming p = psutil.Process()):

Already deprecated name	New name
p.get_children()	p.children()
p.get_connections()	p.connections()
p.get_cpu_affinity()	p.cpu_affinity()
p.get_cpu_percent()	p.cpu_percent()
p.get_cpu_times()	p.cpu_times()
p.get_io_counters()	p.io_counters()
p.get_ionice()	p.ionice()
p.get_memory_info()	p.memory_info()
p.get_ext_memory_info()	p.memory_info_ex()
p.get_memory_maps()	p.memory_maps()
p.get_memory_percent()	p.memory_percent()
p.get_nice()	p.nice()
p.get_num_ctx_switches()	p.num_ctx_switches()
p.get_num_fds()	p.num_fds()
p.get_num_threads()	p.num_threads()
p.get_open_files()	p.open_files()
p.get_rlimit()	p.rlimit()
p.get_threads()	p.threads()
p.getcwd()	p.cwd()
p.set_cpu_affinity()	p.cpu_affinity()
p.set_ionice()	p.ionice()
p.set_nice()	p.nice()
p.set_rlimit()	p.rlimit()

If your code suddenly breaks with AttributeError after you upgraded psutil it means you were using one of those deprecated aliases. In that case just take a look at the table above and rename stuff in accordance.

Bug fixes

I fixed a lot of stuff (full list here), but here's the list of things which I think are worth mentioning:

• #512: [FreeBSD] fix segfault in net_connections().
• #593: [FreeBSD] Process.memory_maps() segfaults.
• #606: Process.parent() may swallow NoSuchProcess exceptions.
• #614: [Linux]: cpu_count(logical=False) return the number of physical CPUs instead of physical cores.
• #628: [Linux] Process.name() truncates process name in case it contains spaces or parentheses.

Ease of development

These are not enhancements you will directly benefit from but I put some effort into making my life easier every time I work on psutil.

• I care about psutil code being fully PEP8 compliant so I added a pre-commit GIT hook which runs flake8 on every commit and rejects it if the coding style is not compliant. The way I install this is via make install-git-hooks.
• I added a make install-dev-deps command which installs all deps and stuff which is useful for testing (ipdb, coverage, etc).
• A new make coverage command which runs coverage. With this I discovered some of parts in the code which weren't covered by tests and I fixed that.
• I started using tox to easily test psutil against all supported Python versions (from 2.6 to 3.4) in one shot.
• I reorganized tests so that now they can be easily executed with py.test and nose (before, only unittest runner was fully supported)

Final words