1. Why does b+=(4,) work and b = b + (4,) doesn't work when b is a list? - [61/6]
2. What exactly is meant by "partial function" in functional programming? - [19/3]
3. How do I remove 'None' items from the end of a list in Python - [13/3]
4. Fastest algorithm to find the largest palindrome that is the product of 2 numbers with the same number of digits - [7/3]
5. Problem with autocompletion with Pandas in Jupyter - [7/1]
6. Chapel-Python integration questions - [7/1]
7. How to get Conda and Virtualenv to work on mac OS Catalina? - [6/7]
8. How to make tkinter frames in a loop and update object values? - [6/3]
9. Why does using `or` within an except clause not cause a SyntaxError? Is there a valid use for it? - [6/2]
10. Speeding up pandas profiling analysis using check_correlation? - [6/2]

This tutorial describes various ways to convert Python string to int and from an integer to string. You may often need to perform such operations in day to day programming. Hence, you should know them to write better programs. Also, an integer can be represented in different bases, so we’ll explain that too in this post. And there happen to be scenarios where conversion fails. Hence, you should consider such cases as well and can find a full reference given here with examples. By the way, it will be useful if you have some elementary knowledge about the Python data

The post How to Convert Python String to Int and Back to String appeared first on Learn Programming and Software Testing.

The post Thousands of Scientific Papers May be Invalid Due to Misunderstanding Python appeared first on The Mouse Vs. The Python.

As a reminder… this Wednesday (Oct 16) I’m giving a webinar on React+TypeScript+TDD in PyCharm. I’m doing some blog posts about material that will be covered.

See the first blog post for some background on this webinar and its topic.

Spotlight: Class Components With Props

As we saw in the previous step, child components get props from the parent, and when using TypeScript, you can model that contract with an interface. In his Class Components With Props tutorial step, we switch from a stateless functional component to a class component.

We do this change, of course, by first writing a failing test. Then, as we do the change, we use IDE features to speed us up, both in the test writing as well as the code changes. At no time do we switch to the browser and disrupt our “flow”.

Here’s the full video that accompanies the writeup and code:

I previously wrote a post about shipping a PyGame app to users on macOS. It’s now substantially updated for the new Notarization requirements in Catalina. I hope it’s useful to somebody!

A few nights ago I wrote daudin, a command-line shell based on Python. It allows you to easily mix UNIX and Python on the command line.

Source code and documentation: https://github.com/terrycojones/daudin.

Install via pip install daudin.

Daudin was a French zoologist who named the Python genus in 1826.

The post How to Read SAS Files in Python with Pandas appeared first on Erik Marsja.

In this post, we are going to learn how to read SAS (.sas7dbat) files in Python.

As previously described (in the read .sav files in Python post) Python is a general-purpose language that also can be used for doing data analysis and data visualization.

One potential downside, however, is that Python is not really user-friendly for data storage. This has, of course, lead to that our data many times are stored using Excel, SPSS, SAS, or similar software. See, for instance, the posts about reading .sav and .xlxs files in Python:

• How to read and write SPSS files in Python
• How to read and write Excel files in Pandas

Can I Open a SAS File in Python?

Now we may want to answer the question whether how to open a SAS file in Python? In Python, there the two useful packages Pyreadstat, and Pandas that enables us to open SAS files. If we are working with Pandas, the   read_sas method will load a .sav file into a Pandas dataframe. Note, Pyreadstat that is dependent on Pandas, will also create a Pandas dataframe from a .sas file.

How to Open a SAS file in Python

In this secion, we are going to learn how to load a SAS file in Python using the Python package Pyreadstat. Of course, before we use Pyreadstat we need to make sure we have it installed.

How to install Pyreadstat:

Pyreadstat can be installed either using pip or conda:

1. Install Pyreadstat using pip:
   Open up a terminal, or Windows PowerShell, and type pip install pyreadstat
   
2. Install using Conda:
   Open up a terminal, or Windows PowerShell, and type conda install -c conda-forge pyreadstat

How to Load a .sas7bdat File in Python Using Pyreadstat

In this section, we are going to use pyreadstat to import data into a Pandas dataframe. First, we import pyreadstat:

import pyreadstat

Now, we are ready to import SAS files using the method read_sas7bdat (download airline.sas7dbat). Note that, when we load a file using the Pyreadstat package, recognize that it will look for the file in Python’s working directory.

df, meta = pyreadstat.read_sas7bdat('airline.sas7bdat')

In the code chunk above we create two variables; df, and meta. As can be seen when using type the variable “df” is a Pandas dataframe:

type(df)

Thus, we can use all methods available for Pandas dataframe objects. In the next line of code, we are going to print the 5 first rows of the dataframe using pandas head method.

df.head()

See more about working with Pandas dataframes in the following tutorials:

• Python Groupby Tutorial: Here you will learn about working the the groupby method to group Pandas dataframes.
• Learn how to take random samples from a pandas dataframe
• A more general, overview, of how to work with Pandas dataframe objects can be found in the Pandas Dataframe tutorial.

How to Read a SAS file with Python Using Pandas

In this section, we are going to load the same .sav7bdat file into a Pandas dataframe but by using Pandas read_sas method, instead. This have the advantage that we can load the SAS file from an URL.

Before we continue, we need to import Pandas:

import pandas as pd

Now, when we have done that, we can read the .sas7bdat file into a Pandas dataframe using the read_sas method. In the read SAS example here, we are importing the same data file as in the previous example.

Here, we print the 5 last rows of the dataframe using Pandas tail method.

url = 'http://www.principlesofeconometrics.com/sas/airline.sas7bdat'

df = pd.read_sas(url)
df.tail()

How to Read a SAS File and Specific Columns

Note, that read_sas7bdat (Pyreadstat) have the argument “usecols”. By using this argument, we can also select which columns we want to load from the SPSS file to the dataframe:

cols = ['YEAR', 'Y', 'W']
df, meta = pyreadstat.read_sas7bdat('airline.sas7bdat', usecols=cols)
df.head()

How to Save a SAS file to CSV

In this section of the Pandas SAS tutorial we are going to export the .sas7bdat file to a .csv file. This is easy done, we just have to use the to_csv method from the dataframe object we created earlier:

df.to_csv('data_from_sas.csv', index=False)

Remember to put the right path, as second argument, when using to_csv to save a .sas7bdat file as CSV.

Summary: Read SAS Files using Python

Now we have learned how to read and write SAS files in Python. It was quite simple and both methods are, in fact, using the same Python packages.

The post How to Read SAS Files in Python with Pandas appeared first on Erik Marsja.

The post PyDev of the Week: Elana Hashman appeared first on The Mouse Vs. The Python.

Django 3.0 beta 1 is now available. It represents the second stage in the 3.0 release cycle and is an opportunity for you to try out the changes coming in Django 3.0.

Django 3.0 has a raft of new features which you can read about in the in-development 3.0 release notes.

Only bugs in new features and regressions from earlier versions of Django will be fixed between now and 3.0 final (also, translations will be updated following the "string freeze" when the release candidate is issued). The current release schedule calls for a release candidate in a month from now with the final release to follow about two weeks after that around December 2. Early and often testing from the community will help minimize the number of bugs in the release. Updates on the release schedule schedule are available on the django-developers mailing list.

As with all beta and beta packages, this is not for production use. But if you'd like to take some of the new features for a spin, or to help find and fix bugs (which should be reported to the issue tracker), you can grab a copy of the beta package from our downloads page or on PyPI.

The PGP key ID used for this release is Mariusz Felisiak: 2EF56372BA48CD1B.

I always wanted to have a tool to monitor the network connections from my laptop/desktop. I wanted to have alerts for random processes making network connections, and a way to block those (if I want to).

Such a tool can provide peace of mind in a few cases. A reverse shell is one the big one, just in case if I manage to open any random malware (read downloads) on my regular Linux system, I want to be notified about the connections it will make. The same goes for trying out any new application. I prefer to use Qubes OS based VMs testing random binaries and applications, and it is also my daily driver. But, the search for a proper tool continued for some time.

Introducing unoon

Unoon is a desktop tool that I started writing for monitoring network connections for my system. It has two parts, the backend is written in Go and that monitor and adds details to a local Redis instance (this should be password protected).

I started writing this backend in Rust, but then I had to rewrite it in Go as I wanted to reuse parts of my code from another project so that I can track all DNS queries from the system. This helps to make sense of the data; otherwise, we will see some random IP numbers in the UI.

The frontend is written using PyQt5. Around 14 years ago, I released my first ever released tool using PyQt, and it is still my favorite library to create a desktop application.

Using the development version of unoon

The README has the build steps. You have to start the backend as a daemon, the easiest option is to run it inside of a tmux shell. At first, it will show all the currently running processes in the first “Current processes” tab. If you add any executable (via the absolute path) in the Edit->whitelists dialog and then save (and then restart the UI app), those will turn up the whitelisted processes.

For any new process making network calls, you will get an alert dialog. In the future, we will have the option to block hosts/ips via this alert dialog.

The history tabs will show all alerts history in the runtime. Again, we will have to save this information in a local database, so that we can have better statistics shown to the users.

You can move between different tabs/tables via Alt+1 or Alt+2 and Alt+3 key combinations.

I will add more options to create better-whitelisted processes. There is also ongoing work to mark any normal process as a whitelisted one from the UI (by right-clicking).

Last week, Micah and I managed to spend some late-night hotel room hacking on this tool.

How can you help?

You can start by testing the code base, and provide suggestions on how to improve the tool. Help in UX (major concern) and patches are always welcome.

A small funny story

A few weeks back, on a Sunday late night, I was demoing the very initial version of the tool to Saptak. While we were talking about the tool, suddenly, an entry popped up in the UI /usr/bin/ssh, to a random host. A little bit of search showed that the IP belongs to an EC2 instance. For the next 40 minutes, we both were trying to debug to find out what happened and if the system was already compromised or not. Luckily I was talking about something else before, and to demo something (we totally forgot that topic), I was running Wireshark on the system. From there, we figured that the IP belongs to github.com. It took some more time to figure out that one of my VS Code extension was updating the git, and was using ssh. This is when I understood that I need to show the real domain names on the UI than random IP addresses.

The newest version of Python is released today! Python 3.8 has been available in beta versions since the summer, but on October 14th, 2019 the first official version is ready. Now, we can all start playing with the new features and benefit from the latest improvements.

What does Python 3.8 bring to the table? The documentation gives a good overview of the new features. However, this article will go more in depth on some of the biggest changes, and show you how you can take advantage of Python 3.8.

In this article, you’ll learn about:

• Using assignment expressions to simplify some code constructs
• Enforcing positional-only arguments in your own functions
• Specifying more precise type hints
• Using f-strings for simpler debugging

With a few exceptions, Python 3.8 contains many small improvements over the earlier versions. Towards the end of the article, you’ll see many of these less attention-grabbing changes, as well as a discussion about some of the optimizations that make Python 3.8 faster than its predecessors. Finally, you’ll get some advice about upgrading to the new version.

The Walrus in the Room: Assignment Expressions

The biggest change in Python 3.8 is the introduction of assignment expressions. They are written using a new notation (:=). This operator is often called the walrus operator as it resembles the eyes and tusks of a walrus on its side.

Assignment expressions allow you to assign and return a value in the same expression. For example, if you want to assign to a variable and print its value, then you typically do something like this:

>>> walrus=False>>> print(walrus)False

In Python 3.8, you’re allowed to combine these two statements into one, using the walrus operator:

>>> print(walrus:=True)True

The assignment expression allows you to assign True to walrus, and immediately print the value. But keep in mind that the walrus operator does not do anything that isn’t possible without it. It only makes certain constructs more convenient, and can sometimes communicate the intent of your code more clearly.

One pattern that shows some of the strengths of the walrus operator is while loops where you need to initialize and update a variable. For example, the following code asks the user for input until they type quit:

inputs=list()current=input("Write something: ")whilecurrent!="quit":inputs.append(current)current=input("Write something: ")

This code is less than ideal. You’re repeating the input() statement, and somehow you need to add current to the list before asking the user for it. A better solution is to set up an infinite while loop, and use break to stop the loop:

inputs=list()whileTrue:current=input("Write something: ")ifcurrent=="quit":breakinputs.append(current)

This code is equivalent to the one above, but avoids the repetition and somehow keeps the lines in a more logical order. If you use an assignment expression, you can simplify this loop further:

inputs=list()while(current:=input("Write something: "))!="quit":inputs.append(current)

This moves the test back to the while line, where it should be. However, there are now several things happening at that line, so it takes a bit more effort to read it properly. Use your best judgement about when the walrus operator helps make your code more readable.

PEP 572 describes all the details of assignment expressions, including some of the rationale for introducing them into the language, as well as several examples of how the walrus operator can be used.

Positional-Only Arguments

The built-in function float() can be used for converting text strings and numbers to float objects. Consider the following example:

>>> float("3.8")3.8>>> help(float)class float(object) |  float(x=0, /) |   |  Convert a string or number to a floating point number, if possible.[...]

Look closely at the signature of float(). Notice the slash (/) after the parameter. What does it mean?

It turns out that while the one parameter of float() is called x, you’re not allowed to use its name:

>>> float(x="3.8")Traceback (most recent call last):
  File "<stdin>", line 1, in <module>TypeError: float() takes no keyword arguments

When using float() you’re only allowed to specify arguments by position, not by keyword. Before Python 3.8, such positional-only arguments were only possible for built-in functions. There was no easy way to specify that arguments should be positional-only in your own functions:

>>> defincr(x):... returnx+1... >>> incr(3.8)4.8>>> incr(x=3.8)4.8

It’s possible to simulate positional-only arguments using *args, but this is less flexible, less readable, and forces you to implement your own argument parsing. In Python 3.8, you can use / to denote that all arguments before it must be specified by position. You can rewrite incr() to only accept positional arguments:

>>> defincr(x,/):... returnx+1... >>> incr(3.8)4.8>>> incr(x=3.8)Traceback (most recent call last):
  File "<stdin>", line 1, in <module>TypeError: incr() got some positional-only arguments passed as           keyword arguments: 'x'

By adding / after x, you specify that x is a positional-only argument. You can combine regular arguments with positional-only ones by placing the regular arguments after the slash:

>>> defgreet(name,/,greeting="Hello"):... returnf"{greeting}, {name}"... >>> greet("Łukasz")'Hello, Łukasz'>>> greet("Łukasz",greeting="Awesome job")'Awesome job, Łukasz'>>> greet(name="Łukasz",greeting="Awesome job")Traceback (most recent call last):
  File "<stdin>", line 1, in <module>TypeError: greet() got some positional-only arguments passed as           keyword arguments: 'name'

In greet(), the slash is placed between name and greeting. This means that name is a positional-only argument, while greeting is a regular argument that can be passed either by position or by keyword.

At first glance, positional-only arguments can seem a bit limiting and contrary to Python’s mantra about the importance of readability. You will probably find that there are not a lot of occasions where positional-only arguments improve your code.

However, in the right circumstances, positional-only arguments can give you some flexibility when you’re designing functions. First, positional-only arguments make sense when you have arguments that have a natural order but are hard to give good, descriptive names to.

Another possible benefit of using positional-only arguments is that you can more easily refactor your functions. In particular, you can change the name of your parameters without worrying that other code depends on those names.

Positional-only arguments nicely complement keyword-only arguments. In any version of Python 3, you can specify keyword-only arguments using the star (*). Any argument after* must be specified using a keyword:

>>> defto_fahrenheit(*,celsius):... return32+celsius*9/5... >>> to_fahrenheit(40)Traceback (most recent call last):
  File "<stdin>", line 1, in <module>TypeError: to_fahrenheit() takes 0 positional arguments but 1 was given>>> to_fahrenheit(celsius=40)104.0

celsius is a keyword-only argument, so Python raises an error if you try to specify it based on position, without the keyword.

You can combine positional-only, regular, and keyword-only arguments, by specifying them in this order separated by / and *. In the following example, text is a positional-only argument, border is a regular argument with a default value, and width is a keyword-only argument with a default value:

>>> defheadline(text,/,border="♦",*,width=50):... returnf" {text}".center(width,border)... 

Since text is positional-only, you can’t use the keyword text:

>>> headline("Positional-only Arguments")'♦♦♦♦♦♦♦♦♦♦♦ Positional-only Arguments ♦♦♦♦♦♦♦♦♦♦♦♦'>>> headline(text="This doesn't work!")Traceback (most recent call last):
  File "<stdin>", line 1, in <module>TypeError: headline() got some positional-only arguments passed as           keyword arguments: 'text'

border, on the other hand, can be specified both with and without the keyword:

>>> headline("Python 3.8","=")'=================== Python 3.8 ==================='>>> headline("Real Python",border=":")':::::::::::::::::: Real Python :::::::::::::::::::'

Finally, width must be specified using the keyword:

>>> headline("Python","🐍",width=38)'🐍🐍🐍🐍🐍🐍🐍🐍🐍🐍🐍🐍🐍🐍🐍 Python 🐍🐍🐍🐍🐍🐍🐍🐍🐍🐍🐍🐍🐍🐍🐍'>>> headline("Python","🐍",38)Traceback (most recent call last):
  File "<stdin>", line 1, in <module>TypeError: headline() takes from 1 to 2 positional arguments           but 3 were given

You can read more about positional-only arguments in PEP 570.

More Precise Types

Python’s typing system is quite mature at this point. However, in Python 3.8, some new features have been added to typing to allow more precise typing:

• Literal types
• Typed dictionaries
• Final objects
• Protocols

Python supports optional type hints, typically as annotations on your code:

defdouble(number:float)->float:return2*number

In this example, you say that number should be a float and the double() function should return a float, as well. However, Python treats these annotations as hints. They are not enforced at runtime:

>>> double(3.14)6.28>>> double("I'm not a float")"I'm not a floatI'm not a float"

double() happily accepts "I'm not a float" as an argument, even though that’s not a float. There are libraries that can use types at runtime, but that is not the main use case for Python’s type system.

Instead, type hints allow static type checkers to do type checking of your Python code, without actually running your scripts. This is reminiscent of compilers catching type errors in other languages like Java, Rust, and Crystal. Additionally, type hints act as documentation of your code, making it easier to read, as well as improving auto-complete in your IDE.

$ python -m pip install mypy

You can find more information about type hints in Python in the original PEP 484, as well as in Python Type Checking (Guide).

There are four new PEPs about type checking that have been accepted and included in Python 3.8. You’ll see short examples from each of these.

PEP 586 introduce the Literal type. Literal is a bit special in that it represents one or several specific values. One use case of Literal is to be able to precisely add types, when string arguments are used to describe specific behavior. Consider the following example:

# draw_line.pydefdraw_line(direction:str)->None:ifdirection=="horizontal":...# Draw horizontal lineelifdirection=="vertical":...# Draw vertical lineelse:raiseValueError(f"invalid direction {direction!r}")draw_line("up")

The program will pass the static type checker, even though "up" is an invalid direction. The type checker only checks that "up" is a string. In this case, it would be more precise to say that direction must be either the literal string "horizontal" or the literal string "vertical". Using Literal, you can do exactly that:

# draw_line.pyfromtypingimportLiteraldefdraw_line(direction:Literal["horizontal","vertical"])->None:ifdirection=="horizontal":...# Draw horizontal lineelifdirection=="vertical":...# Draw vertical lineelse:raiseValueError(f"invalid direction {direction!r}")draw_line("up")

By exposing the allowed values of direction to the type checker, you can now be warned about the error:

$ mypy draw_line.py 
draw_line.py:15: error:    Argument 1 to "draw_line" has incompatible type "Literal['up']";    expected "Union[Literal['horizontal'], Literal['vertical']]"Found 1 error in 1 file (checked 1 source file)

The basic syntax is Literal[<literal>]. For instance, Literal[38] represents the literal value 38. You can express one of several literal values using Union:

Union[Literal["horizontal"],Literal["vertical"]]

Since this is a fairly common use case, you can (and probably should) use the simpler notation Literal["horizontal", "vertical"] instead. You already used the latter when adding types to draw_line(). If you look carefully at the output from Mypy above, you can see that it translated the simpler notation to the Union notation internally.

There are cases where the type of the return value of a function depends on the input arguments. One example is open() which may return a text string or a byte array depending on the value of mode. This can be handled through overloading.

The following example shows the skeleton of a calculator that can return the answer either as regular numbers (38), or as roman numerals (XXXVIII):

# calculator.pyfromtypingimportUnionARABIC_TO_ROMAN=[(1000,"M"),(900,"CM"),(500,"D"),(400,"CD"),(100,"C"),(90,"XC"),(50,"L"),(40,"XL"),(10,"X"),(9,"IX"),(5,"V"),(4,"IV"),(1,"I")]def_convert_to_roman_numeral(number:int)->str:"""Convert number to a roman numeral string"""result=list()forarabic,romaninARABIC_TO_ROMAN:count,number=divmod(number,arabic)result.append(roman*count)return"".join(result)defadd(num_1:int,num_2:int,to_roman:bool=True)->Union[str,int]:"""Add two numbers"""result=num_1+num_2ifto_roman:return_convert_to_roman_numeral(result)else:returnresult

The code has the correct type hints: the result of add() will be either str or int. However, often this code will be called with a literal True or False as the value of to_roman in which case you would like the type checker to infer exactly whether str or int is returned. This can be done using Literal together with @overload:

# calculator.pyfromtypingimportLiteral,overload,UnionARABIC_TO_ROMAN=[(1000,"M"),(900,"CM"),(500,"D"),(400,"CD"),(100,"C"),(90,"XC"),(50,"L"),(40,"XL"),(10,"X"),(9,"IX"),(5,"V"),(4,"IV"),(1,"I")]def_convert_to_roman_numeral(number:int)->str:"""Convert number to a roman numeral string"""result=list()forarabic,romaninARABIC_TO_ROMAN:count,number=divmod(number,arabic)result.append(roman*count)return"".join(result)@overloaddefadd(num_1:int,num_2:int,to_roman:Literal[True])->str:...@overloaddefadd(num_1:int,num_2:int,to_roman:Literal[False])->int:...defadd(num_1:int,num_2:int,to_roman:bool=True)->Union[str,int]:"""Add two numbers"""result=num_1+num_2ifto_roman:return_convert_to_roman_numeral(result)else:returnresult

The added @overload signatures will help your type checker infer str or int depending on the literal values of to_roman. Note that the ellipses (...) are a literal part of the code. They stand in for the function body in the overloaded signatures.

As a complement to Literal, PEP 591 introduces Final. This qualifier specifies that a variable or attribute should not be reassigned, redefined, or overridden. The following is a typing error:

fromtypingimportFinalID:Final=1...ID+=1

Mypy will highlight the line ID += 1, and note that you Cannot assign to final name "ID". This gives you a way to ensure that constants in your code never change their value.

Additionally, there is also a @final decorator that can be applied to classes and methods. Classes decorated with @final can’t be subclassed, while @final methods can’t be overridden by subclasses:

fromtypingimportfinal@finalclassBase:...classSub(Base):...

Mypy will flag this example with the error message Cannot inherit from final class "Base". To learn more about Final and @final, see PEP 591.

The third PEP allowing for more specific type hints is PEP 589, which introduces TypedDict. This can be used to specify types for keys and values in a dictionary using a notation that is similar to the typed NamedTuple.

Traditionally, dictionaries have been annotated using Dict. The issue is that this only allowed one type for the keys and one type for the values, often leading to annotations like Dict[str, Any]. As an example, consider a dictionary that registers information about Python versions:

py38={"version":"3.8","release_year":2019}

The value corresponding to version is a string, while release_year is an integer. This can’t be precisely represented using Dict. With the new TypedDict, you can do the following:

fromtypingimportTypedDictclassPythonVersion(TypedDict):version:strrelease_year:intpy38=PythonVersion(version="3.8",release_year=2019)

The type checker will then be able to infer that py38["version"] has type str, while py38["release_year"] is an int. At runtime, a TypedDict is a regular dict, and type hints are ignored as usual. You can also use TypedDict purely as an annotation:

py38:PythonVersion={"version":"3.8","release_year":2019}

Mypy will let you know if any of your values has the wrong type, or if you use a key that has not been declared. See PEP 589 for more examples.

Mypy has supported Protocols for a while already. However, the official acceptance only happened in May 2019.

Protocols are a way of formalizing Python’s support for duck typing:

When I see a bird that walks like a duck and swims like a duck and quacks like a duck, I call that bird a duck. (Source)

Duck typing allows you to, for example, read .name on any object that has a .name attribute, without really caring about the type of the object. It may seem counter-intuitive for the typing system to support this. Through structural subtyping, it’s still possible to make sense of duck typing.

You can for instance define a protocol called Named that can identify all objects with a .name attribute:

fromtypingimportProtocolclassNamed(Protocol):name:strdefgreet(obj:Named)->None:print(f"Hi {obj.name}")

Here, greet() takes any object, as long as it defines a .name attribute. See PEP 544 and the Mypy documentation for more information about protocols.

Simpler Debugging With f-Strings

f-strings were introduced in Python 3.6, and have become very popular. They might be the most common reason for Python libraries only being supported on version 3.6 and later. An f-string is a formatted string literal. You can recognize it by the leading f:

>>> style="formatted">>> f"This is a {style} string"'This is a formatted string'

When you use f-strings, you can enclose variables and even expressions inside curly braces. They will then be evaluated at runtime and included in the string. You can have several expressions in one f-string:

>>> importmath>>> r=3.6>>> f"A circle with radius {r} has area {math.pi * r * r:.2f}"'A circle with radius 3.6 has area 40.72'

In the last expression, {math.pi * r * r:.2f}, you also use a format specifier. Format specifiers are separated from the expressions with a colon.

.2f means that the area is formatted as a floating point number with 2 decimals. The format specifiers are the same as for .format(). See the official documentation for a full list of allowed format specifiers.

In Python 3.8, you can use assignment expressions inside f-strings. Just make sure to surround the assignment expression with parentheses:

>>> importmath>>> r=3.8>>> f"Diameter {(diam := 2 * r)} gives circumference {math.pi * diam:.2f}"'Diameter 7.6 gives circumference 23.88'

However, the real f-news in Python 3.8 is the new debugging specifier. You can now add = at the end of an expression, and it will print both the expression and its value:

>>> python=3.8>>> f"{python=}"'python=3.8'

This is a short-hand, that typically will be most useful when working interactively or adding print statements to debug your script. In earlier versions of Python, you needed to spell out the variable or expression twice to get the same information:

>>> python=3.7>>> f"python={python}"'python=3.7'

You can add spaces around =, and use format specifiers as usual:

>>> name="Eric">>> f"{name = }""name = 'Eric'">>> f"{name = :>10}"'name =       Eric'

The >10 format specifier says that name should be right-aligned within a 10 character string. = works for more complex expressions as well:

>>> f"{name.upper()[::-1] = }""name.upper()[::-1] = 'CIRE'"

For more information about f-strings, see Python 3’s f-Strings: An Improved String Formatting Syntax (Guide).

The Python Steering Council

Technically, Python’s governance is not a language feature. However, Python 3.8 is the first version of Python not developed under the benevolent dictatorship of Guido van Rossum. The Python language is now governed by a steering council consisting of five core developers:

• Barry Warsaw
• Brett Cannon
• Carol Willing
• Guido van Rossum
• Nick Coghlan

The road to the new governance model for Python was an interesting study in self-organization. Guido van Rossum created Python in the early 1990s, and has been affectionally dubbed Python’s Benevolent Dictator for Life (BDFL). Through the years, more and more decisions about the Python language were made through Python Enhancement Proposals (PEPs). Still, Guido officially had the last word on any new language feature.

After a long and drawn out discussion about assignment expressions, Guido announced in July 2018 that he was retiring from his role as BDFL (for real this time). He purposefully did not name a successor. Instead, he asked the team of core developers to figure out how Python should be governed going forward.

Luckily, the PEP process was already well established, so it was natural to use PEPs to discuss and decide on a new governance model. Through the fall of 2018, several models were proposed, including electing a new BDFL (renamed the Gracious Umpire Influencing Decisions Officer: the GUIDO), or moving to a community model based on consensus and voting, without centralized leadership. In December 2018, the steering council model was chosen after a vote among the core developers.

The steering council consists of five members of the Python community, as listed above. There will be an election for a new steering council after every major release of Python. In other words, there will be an election following the release of Python 3.8.

Although it’s an open election, it’s expected that most, if not all, of the inaugural steering council will be reelected. The steering council has broad powers to make decisions about the Python language, but should strive to exercise those powers as little as possible.

You can read all about the new governance model in PEP 13, while the process of deciding on the new model is described in PEP 8000. For more information, see the PyCon 2019 Keynote, and listen to Brett Cannon on Talk Python To Me and on The Changelog podcast. You can follow updates from the steering council on GitHub.

Other Pretty Cool Features

So far, you’ve seen the headline news regarding what’s new in Python 3.8. However, there are many other changes that are also pretty cool. In this section, you’ll get a quick look at some of them.

importlib.metadata

There is one new module available in the standard library in Python 3.8: importlib.metadata. Through this module, you can access information about installed packages in your Python installation. Together with its companion module, importlib.resources, importlib.metadata improves on the functionality of the older pkg_resources.

As an example, you can get some information about pip:

>>> fromimportlibimportmetadata>>> metadata.version("pip")'19.2.3'>>> pip_metadata=metadata.metadata("pip")>>> list(pip_metadata)['Metadata-Version', 'Name', 'Version', 'Summary', 'Home-page', 'Author', 'Author-email', 'License', 'Keywords', 'Platform', 'Classifier',  'Classifier', 'Classifier', 'Classifier', 'Classifier', 'Classifier',  'Classifier', 'Classifier', 'Classifier', 'Classifier', 'Classifier',  'Classifier', 'Classifier', 'Requires-Python']>>> pip_metadata["Home-page"]'https://pip.pypa.io/'>>> pip_metadata["Requires-Python"]'>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*'>>> len(metadata.files("pip"))668

The currently installed version of pip is 19.2.3. metadata() gives access to most of the information that you can see on PyPI. You can for instance see that this version of pip requires either Python 2.7, or Python 3.5 or higher. With files(), you get a listing of all files that make up the pip package. In this case, there are almost 700 files.

files() returns a list of Path objects. These give you a convenient way of looking into the source code of a package, using read_text(). The following example prints out __init__.py from the realpython-reader package:

>>> [pforpinmetadata.files("realpython-reader")ifp.suffix==".py"][PackagePath('reader/__init__.py'), PackagePath('reader/__main__.py'), PackagePath('reader/feed.py'), PackagePath('reader/viewer.py')]>>> init_path=_[0]# Underscore access last returned value in the REPL>>> print(init_path.read_text())"""Real Python feed readerImport the `feed` module to work with the Real Python feed:>>> from reader import feed>>> feed.get_titles()    ['Logging in Python', 'The Best Python Books', ...]See https://github.com/realpython/reader/ for more information"""# Version of realpython-reader package__version__ = "1.0.0"...

You can also access package dependencies:

>>> metadata.requires("realpython-reader")['feedparser', 'html2text', 'importlib-resources', 'typing']

requires() lists the dependencies of a package. You can see that realpython-reader for instance uses feedparser in the background to read and parse a feed of articles.

There is a backport of importlib.metadataavailable on PyPI that works on earlier versions of Python. You can install it using pip:

$ python -m pip install importlib-metadata

You can fall back on using the PyPI backport in your code as follows:

try:fromimportlibimportmetadataexceptImportError:importimportlib_metadataasmetadata...

See the documentation for more information about importlib.metadata

New and Improved math and statistics Functions

Python 3.8 brings many improvements to existing standard library packages and modules. math in the standard library has a few new functions. math.prod() works similarly to the built-in sum(), but for multiplicative products:

>>> importmath>>> math.prod((2,8,7,7))784>>> 2*8*7*7784

The two statements are equivalent. prod() will be easier to use when you already have the factors stored in an iterable.

Another new function is math.isqrt(). You can use isqrt() to find the integer part of square roots:

>>> importmath>>> math.isqrt(9)3>>> math.sqrt(9)3.0>>> math.isqrt(15)3>>> math.sqrt(15)3.872983346207417

The square root of 9 is 3. You can see that isqrt() returns an integer result, while math.sqrt() always returns a float. The square root of 15 is almost 3.9. Note that isqrt()truncates the answer down to the next integer, in this case 3.

Finally, you can now more easily work with n-dimensional points and vectors in the standard library. You can find the distance between two points with math.dist(), and the length of a vector with math.hypot():

>>> importmath>>> point_1=(16,25,20)>>> point_2=(8,15,14)>>> math.dist(point_1,point_2)14.142135623730951>>> math.hypot(*point_1)35.79106033634656>>> math.hypot(*point_2)22.02271554554524

This makes it easier to work with points and vectors using the standard library. However, if you will be doing many calculations on points or vectors, you should check out NumPy.

The statistics module also has several new functions:

• statistics.fmean() calculates the mean of float numbers.
• statistics.geometric_mean() calculates the geometric mean of float numbers.
• statistics.multimode() finds the most frequently occurring values in a sequence.
• statistics.quantiles() calculates cut points for dividing data into n continuous intervals with equal probability.

The following example shows the functions in use:

>>> importstatistics>>> data=[9,3,2,1,1,2,7,9]>>> statistics.fmean(data)4.25>>> statistics.geometric_mean(data)3.013668912157617>>> statistics.multimode(data)[9, 2, 1]>>> statistics.quantiles(data,n=4)[1.25, 2.5, 8.5]

In Python 3.8, there is a new statistics.NormalDist class that makes it more convenient to work with the Gaussian normal distribution.

To see an example of using NormalDist, you can try to compare the speed of the new statistics.fmean() and the traditional statistics.mean():

>>> importrandom>>> importstatistics>>> fromtimeitimporttimeit>>> # Create 10,000 random numbers>>> data=[random.random()for_inrange(10_000)]>>> # Measure the time it takes to run mean() and fmean()>>> t_mean=[timeit("statistics.mean(data)",number=100,globals=globals())... for_inrange(30)]>>> t_fmean=[timeit("statistics.fmean(data)",number=100,globals=globals())... for_inrange(30)]>>> # Create NormalDist objects based on the sampled timings>>> n_mean=statistics.NormalDist.from_samples(t_mean)>>> n_fmean=statistics.NormalDist.from_samples(t_fmean)>>> # Look at sample mean and standard deviation>>> n_mean.mean,n_mean.stdev(0.825690647733245, 0.07788573997674526)>>> n_fmean.mean,n_fmean.stdev(0.010488564966666065, 0.0008572332785645231)>>> # Calculate the lower 1 percentile of mean>>> n_mean.quantiles(n=100)[0]0.6445013221202459

In this example, you use timeit to measure the execution time of mean() and fmean(). To get reliable results, you let timeit execute each function 100 times, and collect 30 such time samples for each function. Based on these samples, you create two NormalDist objects. Note that if you run the code yourself, it might take up to a minute to collect the different time samples.

NormalDist has many convenient attributes and methods. See the documentation for a complete list. Inspecting .mean and .stdev, you see that the old statistics.mean() runs in 0.826 ± 0.078 seconds, while the new statistics.fmean() spends 0.0105 ± 0.0009 seconds. In other words, fmean() is about 80 times faster for these data.

If you need more advanced statistics in Python than the standard library offers, check out statsmodels and scipy.stats.

Warnings About Dangerous Syntax

Python has a SyntaxWarning which can warn about dubious syntax that is typically not a SyntaxError. Python 3.8 adds a few new ones that can help you during coding and debugging.

The difference between is and == can be confusing. The latter checks for equal values, while is is True only when objects are the same. Python 3.8 will try to warn you about cases when you should use == instead of is:

>>> # Python 3.7>>> version="3.7">>> versionis"3.7"False>>> # Python 3.8>>> version="3.8">>> versionis"3.8"<stdin>:1: SyntaxWarning: "is" with a literal. Did you mean "=="?False>>> version=="3.8"True

It’s easy to miss a comma when you’re writing out a long list, especially when formatting it vertically. Forgetting a comma in a list of tuples will give a confusing error message about tuples not being callable. Python 3.8 additionally emits a warning that points toward the real issue:

>>> [... (1,3)... (2,4)... ]<stdin>:2: SyntaxWarning: 'tuple' object is not callable; perhaps           you missed a comma?Traceback (most recent call last):
  File "<stdin>", line 2, in <module>TypeError: 'tuple' object is not callable

The warning correctly identifies the missing comma as the real culprit.

Optimizations

There are several optimizations made for Python 3.8. Some that make code run faster. Others reduce the memory footprint. For example, looking up fields in a namedtuple is significantly faster in Python 3.8 compared with Python 3.7:

>>> importcollections>>> fromtimeitimporttimeit>>> Person=collections.namedtuple("Person","name twitter")>>> raymond=Person("Raymond","@raymondh")>>> # Python 3.7>>> timeit("raymond.twitter",globals=globals())0.05876131607996285>>> # Python 3.8>>> timeit("raymond.twitter",globals=globals())0.0377705999400132

You can see that looking up .twitter on the namedtuple is 30-40% faster in Python 3.8. Lists save some space when they are initialized from iterables with a known length. This can save memory:

>>> importsys>>> # Python 3.7>>> sys.getsizeof(list(range(20191014)))181719232>>> # Python 3.8>>> sys.getsizeof(list(range(20191014)))161528168

In this case, the list uses about 11% less memory in Python 3.8 compared with Python 3.7.

Other optimizations include better performance in subprocess, faster file copying with shutil, improved default performance in pickle, and faster operator.itemgetter operations. See the official documentation for a complete list of optimizations.

So, Should You Upgrade to Python 3.8?

Let’s start with the simple answer. If you want to try out any of the new features you have seen here, then you do need to be able to use Python 3.8. Tools like pyenv and Anaconda make it easy to have several versions of Python installed side by side. Alternatively, you can run the official Python 3.8 Docker container. There is no downside to trying out Python 3.8 for yourself.

Now, for the more complicated questions. Should you upgrade your production environment to Python 3.8? Should you make your own project dependent on Python 3.8 to take advantage of the new features?

You should have very few issues running Python 3.7 code in Python 3.8. Upgrading your environment to run Python 3.8 is therefore quite safe, and you would be able to take advantage of the optimizations made in the new version. Different beta-versions of Python 3.8 have already been available for months, so hopefully most bugs are already squashed. However, if you want to be conservative, you might hold out until the first maintenance release (Python 3.8.1) is available.

Once you’ve upgraded your environment, you can start to experiment with features that are only in Python 3.8, such as assignment expressions and positional-only arguments. However, you should be conscious about whether other people depend on your code, as this will force them to upgrade their environment as well. Popular libraries will probably mostly support at least Python 3.6 for quite a while longer.

See Porting to Python 3.8 for more information about preparing your code for Python 3.8.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Introduction

In this post, we will take a look at a few cases, where KMC algorithm does not perform well or may produce unintuitive results. In particular, we will look at the following scenarios:

1. Our guess on the number of (real) clusters is off.
2. Feature space is highly dimensional.
3. The clusters come in strange or irregular shapes.

All of these conditions can lead to problems with K-Means, so let’s have a look.

Wrong number of clusters

To make it easier, let’s define a helper function compare, which will create and solve the clustering problem for us and then compare the results.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
fromsklearnimportdatasetsfromsklearn.clusterimportKMeansfromsklearn.datasetsimportmake_blobs,make_circles,make_moonsfrommpl_toolkits.mplot3dimportAxes3Dimportnumpyasnpimportpandasaspdimportitertoolsdefcompare(N_features,C_centers,K_clusters,dims=[0,1],*args):data,targets=make_blobs(n_samples=n_samplesif'n_samples'inargselse400,n_features=N_features,centers=C_centers,cluster_std=cluster_stdif'cluster_std'inargselse0.5,shuffle=True,random_state=random_stateif'random_state'inargselse0)FEATS=['x'+str(x)forxinrange(N_features)]X=pd.DataFrame(data,columns=FEATS)X['cluster']= \
		KMeans(n_clusters=K_clusters,random_state=0).fit_predict(X)fig,axs=plt.subplots(1,2,figsize=(12,4))axs[0].scatter(data[:,dims[0]],data[:,dims[1]],c='white',marker='o',edgecolor='black',s=20)axs[0].set_xlabel('x{} [a.u.]'.format(dims[0]))axs[0].set_ylabel('x{} [a.u.]'.format(dims[1]))axs[0].set_title('Original dataset')axs[1].set_xlabel('x{} [a.u.]'.format(dims[0]))axs[1].set_ylabel('x{} [a.u.]'.format(dims[1]))axs[1].set_title('Applying clustering')colors=itertools.cycle(['r','g','b','m','c','y'])forkinrange(K_clusters):x=X[X['cluster']==k][FEATS].to_numpy()axs[1].scatter(x[:,dims[0]],x[:,dims[1]],color=next(colors),edgecolor='k',alpha=0.5)plt.show()

Too few clusters

✕

Figure 1a.

Example of a 2-dimensional dataset with 4 centres, requesting 3 clusters (compare(2, 4, 3)).

Despite having distinct clusters in the data, we underestimated their number. As a consequence, some disjoint groups of data are forced to fit into one larger cluster.

Too many clusters

✕

Figure 1b.

Example of a 2-dimensional dataset with 2 centres, requesting 4 clusters (compare(2, 2, 4)).

In contrary to the last situation, trying to wrap the data into too many clusters creates artificial boundaries within real data clusters.

High(er) dimensional data

A dataset does not need to be that high in dimentionality before we begin to see problems. Although visualization and thus somewhat analysis of high dimensional data is already challenging (cursing now…), as KMC is often used to gain insight into the data, it does not help to be presented with ambiguities.

To explain the point, let’s generate a three-dimensional dataset with clearly distinct clusters.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
fig=plt.figure(figsize=(14,8))ax=fig.add_subplot(111,projection='3d')data,targets=make_blobs(n_samples=400,n_features=3,centers=3,cluster_std=0.5,shuffle=True,random_state=0)ax.scatter(data[:,0],data[:,1],zs=data[:,2],zdir='z',s=25,c='black',depthshade=True)ax.set_xlabel('x0 [a.u.]')ax.set_ylabel('x1 [a.u.]')ax.set_zlabel('x2 [a.u.]')ax.set_title('Original distribution.')plt.grid()plt.show()

✕

Figure 2.

Example of a 3-dimensional dataset with 3 centers.

Although there are infinitely many ways we can project this 3D dataset onto 2D, there are three primary orthogonal sub-spaces:

• x0 : x1
• x1 : x2
• x2 : x0

Looking at the x2 : x0 projection, the dataset looks like as if it only had two clusters. The lower-right “supercluster” is, in fact, two distinct groups and even if we guess K right (K = 3), it looks like an apparent error, despite the clusters are very localized.

✕

Figure 3a.

Projection on `x0 : x2` shows spurious result (compare(2, 2, 4, dims=[0, 2])).

To be sure, we have to look at the remaining projections to see the problem, literally, from different angles.

✕

Figure 3b.

Projection on `x1 : x2` resolves the ambiguity (compare(2, 2, 4, dims=[1, 2])).

✕

Figure 3c.

Projection on `x0 : x1` resolves the ambiguity (compare(2, 2, 4, dims=[0, 1])).

This makes more sense!

On the flip side, we had an incredible advantage. First, with three dimensions, we were able to plot the entire dataset. Secondly, the clusters that exist within the dataset were very distinct thus easy to spot. Finally, with three-dimensional dataset, we were facing only three standard 2D projections.

The case of N, N > 3 features, we would not be able to plot the whole dataset, and the number of 2D projections would scale quadratically with N:

not to mention that the dataset may have strangely shaped or non-localized clusters, which is our next challenge.

Irregular datasets

So far we mentioned problems that are on “our side”. We looked at a very “well-behaved” dataset and discussed issues on the analytics side. However, what about if the dataset does not fit our solution, or, our solution does not fit the problem? This is exactly the case, where data distribution comes in strange or irregular shapes.

Being presented with just this graph, we may be tricked into believing that there are only two clusters in the data. However, when plotting the remaining projections, we quickly learn that this is not true.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
fig,axs=plt.subplots(1,3,figsize=(14,4))# unequal varianceX,y=make_blobs(n_samples=1400,cluster_std=[1.0,2.5,0.2],random_state=2)y_pred=KMeans(n_clusters=3,random_state=2).fit_predict(X)colors=[['r','g','b'][c]forciny_pred]axs[0].scatter(X[:,0],X[:,1],color=colors,edgecolor='k',alpha=0.5)axs[0].set_title("Unequal Variance")# anisotropically distributed dataX,y=make_blobs(n_samples=1400,random_state=156)transformation=[[0.60834549,-0.63667341],[-0.40887718,0.85253229]]X=np.dot(X,transformation)y_pred=KMeans(n_clusters=3,random_state=0).fit_predict(X)colors=[['r','g','b'][c]forciny_pred]axs[1].scatter(X[:,0],X[:,1],color=colors,edgecolor='k',alpha=0.5)axs[1].set_title("Anisotropicly Distributed Blobs")# irregular shaped dataX,y=make_moons(n_samples=1400,shuffle=True,noise=0.1,random_state=120)y_pred=KMeans(n_clusters=2,random_state=0).fit_predict(X)colors=[['r','g','b'][c]forciny_pred]axs[2].scatter(X[:,0],X[:,1],color=colors,edgecolor='k',alpha=0.5)axs[2].set_title("Irregular Shaped Data")plt.show()

✕

Figure 4.

Misleading clustering results are shown on irregular datasets.

The left graph shows data whose distribution although, Gaussian, does not have equal standard deviation. The middle graph presents anisotropic data, meaning data that is elongated along a specific axis. Finally, the right graph shows data that is completely non-Gaussian, despite organized in clear clusters.

In either case, the irregularity makes KMC algorithm underperform. Since the algorithm treats every data point equally and completely independently from other points, the algorithm fails to spot any possible continuity or local variations within a cluster. What it does is simply taking the same metrics and applying it to every point. As a result, the KMC algorithm may produce strange or counter-intuitive clustering within the data even if we guess K correctly and the features N are not that many.

Conclusions

In this post, we have discussed three main reasons for the K-Means Clustering algorithm to give us wrong answers.

• First, as the number of clusters K needs to be decided a priori, there is a high chance that we will guess it wrongly.
• Secondly, clustering in higher dimensional space becomes cumbersome from the analytics point of view, in which case KMC will provide us with insights that may be misleading.
• Finally, for any irregularly shaped data, KMC is likely to artificial clusters that do not conform to common sense.

Knowing these three fallacies, KMC is still a useful tool, especially when inspecting of the data or constructing labels.

I’m using Spotify since 2013 as the main source of music, and back at that time the app automatically created a playlist for songs that I liked from artists’ radios. By innertion I’m still using the playlist to save songs that I like. As the playlist became a bit big and a bit old (6 years, huh), I’ve decided to try to analyze it.

Boring preparation

To get the data I used Spotify API and spotipy as a Python client. I’ve created an application in the Spotify Dashboard and gathered the credentials. Then I was able to initialize and authorize the client:

importspotipyimportspotipy.utilasutiltoken=util.prompt_for_user_token(user_id,'playlist-read-collaborative',client_id=client_id,client_secret=client_secret,redirect_uri='http://localhost:8000/')sp=spotipy.Spotify(auth=token)

Tracks metadata

As everything is inside just one playlist, it was easy to gather. The only problem was that user_playlist method in spotipy doesn’t support pagination and can only return the first 100 track, but it was easily solved by just going down to private and undocumented _get:

playlist=sp.user_playlist(user_id,playlist_id)tracks=playlist['tracks']['items']next_uri=playlist['tracks']['next']for_inrange(int(playlist['tracks']['total']/playlist['tracks']['limit'])):response=sp._get(next_uri)tracks+=response['items']next_uri=response['next']tracks_df=pd.DataFrame([(track['track']['id'],track['track']['artists'][0]['name'],track['track']['name'],parse_date(track['track']['album']['release_date'])iftrack['track']['album']['release_date']elseNone,parse_date(track['added_at']))fortrackinplaylist['tracks']['items']],columns=['id','artist','name','release_date','added_at'])

tracks_df.head(10)

The first naive idea of data to get was the list of the most appearing artists:

tracks_df \
    .groupby('artist') \
    .count()['id'] \
    .reset_index() \
    .sort_values('id',ascending=False) \
    .rename(columns={'id':'amount'}) \
    .head(10)

But as taste can change, I’ve decided to get top five artists from each year and check if I was adding them to the playlist in other years:

counted_year_df=tracks_df \
    .assign(year_added=tracks_df.added_at.dt.year) \
    .groupby(['artist','year_added']) \
    .count()['id'] \
    .reset_index() \
    .rename(columns={'id':'amount'}) \
    .sort_values('amount',ascending=False)in_top_5_year_artist=counted_year_df \
    .groupby('year_added') \
    .head(5) \
    .artist \
    .unique()counted_year_df \
    [counted_year_df.artist.isin(in_top_5_year_artist)] \
    .pivot('artist','year_added','amount') \
    .fillna(0) \
    .style.background_gradient()

As a bunch of artists was reappearing in different years, I decided to check if that correlates with new releases, so I’ve checked the last ten years:

counted_release_year_df=tracks_df \
    .assign(year_added=tracks_df.added_at.dt.year,year_released=tracks_df.release_date.dt.year) \
    .groupby(['year_released','year_added']) \
    .count()['id'] \
    .reset_index() \
    .rename(columns={'id':'amount'}) \
    .sort_values('amount',ascending=False)counted_release_year_df \
    [counted_release_year_df.year_released.isin(sorted(tracks_df.release_date.dt.year.unique())[-11:])] \
    .pivot('year_released','year_added','amount') \
    .fillna(0) \
    .style.background_gradient()

Audio features

Spotify API has an endpoint that provides features like danceability, energy, loudness and etc for tracks. So I gathered features for all tracks from the playlist:

features=[]forn,chunk_seriesintracks_df.groupby(np.arange(len(tracks_df))//50).id:features+=sp.audio_features([*map(str,chunk_series)])features_df=pd.DataFrame.from_dict(filter(None,features))tracks_with_features_df=tracks_df.merge(features_df,on=['id'],how='inner')

tracks_with_features_df.head()

After that I’ve checked changes in features over time, only instrumentalness had some visible difference:

sns.boxplot(x=tracks_with_features_df.added_at.dt.year,y=tracks_with_features_df.instrumentalness)

Then I had an idea to check seasonality and valence, and it kind of showed that in depressing months valence is a bit lower:

sns.boxplot(x=tracks_with_features_df.added_at.dt.month,y=tracks_with_features_df.valence)

To play a bit more with data, I decided to check that danceability and valence might correlate:

tracks_with_features_df.plot(kind='scatter',x='danceability',y='valence')

And to check that the data is meaningful, I checked instrumentalness vs speechiness, and those featues looked mutually exclusive as expected:

tracks_with_features_df.plot(kind='scatter',x='instrumentalness',y='speechiness')

Tracks difference and similarity

As I already had a bunch of features classifying tracks, it was hard not to make vectors out of them:

encode_fields=['danceability','energy','key','loudness','mode','speechiness','acousticness','instrumentalness','liveness','valence','tempo','duration_ms','time_signature',]defencode(row):returnnp.array([(row[k]-tracks_with_features_df[k].min())/(tracks_with_features_df[k].max()-tracks_with_features_df[k].min())forkinencode_fields])tracks_with_features_encoded_df=tracks_with_features_df.assign(encoded=tracks_with_features_df.apply(encode,axis=1))

Then I just calculated distance between every two tracks:

tracks_with_features_encoded_product_df=tracks_with_features_encoded_df \
    .assign(temp=0) \
    .merge(tracks_with_features_encoded_df.assign(temp=0),on='temp',how='left') \
    .drop(columns='temp')tracks_with_features_encoded_product_df=tracks_with_features_encoded_product_df[tracks_with_features_encoded_product_df.id_x!=tracks_with_features_encoded_product_df.id_y]tracks_with_features_encoded_product_df['merge_id']=tracks_with_features_encoded_product_df \
    .apply(lambdarow:''.join(sorted([row['id_x'],row['id_y']])),axis=1)tracks_with_features_encoded_product_df['distance']=tracks_with_features_encoded_product_df \
    .apply(lambdarow:np.linalg.norm(row['encoded_x']-row['encoded_y']),axis=1)

After that I was able to get most similar songs/songs with the minimal distance, and it selected kind of similar songs:

tracks_with_features_encoded_product_df \
    .sort_values('distance') \
    .drop_duplicates('merge_id') \
    [['artist_x','name_x','release_date_x','artist_y','name_y','release_date_y','distance']] \
    .head(10)