Anthony Briggs (who blogs at NeoLudic) got in touch via Twitter with a proof-of-concept for controlling PyGameZero games with a gamepad (such as the one you’ll use with your XBox or Playstation).

The source code is nice and simple, although it relies an two patches which have not yet made it into a release of PyGameZero (so mark this as coming soon). One patch adds joystick support (i.e. what you do with the gamepad) and the other makes it easy to flip sprites.

Coming soon to a PyGameZero/Mu release near you!

Web Scraping can be beneficial to individuals and companies. The intentions of this post is to host a set of examples on Web Scraping using Golang and goquery. I will be using github’s trending page https://github.com/trending throughout this post for the examples, especially because it best suits for applying various goquery methods. There are two […]

The post Web Scraping using Golang appeared first on The Tara Nights.

One of the programmers’ favorite jokes about regular expressions is: "There is a problem when working with strings? Excellent, in this situation I can use regular expressions." And now the developer has 2 problems... In this article you’ll be able to get acquainted with the basics of working with regular expressions and not become the same programmer from a joke that, using regular expressions, only complicated the task at hand.

Introduction

Have you ever wondered how Netflix suggests movies to you based on the movies you have already watched? Or how does an e-commerce websites display options such as "Frequently Bought Together"? They may look relatively simple options but behind the scenes, a complex statistical algorithm executes in order to predict these recommendations. Such systems are called Recommender Systems, Recommendation Systems, or Recommendation Engines. A Recommender System is one of the most famous applications of data science and machine learning.

A Recommender System employs a statistical algorithm that seeks to predict users' ratings for a particular entity, based on the similarity between the entities or similarity between the users that previously rated those entities. The intuition is that similar types of users are likely to have similar ratings for a set of entities.

Currently, almost all of the big companies have employed Recommender Systems in one way or the other. For instance, YouTube uses Recommender Systems for video suggestions, Amazon uses it for a product recommendation, Facebook uses Recommender Systems for recommending people to follow and pages to like.

In this article, we will see how we can build a simple recommender system in Python.

Types of Recommender Systems

There are two major approaches to build recommender systems: Content-Based Filtering and Collaborative Filtering:

Content-Based Filtering

In content-based filtering, the similarity between different products is calculated on the basis of the attributes of the products. For instance, in a content-based movie recommender system, the similarity between the movies is calculated on the basis of genres, the actors in the movie, the director of the movie, etc.

Collaborative Filtering

Collaborative filtering leverages the power of the crowd. The intuition behind collaborative filtering is that if a user A likes products X and Y, and if another user B likes product X, there is a fair bit of chance that he will like the product Y as well.

Take the example of a movie recommender system. Suppose a huge number of users have assigned the same ratings to movies X and Y. A new user comes who has assigned the same rating to movie X but hasn't watched movie Y yet. Collaborative filtering system will recommend him the movie Y.

Movie Recommender System Implementation in Python

In this section, we'll develop a very simple movie recommender system in Python that uses the correlation between the ratings assigned to different movies, in order to find the similarity between the movies.

The dataset that we are going to use for this problem is the MovieLens Dataset. To download the dataset, go the home page of the dataset and download the "ml-latest-small.zip" file, which contains a subset of the actual movie dataset and contains 100000 ratings for 9000 movies by 700 users.

Once you unzip the downloaded file, you will see "links.csv", "movies.csv", "ratings.csv" and "tags.csv" files, along with the "README" document. In this article, we are going to use the "movies.csv" and "ratings.csv" files.

For the scripts in this article, the unzipped "ml-latest-small" folder has been placed inside the "Datasets" folder in the "E" drive.

Data Visualization and Preprocessing

The first step in every data science problem is to visualize and preprocess the data. We will do the same, so let's first import the "ratings.csv" file and see what it contains. Execute the following script:

import numpy as np  
import pandas as pd

ratings_data = pd.read_csv("E:\Datasets\ml-latest-small\\ratings.csv")  
ratings_data.head()  

In the script above we use the read_csv() method of the Pandas library to read the "ratings.csv" file. Next, we call the head() method from the dataframe object returned by the read_csv() function, which will display the first five rows of the dataset.

The output looks likes this:

You can see from the output that the "ratings.csv" file contains the userId, movieId, ratings, and timestamp attributes. Each row in the dataset corresponds to one rating. The userId column contains the ID of the user who left the rating. The movieId column contains the Id of the movie, the rating column contains the rating left by the user. Ratings can have values between 1 and 5. And finally, the timestamp refers to the time at which the user left the rating.

There is one problem with this dataset. It contains the IDs of the movies but not their titles. We'll need movie names for the movies we're recommending. The movie names are stored in the "movies.csv" file. Let's import the file and see the data it contains. Execute the following script:

movie_names = pd.read_csv("E:\Datasets\ml-latest-small\\movies.csv")  
movie_names.head()  

The output looks likes this:

As you can see, this dataset contains movieId, the title of the movie, and its genre. We need a dataset that contains the userId, movie title, and its ratings. We have this information in two different dataframe objects: "ratings_data" and "movie_names". To get our desired information in a single dataframe, we can merge the two dataframes objects on the movieId column since it is common between the two dataframes.

We can do this using merge() function from the Pandas library, as shown below:

movie_data = pd.merge(ratings_data, movie_names, on='movieId')  

Now let's view our new dataframe:

movie_data.head()  

The output looks likes this:

You can see our newly created dataframe contains userId, title, and rating of the movie as required.

Now let's take a look at the average rating of each movie. To do so, we can group the dataset by the title of the movie and then calculate the mean of the rating for each movie. We will then display the first five movies along with their average rating using the head() method. Look at the the following script:

movie_data.groupby('title')['rating'].mean().head()  

The output looks likes this:

title  
"Great Performances" Cats (1998)           1.750000
$9.99 (2008)                               3.833333
'Hellboy': The Seeds of Creation (2004)    2.000000  
'Neath the Arizona Skies (1934)            0.500000  
'Round Midnight (1986)                     2.250000  
Name: rating, dtype: float64  

You can see that the average ratings are not sorted. Let's sort the ratings in the descending order of their average ratings:

movie_data.groupby('title')['rating'].mean().sort_values(ascending=False).head()  

If you execute the above script, the output will look like this:

title  
Burn Up! (1991)                                     5.0  
Absolute Giganten (1999)                            5.0  
Gentlemen of Fortune (Dzhentlmeny udachi) (1972)    5.0  
Erik the Viking (1989)                              5.0  
Reality (2014)                                      5.0  
Name: rating, dtype: float64  

The movies have now been sorted according to the ascending order of their ratings. However, there is a problem. A movie can make it to the top of the above list even if only a single user has given it five stars. Therefore, the above stats can be misleading. Normally, a movie which is really a good one gets a higher rating by a large number of users.

Let's now plot the total number of ratings for a movie:

movie_data.groupby('title')['rating'].count().sort_values(ascending=False).head()  

Executing the above script returns the following output:

title  
Forrest Gump (1994)                          341  
Pulp Fiction (1994)                          324  
Shawshank Redemption, The (1994)             311  
Silence of the Lambs, The (1991)             304  
Star Wars: Episode IV - A New Hope (1977)    291  
Name: rating, dtype: int64  

Now you can see some really good movies at the top. The above list supports our point that good movies normally receive higher ratings. Now we know that both the average rating per movie and the number of ratings per movie are important attributes. Let's create a new dataframe that contains both of these attributes.

Execute the following script to create ratings_mean_count dataframe and first add the average rating of each movie to this dataframe:

ratings_mean_count = pd.DataFrame(movie_data.groupby('title')['rating'].mean())  

Next, we need to add the number of ratings for a movie to the ratings_mean_count dataframe. Execute the following script to do so:

ratings_mean_count['rating_counts'] = pd.DataFrame(movie_data.groupby('title')['rating'].count())  

Now let's take a look at our newly created dataframe.

ratings_mean_count.head()  

The output looks like this:

You can see movie title, along with the average rating and number of ratings for the movie.

Let's plot a histogram for the number of ratings represented by the "rating_counts" column in the above dataframe. Execute the following script:

import matplotlib.pyplot as plt  
import seaborn as sns  
sns.set_style('dark')  
%matplotlib inline

plt.figure(figsize=(8,6))  
plt.rcParams['patch.force_edgecolor'] = True  
ratings_mean_count['rating_counts'].hist(bins=50)  

Here is the output of the script above:

From the output, you can see that most of the movies have received less than 50 ratings. While the number of movies having more than 100 ratings is very low.

Now we'll plot a histogram for average ratings. Here is the code to do so:

plt.figure(figsize=(8,6))  
plt.rcParams['patch.force_edgecolor'] = True  
ratings_mean_count['rating'].hist(bins=50)  

The output looks likes this:

You can see that the integer values have taller bars than the floating values since most of the users assign rating as integer value i.e. 1, 2, 3, 4 or 5. Furthermore, it is evident that the data has a weak normal distribution with the mean of around 3.5. There are a few outliers in the data.

Earlier, we said that movies with a higher number of ratings usually have a high average rating as well since a good movie is normally well-known and a well-known movie is watched by a large number of people, and thus usually has a higher rating. Let's see if this is also the case with the movies in our dataset. We will plot average ratings against the number of ratings:

plt.figure(figsize=(8,6))  
plt.rcParams['patch.force_edgecolor'] = True  
sns.jointplot(x='rating', y='rating_counts', data=ratings_mean_count, alpha=0.4)  

The output looks likes this:

The graph shows that, in general, movies with higher average ratings actually have more number of ratings, compared with movies that have lower average ratings.

Finding Similarities Between Movies

We spent quite a bit of time on visualizing and preprocessing our data. Now is the time to find the similarity between movies.

We will use the correlation between the ratings of a movie as the similarity metric. To find the correlation between the ratings of the movie, we need to create a matrix where each column is a movie name and each row contains the rating assigned by a specific user to that movie. Bear in mind that this matrix will have a lot of null values since every movie is not rated by every user.

To create the matrix of movie titles and corresponding user ratings, execute the following script:

user_movie_rating = movie_data.pivot_table(index='userId', columns='title', values='rating')  

user_movie_rating.head()  

We know that each column contains all the user ratings for a particular movie. Let's find all the user ratings for the movie "Forrest Gump (1994)" and find the movies similar to it. We chose this movie since it has the highest number of ratings and we want to find the correlation between movies that have a higher number of ratings.

To find the user ratings for "Forrest Gump (1994)", execute the following script:

forrest_gump_ratings = user_movie_rating['Forrest Gump (1994)']  

The above script will return a Pandas series. Let's see how it looks.

forrest_gump_ratings.head()  

userId  
1    NaN  
2    3.0  
3    5.0  
4    5.0  
5    4.0  
Name: Forrest Gump (1994), dtype: float64  

Now let's retrieve all the movies that are similar to "Forrest Gump (1994)". We can find the correlation between the user ratings for the "Forest Gump (1994)" and all the other movies using corrwith() function as shown below:

movies_like_forest_gump = user_movie_rating.corrwith(forrest_gump_ratings)

corr_forrest_gump = pd.DataFrame(movies_like_forest_gump, columns=['Correlation'])  
corr_forrest_gump.dropna(inplace=True)  
corr_forrest_gump.head()  

In the above script, we first retrieved the list of all the movies related to "Forrest Gump (1994)" along with their correlation value, using corrwith() function. Next, we created a dataframe that contains movie title and correlation columns. We then removed all the NA values from the dataframe and displayed its first 5 rows using the head function.

The output looks likes this:

Let's sort the movies in descending order of correlation to see highly correlated movies at the top. Execute the following script:

corr_forrest_gump.sort_values('Correlation', ascending=False).head(10)  

Here is the output of the script above:

From the output you can see that the movies that have high correlation with "Forrest Gump (1994)" are not very well known. This shows that correlation alone is not a good metric for similarity because there can be a user who watched '"Forest Gump (1994)" and only one other movie and rated both of them as 5.

A solution to this problem is to retrieve only those correlated movies that have at least more than 50 ratings. To do so, will add the rating_counts column from the rating_mean_count dataframe to our corr_forrest_gump dataframe. Execute the following script to do so:

corr_forrest_gump = corr_forrest_gump.join(ratings_mean_count['rating_counts'])  
corr_forrest_gump.head()  

The output looks likes this:

You can see that the movie "$9.99", which has the highest correlation has only three ratings. This means that only three users gave same ratings to "Forest Gump (1994)", "$9.99". However, we can deduce that a movie cannot be declared similar to the another movie based on just 3 ratings. This is why we added "rating_counts" column. Let's now filter movies correlated to "Forest Gump (1994)", that have more than 50 ratings. The following code will do that:

corr_forrest_gump[corr_forrest_gump ['rating_counts']>50].sort_values('Correlation', ascending=False).head()  

The output of the script, looks likes this:

Now you can see from the output the movies that are highly correlated with "Forrest Gump (1994)". The movies in the list are some of the most famous movies Hollywood movies, and since "Forest Gump (1994)" is also a very famous movie, there is a high chance that these movies are correlated.

Conclusion

In this article, we studied what a recommender system is and how we can create it in Python using only the Pandas library. It is important to mention that the recommender system we created is very simple. Real-life recommender systems use very complex algorithms and will be discussed in a later article.

If you want to learn more about recommender systems, I suggest checking out this very good course Building Recommender Systems with Machine Learning and AI. It goes much more in-depth and covers more complex and accurate methods than we did in this article.

Details

Main talk: Introduction to Packet Manipulation with Scapy

By Elizabeth Winkler Another Strata conference has come and gone. We had an incredible time meeting with a huge number of Anaconda users who came by our booth to chat! We also noticed some really interesting trends when it comes to the future of data science, machine learning, and AI. The future of ML/AI is containerized. …
Read more →

The post Key Trends and Takeaways from Strata New York 2018 appeared first on Anaconda.

Concurrency is often misunderstood and mistaken for parallelism. However, concurrency and parallelism are not the same thing. But why should you care? All you want is to make your Python application fast and responsive. Which you can achieve by distributing it across many CPUs. What difference does it make whether you call it concurrency or parallelism?

Why should you care?

Turns out, a lot actually. What happens is that you want to make your application fast, so you run it on more processors and… it gets slower. And you think: this is broken, this doesn’t make any sense. But what is broken is the understanding of concurrency and parallelism. That’s why in this blog post I’ll explain the difference between concurrency and parallelism and what’s in it for you.

Parallelism is the simultaneous execution of multiple things. These things are possibly related, possibly not. Parallelism is about doing a lot of things at once. Is is the act of running multiple computations simultaneously. Parallel computing can be of different types: bit-level, instruction-level, data and task parallelism. But it is entirely about execution.

Concurrency is the ability to deal with a lot of things at once. It is about the composition of independently executing things. Concurrency is a way to structure a program by breaking it into pieces that can be executed independently. This involves structuring but also coordinating thiese pieces, which is communication. Concurrency is the act of managing and running multiple computations at the same time.

For example, an operating system has multiple I/O devices: a keyboard driver, a mouse driver, a network driver and a screen driver. All these devices are managed by the operationg system as independent things. They are concurrent things. But they are not necessarily parallel. In fact, it does not matter. If you only have one processor, only one process can only ever run at a time. So, this is a concurrent model, but it is not necessarily parallel. It does not need to be parallel, but it is very benefical for it to be concurrent.

Concurrency is a way to structure things so that you can (maybe) run these things in parallel to do a better or faster job. But parallelism is not the goal of concurrency, the goal of concurrency is a good structure. A structure that allows you to scale.

Concurrency and parallelism with Celery and Dask

An example where concurrency matters is one of my consulting projects. A hedge fund needed to fetch (a lot of) market data from an external data vendor (Bloomberg). This process would run once a day and write the data to an internal database. We broke the process down into small pieces: request the data from the Bloomberg webservice, poll the webservice at regular intervals to establish whether the data is ready for collection, collect the data, transform the data, write the data into the database. We defined the tasks in such a way that each of them can be executed independently. Which means they can run concurrently. And in order to run the entire process faster, we made it parallel by scaling up the number of workers (we used Celery). But this could not have happened without concurrency.

An example where parallel computation matters - and concurrency not - is when dealing with large Pandas datasets in a Jupyter notebook. If your datasets are so large that they do not fit into your computer’s memory, you can solve that problem by parallelising the dataset across a cluster of machines. In this case, you want data parallelism. Concurrency does not matter. And this is a textbook example for Dask.

These are the ten most rated questions at Stack Overflow last week.
Between brackets: [question score / answers count]
Build date: 2018-09-15 20:02:19 GMT

1. Remainder function (%) runtime on numpy arrays is far longer than manual remainder calculation - [29/1]
2. Same name functions in same class, elegant way to determine which to call? - [24/7]
3. Why does adding a semicolon in Python change the result? - [13/2]
4. Returning int from a __str__ method in Python - [12/5]
5. Python - For loop millions of rows - [12/4]
6. Check if two file pointers point to same file in Python - [12/1]
7. Why can't eval find a variable defined in an outer function? - [9/1]
8. What do Unix Timestamps actually track? - [8/1]
9. How can I find out which index is out of range? - [7/5]
10. Can I replace an existing method of an object in Python? - [7/3]

A handy template to generate basic Python project structures.

Submitted by @Erik.

Nice one! A great #template for generating a basic #Python project structure! @pybiteshttps://t.co/zaxboOMGwN

— Julian Sequeira (@_juliansequeira) September 16, 2018

Ooo Language focused Docker images

Submitted by @dgjustice.

You have few options when it comes to making your Docker images smaller: 1. Use multi-stage builds 2. Use distroles… https://t.co/GEbFkoixav

— learnk8s (@learnk8s) September 09, 2018

Hacktoberfest is almost here!!

It's year 5 for #hacktoberfest! Celebrate Open Source with @DigitalOcean@Github and @Twilio. More info coming soon… https://t.co/4z3P8RZX9V

— Pybites (@pybites) September 07, 2018

A great instructional on Decorators!

Submitted by @Erik.

Make Your #Python Prettier With Decorators https://t.co/rF37XNU33r

— Amged Rustom (@codehill) September 08, 2018

Given Hacktoberfest is on the way: 50 Popular Python open-source projects

Want to get involved in python #OSS projects? Here's a list of projects to get you started 🐍 https://t.co/oFybFsJMmn

— Women Who Code BOS (@wwcboston) September 08, 2018

Snowy: A Small Python 3 module for manipulating and generating images

Submitted by @Erik (thanks again mate!).

Snowy, a pure Python image library accelerated with Numba https://t.co/22noU94DMf

— Python LibHunt (@PythonLibHunt) September 06, 2018

Simple Celery

Submitted by @dgjustice.

"Codementor: Use The File System To Queue Celery Tasks"https://t.co/iamUCxE03Q

— Rui Quintino (@rquintino) September 12, 2018

The UNIX Philosophy

Submitted by @Erik.

The secret sauce behind *nix success: Doing One Thing, Well: The UNIX Philosophy | Hackaday https://t.co/iCOJ21lnfn

— Christine Hall (@BrideOfLinux) September 13, 2018

The man behind the core 100Daysofcode Challenge! A great listen

This week's episode of the https://t.co/IA856cd6oj podcast is all about #100DaysOfCode. You'll hear from @ka11away…https://t.co/AQrAsDDSxy

— freeCodeCamp.org (@freeCodeCamp) September 13, 2018

Python hits the Top 3 in popularity!

Python. Top 3. It's official. https://t.co/viRgawZPlJ#positivepython

— Kenneth ☤ Reitz (@kennethreitz) September 10, 2018

Learn to code Python with Project Python

Submitted by @Erik.

Project Python is a free interactive book that will teach you to code in Python, using graphics, animations, and ga… https://t.co/pJehaZ7mdk

— Phil Shapiro (@philshapiro) September 13, 2018

Awesome Talk Python episode on the Python Community

RT @TalkPython: How well do you know the Python ecosystem? What's the ratio of web devs to data scientists? That question and many more ans…

— Betsy Waliszewski (@betswaliszewski) September 10, 2018

Sentdex is back with his self-driving cars in GTA V!

The return of self-driving cars in GTA V, Charles 2.0 begins: https://t.co/G190tBawO4#machinelearning#deeplearning#python

— Harrison Kinsley🐍 (@Sentdex) September 10, 2018

More Hettinger wisdom

Writing production #python code: * Simplest thing that works * Consider type signatures * Add repr, copy, and pickl… https://t.co/va3yW49m2y

— Raymond Hettinger (@raymondh) September 12, 2018

Cool, plot graphs on the console!

Ever wanted to plot graphs in the console? Try #bashplotlib package. https://t.co/XxAalm2oOkhttps://t.co/xl3XQtsokf

— Daily Python Tip (@python_tip) September 14, 2018

>>>frompybitesimportBob,JulianKeepCalmandCodeinPython!

Most if not all software companies have adopted to cloud infrastructure and services. AWS in particular is very popular amongst all. The intentions of this post is to host a few examples on using boto to make use of one of the services available on AWS i.e EC2. It is more likely than not to […]

The post Python, Boto and AWS EC2 appeared first on The Tara Nights.