Chris Moffitt: Pandas Groupby Warning

Introduction

One of the reasons I like using pandas instead of Excel for data analysis is that it is easier to avoid certain types of copy-paste Excel errors. As great as pandas is, there is still plenty of opportunity to make errors with pandas code. This article discusses a subtle issue with pandas groupby code that can lead to big errors if you’re not careful. I’m writing this because I have happened upon this in the past but it still bit me big time just recently. I hope this article can help a few of you avoid this mistake.

The Problem

To illustrate this problem, we’ll use a simple data set that shows sales for 20 customers and includes their region and an internal customer segment designation of Platinum, Gold or Silver. Here is the full data set:

The data looks pretty simple. There’s only one numeric column so let’s see what it totals to.

importpandasaspddf=pd.read_excel('https://github.com/chris1610/pbpython/raw/master/data/sales_9_2022.xlsx')df["Sales"].sum()

8000000

We have $8,000,000 in sales in the spreadsheet. Keep that number in mind.

Let’s do some simple analysis to summarize sales by region:

df.groupby(['Region']).agg({'Sales':'sum'})

We can double check the math:

df.groupby(['Region']).agg({'Sales':'sum'}).sum()

Sales8000000dtype:int64

Looks good. That’s what we expect. Now let’s see what sales look like by Segment:

df.groupby(['Region','Segment']).agg({'Sales':'sum'})

Which yields this table:

This looks good. No errors and the table seems reasonable. We should continue our analysis right?

Nope. There’s a potentially subtle issue here. Let’s sum the data to double check:

df.groupby(['Region','Segment']).agg({'Sales':'sum'}).sum()

Sales7019000dtype:int64

This only includes $7,019,000. Where did the other $981,000 go? Is pandas broken?

You can see the issue clearly if we use the dropna=False parameter to explicitly include NaN values in our results:

df.groupby(['Region','Segment'],dropna=False).agg({'Sales':'sum'})

Now we can see the NaN combinations with EMEA and the US groupings:

If we check the sum, we can see it totals to $8M.

df.groupby(['Region','Segment'],dropna=False).agg({'Sales':'sum'}).sum()

Sales8000000dtype:int64

The pandas documentation is very clear on this:

dropna: bool, default True

If True, and if group keys contain NA values, NA values together with row/column will be dropped. If False, NA values will also be treated as the key in groups.

If you are ok dropping those values, then use the default dropna=True .

However, if you want to ensure that all values (Sales in this particular case) are included, then make sure to use dropna=False in your groupby

An ounce of prevention

The main way to deal with this potential issue is to understand if you have any NaN values in your data. There are a couple of ways to do this.

You can use pure pandas:

df.isnull().sum()

CustomerID0CustomerName0Region0Segment4Sales0dtype:int64

There are other tools like missingno which provide a more robust interface for exploring the data.

I’m partial to sidetable. Here’s how to use it after it’s installed and imported:

df.stb.missing()

Regardless of the approach you use, its worth keeping in mind that you need to know if you have any null or NaN values in your data and how you would like to handle them in your analysis.

The other alternative to using the dropna is to explicitly fill in the values using fillna

df.fillna('unknown').groupby(['Region','Segment']).agg({'Sales':'sum'})

Now the unknown values are explicitly called out:

Conclusion

When working with pandas groupby , the results can be surprising if you have NaN values in your dataframe columns. The default behavior is to drop those values which means you can effectively “lose” some of your data during the process.

I have been bit by this behavior several times in the past. In some cases, it might not be a big deal. In others, you might need to sheepishly explain why your numbers aren’t adding up.

Have you seen this before? Let me know in the comments below.