In python, we sometimes need to filter not null and null values. In this article, we will discuss different ways to check for not null in pandas using examples.
We can check for not null in pandas using the notna() function and the notnull() function. Let us discuss each function one by one.
Check for Not Null in Pandas Using the notna() Method
As the name suggests, the notna() method works as a negation of the isna() method. The isna() method is used to check for nan values in pandas. The notna() function has the following syntax.
pandas.notna(object)
Here, the object can be a single python object or a collection of objects such as a python list or tuple.
If we pass a single python object to the notna() method as an input argument, it returns False if the python object is None, pd.NA or np.NaN object. For python objects that are not null, the notna() function returns True. You can observe this in the following example.
import pandas as pd
import numpy as np
x=pd.NA
print("The value is:",x)
output=pd.notna(x)
print("Is the value not Null:",output)
Output:
The value is: <NA>
Is the value not Null: False
In the above example, we have passed the pandas.NA object to the notna() function. Hence, it returns False.
When we pass a list or numpy array of elements to the notna() function, the notna() function is executed with each element of the array. After execution, it returns a list or array containing True and False values. The True values of the output array correspond to all the values that are not NA, NaN, or None at the same position in the input list or array. The False values in the output array correspond to all the NA, NaN, or None values at the same position in the input list or array. You can observe this in the following example.
import pandas as pd
import numpy as np
x=[1,2,pd.NA,4,5,None, 6,7,np.nan]
print("The values are:",x)
output=pd.notna(x)
print("Are the values not Null:",output)
Output:
The values are: [1, 2, <NA>, 4, 5, None, 6, 7, nan]
Are the values not Null: [ True True False True True False True True False]
In this example, we have passed a list containing 9 elements to the notna() function. After execution, the notna() function returns a list of 9 boolean values. Each element in the output list is associated with the element at the same index in the input list given to the notna() function. At the indices where the input list does not contain Null values, the output list contains True. Similarly, at indices where the input list contains null values, the output list contains False.
Check for Not NA in a Pandas Dataframe Using notna() Method
Along with the notna() function, python also provides us with the notna() method to check for not null values in pandas dataframes and series objects.
The notna() method, when invoked on a pandas dataframe, returns another dataframe containing True and False values. True values of the output dataframe correspond to all the values that are not NA, NaN, or None at the same position in the input dataframe. The False values in the output dataframe correspond to all the NA, NaN, or None values at the same position in the input dataframe. You can observe this in the following example.
import pandas as pd
import numpy as np
df=pd.read_csv("grade.csv")
print("The dataframe is:")
print(df)
output=df.notna()
print("Are the values not Null:")
print(output)
Output:
The dataframe is:
Class Roll Name Marks Grade
0 1 11 Aditya 85.0 A
1 1 12 Chris NaN A
2 1 14 Sam 75.0 B
3 1 15 Harry NaN NaN
4 2 22 Tom 73.0 B
5 2 15 Golu 79.0 B
6 2 27 Harsh 55.0 C
7 2 23 Clara NaN B
8 3 34 Amy 88.0 A
9 3 15 Prashant NaN B
10 3 27 Aditya 55.0 C
11 3 23 Radheshyam NaN NaN
Are the values not Null:
Class Roll Name Marks Grade
0 True True True True True
1 True True True False True
2 True True True True True
3 True True True False False
4 True True True True True
5 True True True True True
6 True True True True True
7 True True True False True
8 True True True True True
9 True True True False True
10 True True True True True
11 True True True False False
In the above example, we have invoked the notna() method on a dataframe containing NaN values along with other values. The notna() method returns a dataframe containing boolean values. Here, False values of the output dataframe correspond to all the values that are NA, NaN, or None at the same position in the input dataframe. The True values in the output dataframe correspond to all the not null values at the same position in the input dataframe.
Check for Not Null Values in a Column in Pandas Dataframe
Instead of the entire dataframe, you can also check for not null values in a column of a pandas dataframe. For this, you just need to invoke the notna() method on the particular column as shown below.
import pandas as pd
import numpy as np
df=pd.read_csv("grade.csv")
print("The dataframe column is:")
print(df["Marks"])
output=df["Marks"].notna()
print("Are the values not Null:")
print(output)
Output:
The dataframe column is:
0 85.0
1 NaN
2 75.0
3 NaN
4 73.0
5 79.0
6 55.0
7 NaN
8 88.0
9 NaN
10 55.0
11 NaN
Name: Marks, dtype: float64
Are the values not Null:
0 True
1 False
2 True
3 False
4 True
5 True
6 True
7 False
8 True
9 False
10 True
11 False
Name: Marks, dtype: bool
Check for Not NA in a Pandas Series Using notna() Method
Like a dataframe, we can also invoke the notna() method on a pandas Series object. In this case, the notna() method returns a Series containing True and False values. You can observe this in the following example.
import pandas as pd
import numpy as np
x=pd.Series([1,2,pd.NA,4,5,None, 6,7,np.nan])
print("The series is:")
print(x)
output=pd.notna(x)
print("Are the values not Null:")
print(output)
Output:
The series is:
0 1
1 2
2 <NA>
3 4
4 5
5 None
6 6
7 7
8 NaN
dtype: object
Are the values not Null:
0 True
1 True
2 False
3 True
4 True
5 False
6 True
7 True
8 False
dtype: bool
In this example, we have invoked the notna() method on a pandas series. The notna() method returns a Series of boolean values after execution. Here, False values of the output series correspond to all the values that are NA, NaN, or None at the same position in the input series. The True values in the output series correspond to all the not null values at the same position in the input series.
Check for Not Null in Pandas Using the notnull() Method
The notnull() method is an alias of the notna() method. Hence, it works exactly the same as the notna() method.
When we pass a NaN value, pandas.NA value, pandas.NaT value, or None object to the notnull() function, it returns False.
import pandas as pd
import numpy as np
x=pd.NA
print("The value is:",x)
output=pd.notnull(x)
print("Is the value not Null:",output)
Output:
The value is: <NA>
Is the value not Null: False
In the above example, we have passed pandas.NA value to the notnull() function. Hence, it returns False.
When we pass any other python object to the notnull() function, it returns True as shown below.
import pandas as pd
import numpy as np
x=1117
print("The value is:",x)
output=pd.notnull(x)
print("Is the value not Null:",output)
Output:
The value is: 1117
Is the value not Null: True
In this example, we passed the value 1117 to the notnull() function. Hence, it returns True showing that the value is not a null value.
When we pass a list or numpy array to the notnull() function, it returns a numpy array containing True and False values. You can observe this in the following example.
import pandas as pd
import numpy as np
x=[1,2,pd.NA,4,5,None, 6,7,np.nan]
print("The values are:",x)
output=pd.notnull(x)
print("Are the values not Null:",output)
Output:
The values are: [1, 2, <NA>, 4, 5, None, 6, 7, nan]
Are the values not Null: [ True True False True True False True True False]
In this example, we have passed a list to the notnull() function. After execution, the notnull() function returns a list of boolean values. Each element in the output list is associated with the element at the same index in the input list given to the notnull() function. At the indices where the input list contains Null values, the output list contains False. Similarly, at indices where the input list contains integers, the output list contains True.
Check for Not Null in a Pandas Dataframe Using the notnull() Method
You can also invoke the notnull() method on a pandas dataframe to check for nan values as shown below.
import pandas as pd
import numpy as np
df=pd.read_csv("grade.csv")
print("The dataframe is:")
print(df)
output=df.notnull()
print("Are the values not Null:")
print(output)
Output:
The dataframe is:
Class Roll Name Marks Grade
0 1 11 Aditya 85.0 A
1 1 12 Chris NaN A
2 1 14 Sam 75.0 B
3 1 15 Harry NaN NaN
4 2 22 Tom 73.0 B
5 2 15 Golu 79.0 B
6 2 27 Harsh 55.0 C
7 2 23 Clara NaN B
8 3 34 Amy 88.0 A
9 3 15 Prashant NaN B
10 3 27 Aditya 55.0 C
11 3 23 Radheshyam NaN NaN
Are the values not Null:
Class Roll Name Marks Grade
0 True True True True True
1 True True True False True
2 True True True True True
3 True True True False False
4 True True True True True
5 True True True True True
6 True True True True True
7 True True True False True
8 True True True True True
9 True True True False True
10 True True True True True
11 True True True False False
In the output, you can observe that the notnull() method behaves in exactly the same manner as the notna() method.
Instead of the entire dataframe, you can also use the notnull() method to check for not nan values in a column as shown in the following example.
import pandas as pd
import numpy as np
df=pd.read_csv("grade.csv")
print("The dataframe column is:")
print(df["Marks"])
output=df["Marks"].notnull()
print("Are the values not Null:")
print(output)
Output:
The dataframe column is:
0 85.0
1 NaN
2 75.0
3 NaN
4 73.0
5 79.0
6 55.0
7 NaN
8 88.0
9 NaN
10 55.0
11 NaN
Name: Marks, dtype: float64
Are the values not Null:
0 True
1 False
2 True
3 False
4 True
5 True
6 True
7 False
8 True
9 False
10 True
11 False
Name: Marks, dtype: bool
In a similar manner, you can invoke the notnull() method on a pandas series as shown below.
import pandas as pd
import numpy as np
x=pd.Series([1,2,pd.NA,4,5,None, 6,7,np.nan])
print("The series is:")
print(x)
output=pd.notnull(x)
print("Are the values not Null:")
print(output)
Output:
The series is:
0 1
1 2
2 <NA>
3 4
4 5
5 None
6 6
7 7
8 NaN
dtype: object
Are the values not Null:
0 True
1 True
2 False
3 True
4 True
5 False
6 True
7 True
8 False
dtype: bool
In the above example, we have invoked the notnull() method on a series. The notnull() method returns a Series of boolean values after execution. Here, the True values of the output series correspond to all the values that are not NA, NaN, or None at the same position in the input series. The False values in the output series correspond to all the NA, NaN, or None values at the same position in the input series.
Conclusion
In this article, we have discussed different ways to check for not null values in pandas. To learn more about python programming, you can read this article on how to sort a pandas dataframe. You might also like this article on how to drop columns from a pandas dataframe.
I hope you enjoyed reading this article. Stay tuned for more informative articles.
Happy Learning!
The post Check for Not Null Value in Pandas Python appeared first on PythonForBeginners.com.
What's New in Wing 9
