In the previous post, I have written a python program to create the database, earning table as well as input the first row of data into the earning table. In this chapter, I will create a simple UI to accept the user’s input so we do not need to hardcoded the values into the SQL query. I will leave the SQL commit code to the next chapter, we will only create a simple input’s UI in this chapter first.

As you can see I will create the above simple UI with tkinter which can then be further upgraded in the future to include more stuff.

import tkinter as tk
from tkinter import ttk

win = tk.Tk()

win.title("Earning Input")

def submit():
    pass

#create label frame for ui
earn= ttk.Labelframe(win, text = "Daily Earning Input")
earn.grid(column=0, row=0, padx=4, pady=4)
# create label for description
dLabel = ttk.Label(earn, text="Description:").grid(column=0, row=0)
# create text box for description
description = tk.StringVar()
descriptionEntry = ttk.Entry(earn, width=13, textvariable=description)
descriptionEntry.grid(column=1, row=0)

# create label for earning
eLabel = ttk.Label(earn, text="Earning:").grid(column=2, row=0)
# create text box for earning
earning = tk.StringVar()
earningEntry = ttk.Entry(earn, width=13, textvariable=earning)
earningEntry.grid(column=3, row=0)
# create the action button
action = ttk.Button(earn, text="submit", command=submit)
action.grid(column=5, row=0)

win.resizable(0,0)

win.mainloop()

I will write a program to submit the above data to the earning table in the next chapter.

This website will update constantly because I have decided to make writing articles one of my income streams, so do subscribe to the RSS feed of this site, and also, do read the other topic besides the python related one if you love another topic as well!

This is just for your info, I have found out another free to use universal database browser which you can download and play around through this link! If you are a software developer and would like me to include one of your free product in my post that is related to your type of product, do leave your comment under that particular post, I would really like to help a developer likes you to promote your software, so drop me a suggestion!

In Python

I'm sure it's been blogged about a buncha times before but, I couldn't find it, and I had to search too hard to find an example of this. Basically, what I'm trying to do is what Python does in this case, but in JavaScript:

defdo_something(arg="notset",**kwargs):print(f"arg='{arg.upper()}'")do_something(arg="peter")do_something(something="else")do_something()

In Python, the output of all this is:

arg='PETER'
arg='NOTSET'
arg='NOTSET'

It could also have been implemented in a more verbose way:

defdo_something(**kwargs):arg=kwargs.get("arg","notset")print(f"arg='{arg.upper()}'")

This more verbose format has the disadvantage that you can't quickly skim it and see and what the default is. That thing (arg = kwargs.get("arg", "notset")) might happen far away deeper in the function, making it hard work to spot the default.

In JavaScript

Here's the equivalent in JavaScript (ES6?):

functiondoSomething({arg="notset",...kwargs}={}){return`arg='${arg.toUpperCase()}'`;}console.log(doSomething({arg:"peter"}));console.log(doSomething({something:"else"}));console.log(doSomething());

Same output as in Python:

arg='PETER'
arg='NOTSET'
arg='NOTSET'

Notes

I'm still not convinced I like this syntax. It feels a bit too "hip" and too one-liner'y. But it's also pretty useful.

Mind you, the examples here are contrived because they're so short in terms of the number of arguments used in the function.
A more realistic thing like be a function that lists, upfront, all the possible parameters and for some of them, it wants to point out some defaults. E.g.

functionprocessFolder({source,destination="/tmp",quiet=false,verbose=false}={}){console.log({source,destination,quiet,verbose});// outputs// { source: '/user', destination: '/tmp', quiet: true, verbose: false }}console.log(processFolder({source:"/user",quiet:true}));

One could maybe argue that arguments that don't have a default are expected to always be supplied so they can be regular arguments like:

functionprocessFolder(source,{destination="/tmp",quiet=false,verbose=false}={}){console.log({source,destination,quiet,verbose});// outputs// { source: '/user', destination: '/tmp', quiet: true, verbose: false }}console.log(processFolder("/user",{quiet:true}));

But, I quite like keeping all arguments in an object. It makes it easier to write wrapper functions and I find this:

setProfile("My biography here",false,193.5,230,["anders","bengt"],"South Carolina");

...harder to read than...

setProfile({bio:"My biography here",dead:false,height:193.5,weight:230,middlenames:["anders","bengt"],state:"South Carolina"});

Demo Dataset

We are using the NY Taxi Dataset throughout this blog post because it is a real world dataset, has a reasonable size and some nice properties like different datatypes and includes some messy data (like all real world data engineering problems).

mkdirinputcdinputforiin{01..12};dowgetgethttps://s3.amazonaws.com/nyc-tlc/trip+data/yellow_tripdata_2018-$i.csvdone

Looking at the first rows of the data gives us some insight about the columns and data format

$head-n4input/yellow_tripdata_2019-01.csvVendorID,tpep_pickup_datetime,tpep_dropoff_datetime,passenger_count,trip_distance,RatecodeID,store_and_fwd_flag,PULocationID,DOLocationID,payment_type,fare_amount,extra,mta_tax,tip_amount,tolls_amount,improvement_surcharge,total_amount,congestion_surcharge1,2019-01-0100:46:40,2019-01-0100:53:20,1,1.50,1,N,151,239,1,7,0.5,0.5,1.65,0,0.3,9.95,1,2019-01-0100:59:47,2019-01-0101:18:59,1,2.60,1,N,239,246,1,14,0.5,0.5,1,0,0.3,16.3,2,2018-12-2113:48:30,2018-12-2113:52:40,3,.00,1,N,236,236,1,4.5,0.5,0.5,0,0,0.3,5.8,

Each of the files is roughly 700MiB uncompressed data:

$du-shinput/yellow_tripdata_2018-0*737Minput/yellow_tripdata_2018-01.csv715Minput/yellow_tripdata_2018-02.csv794Minput/yellow_tripdata_2018-03.csv784Minput/yellow_tripdata_2018-04.csv777Minput/yellow_tripdata_2018-05.csv734Minput/yellow_tripdata_2018-06.csv661Minput/yellow_tripdata_2018-07.csv661Minput/yellow_tripdata_2018-08.csv678Minput/yellow_tripdata_2018-09.csv

To convert the data to parquet we are going to use pandas to read the csv and store it in one large parquet file:

importglobimportpandasaspdfiles=glob.glob("input/yellow_tripdata_2018-*.csv")defread_csv(filename):returnpd.read_csv(filename,dtype={"store_and_fwd_flag":"bool"},parse_dates=["tpep_pickup_datetime","tpep_dropoff_datetime"],index_col=False,infer_datetime_format=True,true_values=["Y"],false_values=["N"],)dfs=list(map(read_csv,files))df=pd.concat(dfs)df.to_parquet("yellow_tripdata_2018.parquet")

The resulting parquet file has a size of 2.2GiB, while the sum of the original CSV files was 11GiB. Pandas supports two parquet implementations, fastparquet and pyarrow. They both have strengths and weaknesses. A comparison should be topic of an other blog post, and we are going to use pyarrow to analyze the data.

pyarrow can open a parquet file without directly reading all the data. It exposes metadata and only reads the necessary byte ranges of the file to get this information. This is extremely helpful when you are working with parquet files that are not available locally and stored in a remote location (like Amazon S3 or Azure Blob Storage), because you are only reading some kB instead of gigabytes of data to understand your dataset.

importpyarrow.parquetaspqfilename="yellow_tripdata_2018.parquet"pq_file=pq.ParquetFile(filename)data=[["columns:",pq_file.metadata.num_columns],["rows:",pq_file.metadata.num_rows],["row_roups:",pq_file.metadata.num_row_groups]]

So we are working with roughly 10 million records, 18 columns and the file has 2 row groups.

columns:18rows:102804250row_roups:2

The next step is to have a look at the schema of the parquet file:

s=pq_file.metadata.schemadata=[[s.column(i).name,s.column(i).physical_type,s.column(i).logical_type]foriinrange(len(s))]

Each column has a physical type that defines how the column is stored on disk and an optional logical type that is used to determine the actual data type. In case of the tpep_pickup_datetime and tpep_pickup_datetime the values are stored as INT64 types on disk but are represented as timestamps in pandas.

Logical types are used to extend the types that parquet can be used to store, by specifying how the primitive types should be interpreted. This keeps the set of primitive types to a minimum and reuses parquet's efficient encodings. For example, strings are stored as byte arrays (binary) with a UTF8 annotation, the parquet logical type definitions provides comprehensive documentation.

Now let's dive a little deeper into the file. A parquet file consists of one ore more row groups, which are a logical horizontal partitioning of the data into rows.

s=pq_file.metadata.schemadata=[]forrginrange(pq_file.metadata.num_row_groups):rg_meta=pq_file.metadata.row_group(rg)data.append([rg,rg_meta.num_rows,sizeof_fmt(rg_meta.total_byte_size)])

As we have written the parquet file with the default values in pandas we get row groups with a size on disk between 512MiB an 1,5GiB.

To understand the defaults of the row group sizing, a little bit of historical context is necessary. The parquet file format was developed as a columnar data storage format of the Apache Hadoop ecosystem and its underlying Hadoop distributed file system (HDFS):

Larger row groups allow for larger column chunks which makes it possible to do larger sequential IO. Larger groups also require more buffering in the write path (or a two pass write). We recommend large row groups (512MB - 1GB). Since an entire row group might need to be read, we want it to completely fit on one HDFS block. Therefore, HDFS block sizes should also be set to be larger. An optimized read setup would be: 1GB row groups, 1GB HDFS block size, 1 HDFS block per HDFS file. #

When working with parquet in python one does typically not use HDFS as a storage backend, but either the local file system or a cloud blob storage like Amazon S3 or Azure blob store. Depending on the read scenarios different row group sizes make sense.

Instead of concatenating the csv files in pandas and write them in one batch, on can use pyarrow.ParquetWriter directly to control how many rowgroups are written:

importpandasaspdimportpyarrowaspaimportpyarrow.parquetaspqmonths=range(1,13)defread_csv(month):filename="input/yellow_tripdata_2018-{:02d}.csv".format(month)df=pd.read_csv(filename,dtype={"store_and_fwd_flag":"bool"},parse_dates=["tpep_pickup_datetime","tpep_dropoff_datetime"],index_col=False,infer_datetime_format=True,true_values=["Y"],false_values=["N"],)returndf[(df['tpep_pickup_datetime'].dt.year=2018)&(df['tpep_pickup_datetime'].dt.month=month)]dfs=list(map(read_csv,months))table=pa.Table.from_pandas(dfs[0],preserve_index=False)writer=pq.ParquetWriter('yellow_tripdata_2018-rowgroups.parquet',table.schema)fordfindfs:table=pa.Table.from_pandas(df,preserve_index=False)writer.write_table(table)writer.close()

I have also added some data cleansing because, as mentioned earlier, the taxi dataset includes some messy data and we only want to have rows within a monthly data set with the correct pickup_datetime.

If we analyze the new parquet file again we can see that we now have a row group fore each month of data:

Next to the data in the rowgroups the parquet format specifies some metadata that is written per rowgroup:

rg_meta=pq_file.metadata.row_group(0)rg_meta.column(0)

Per column we can retrieve metadata like compression, sizing and datatype, but also statistical information about the values stored in the rowgroup for the particular column.

<pyarrow._parquet.ColumnChunkMetaDataobjectat0x7fa958ab72d0>file_offset:43125536file_path:physical_type:INT64num_values:8759557path_in_schema:tpep_pickup_datetimeis_stats_set:Truestatistics:<pyarrow._parquet.Statisticsobjectat0x7fa958ab7510>has_min_max:Truemin:2001-01-0511:45:23max:2018-01-3123:59:57null_count:0distinct_count:0num_values:8759557physical_type:INT64logical_type:Timestamp(isAdjustedToUTC=false,timeUnit=microseconds,is_from_converted_type=false,force_set_converted_type=false)converted_type(legacy):NONEcompression:SNAPPYencodings:('PLAIN_DICTIONARY','PLAIN','RLE','PLAIN')has_dictionary_page:Truedictionary_page_offset:1312236data_page_offset:2117164total_compressed_size:41813300total_uncompressed_size:68701768

Looking at the min and max statistics of the tpep_pickup_datetime:

column=1# tpep_pickup_datetimedata=[["rowgroup","min","max"]]forrginrange(pq_file.metadata.num_row_groups):rg_meta=pq_file.metadata.row_group(rg)data.append([rg,str(rg_meta.column(column).statistics.min),str(rg_meta.column(column).statistics.max)])print_table(data)

The statistics show an interesting property. The values per rowgroup are disjunct. This means without reading the full data you can know which values to expect in which rowgroup.

If columns are sorted and/or rowgroups have disjunct values in a dataset, readers can take advantagea of this through a feature called predicate pushdown. To get all taxi trips on a certain day 2018-02-20 the parquet reader can now look at the rowgroup statistics, compare the predicate tpep_pickup_datetime.min <= 2019-02-20 and tpep_pickup_datetime.max >= 2019-02-20 against it and only read the parts of the file that potentially include rows for the day. In our case one would only have to read the rowgroup 1 and by this 150MiB instead of 2.1 GiB.

In contrast if we print the statistic for the column trip_distance:

Even if readers would only be interested in rows with a certaintrip_distance, one would have to read the whole dataset most of the time. Only for distances greater 1000 one could skip some of the rowgroups.

Summary

Query engines on parquet files like Hive, Presto or Dremio provide predicate pushdown out of the box to speed up query times and reduce I/O.

In the python ecosystem fastparquet has support for predicate pushdown on row group level. pyarrow has an open ticket for an efficient implementation in the parquet C++ reader.

Implementing predicate pushdown in python on top of the exposed statistics is not that hard. In my team we have done this within kartothek to speed up reads from large datasets from the Azure Blob storage.

1. Use data in pandas data frames to match columns together - [13/4]
2. Filtering DataFrame on groups where count of element is different than 1 - [7/4]
3. How to sort a list of sub-lists by the contents of sub-lists, where sub-lists contain strings and booleans? - [7/4]
4. how can I clean this data type object and transform it to type float maintaining Null and NaN - [6/3]
5. Finding all the combinations of free polyominoes within a specific area with a SAT-solver (Python) - [6/2]
6. Filter numeric column by dictionary of value ranges - [6/2]
7. Select NumPy Values Around Index - [5/4]
8. Slicing arrays with lists - [5/4]
9. How to extract content from <script> using Beautiful Soup - [5/4]
10. How do you search a column and fill another column with what you find? (in pandas) - [5/3]

Chart.js is a cool open source JavaScript library that helps you render HTML5 charts. It is responsive and counts with 8 different chart types.

In this tutorial we are going to explore a little bit of how to make Django talk with Chart.js and render some simple charts based on data extracted from our models.

Installation

For this tutorial all you are going to do is add the Chart.js lib to your HTML page:

<script src="https://cdn.jsdelivr.net/npm/chart.js@2.9.3/dist/Chart.min.js"></script>

You can download it from Chart.js official website and use it locally, or you can use it from a CDN using the URL above.

Example Scenario

I’m going to use the same example I used for the tutorial How to Create Group By Queries With Django ORM which is a good complement to this tutorial because actually the tricky part of working with charts is to transform the data so it can fit in a bar chart / line chart / etc.

We are going to use the two models below, Country and City:

classCountry(models.Model):name=models.CharField(max_length=30)classCity(models.Model):name=models.CharField(max_length=30)country=models.ForeignKey(Country,on_delete=models.CASCADE)population=models.PositiveIntegerField()

And the raw data stored in the database:

Example 1: Pie Chart

For the first example we are only going to retrieve the top 5 most populous cities and render it as a pie chart. In this strategy we are going to return the chart data as part of the view context and inject the results in the JavaScript code using the Django Template language.

views.py

fromdjango.shortcutsimportrenderfrommysite.core.modelsimportCitydefpie_chart(request):labels=[]data=[]queryset=City.objects.order_by('-population')[:5]forcityinqueryset:labels.append(city.name)data.append(city.population)returnrender(request,'pie_chart.html',{'labels':labels,'data':data,})

Basically in the view above we are iterating through the City queryset and building a list of labels and a list of data. Here in this case the data is the population count saved in the City model.

For the urls.py just a simple routing:

urls.py

fromdjango.urlsimportpathfrommysite.coreimportviewsurlpatterns=[path('pie-chart/',views.pie_chart,name='pie-chart'),]

Now the template. I got a basic snippet from the Chart.js Pie Chart Documentation.

pie_chart.html

{% extends 'base.html' %}

{% block content %}
  <divid="container"style="width: 75%;"><canvasid="pie-chart"></canvas></div><script src="https://cdn.jsdelivr.net/npm/chart.js@2.9.3/dist/Chart.min.js"></script><script>varconfig={type:'pie',data:{datasets:[{data:{{data|safe}},backgroundColor:['#696969','#808080','#A9A9A9','#C0C0C0','#D3D3D3'],label:'Population'}],labels:{{labels|safe}}},options:{responsive:true}};window.onload=function(){varctx=document.getElementById('pie-chart').getContext('2d');window.myPie=newChart(ctx,config);};</script>

{% endblock %}

In the example above the base.html template is not important but you can see it in the code example I shared in the end of this post.

This strategy is not ideal but works fine. The bad thing is that we are using the Django Template Language to interfere with the JavaScript logic. When we put {{data|safe}} we are injecting a variable that came from the server directly in the JavaScript code.

The code above looks like this:

Example 2: Bar Chart with Ajax

As the title says, we are now going to render a bar chart using an async call.

views.py

fromdjango.shortcutsimportrenderfromdjango.db.modelsimportSumfromdjango.httpimportJsonResponsefrommysite.core.modelsimportCitydefhome(request):returnrender(request,'home.html')defpopulation_chart(request):labels=[]data=[]queryset=City.objects.values('country__name').annotate(country_population=Sum('population')).order_by('-country_population')forentryinqueryset:labels.append(entry['country__name'])data.append(entry['country_population'])returnJsonResponse(data={'labels':labels,'data':data,})

So here we are using two views. The home view would be the main page where the chart would be loaded at. The other view population_chart would be the one with the sole responsibility to aggregate the data the return a JSON response with the labels and data.

If you are wondering about what this queryset is doing, it is grouping the cities by the country and aggregating the total population of each country. The result is going to be a list of country + total population. To learn more about this kind of query have a look on this post: How to Create Group By Queries With Django ORM

urls.py

fromdjango.urlsimportpathfrommysite.coreimportviewsurlpatterns=[path('',views.home,name='home'),path('population-chart/',views.population_chart,name='population-chart'),]

home.html

{%extends'base.html'%}{%blockcontent%}<divid="container"style="width: 75%;"><canvasid="population-chart"data-url="{%url'population-chart'%}"></canvas></div><script src="https://code.jquery.com/jquery-3.4.1.min.js"></script><script src="https://cdn.jsdelivr.net/npm/chart.js@2.9.3/dist/Chart.min.js"></script><script>$(function(){var$populationChart=$("#population-chart");$.ajax({url:$populationChart.data("url"),success:function(data){varctx=$populationChart[0].getContext("2d");newChart(ctx,{type:'bar',data:{labels:data.labels,datasets:[{label:'Population',backgroundColor:'blue',data:data.data}]},options:{responsive:true,legend:{position:'top',},title:{display:true,text:'Population Bar Chart'}}});}});});</script>{%endblock%}

Now we have a better separation of concerns. Looking at the chart container:

<canvasid="population-chart"data-url="{%url'population-chart'%}"></canvas>

We added a reference to the URL that holds the chart rendering logic. Later on we are using it to execute the Ajax call.

var$populationChart=$("#population-chart");$.ajax({url:$populationChart.data("url"),success:function(data){// ...}});

Inside the success callback we then finally execute the Chart.js related code using the JsonResponse data.

Conclusions

I hope this tutorial helped you to get started with working with charts using Chart.js. I published another tutorial on the same subject a while ago but using the Highcharts library. The approach is pretty much the same: How to Integrate Highcharts.js with Django.

If you want to grab the code I used in this tutorial you can find it here: github.com/sibtc/django-chartjs-example.

In this chapter, we will create the Input class which has a few methods used to create a database and the earning table as well as an insert method to insert earning values into that earning table. After that, we will create an object of that class inside the main program which can be used to create a database and the earning table and then insert the user inputs into that earning table.

First of all, this will be the final folder structure once we have created the input class and created the database’s table with some values in it.

The folder and files of this project

I am using PyCharm 2019.3.1 Community Edition which is a free python editor with a great user interface to create this project.

We have already created the python program used to generate the database, earning table and to submit the user input to the earning table in the previous chapter. In this chapter, we just need to create a class which will interact with the Tkinter program to perform the above tasks.

import sqlite3
from datetime import datetime

class Input:
    def __init__(self, description, earning):
        self.description = description
        self.earning = earning

    def setting(self):
        conn = sqlite3.connect('daily_earning.db')
        print("Opened database successfully")
        try:
            conn.execute('''CREATE TABLE DAILY_EARNING_CHART
                 (ID INTEGER PRIMARY KEY AUTOINCREMENT,
                 DESCRIPTION    TEXT (50)   NOT NULL,
                 EARNING    TEXT  NOT NULL,
                 TIME   TEXT NOT NULL);''')
        except:
            pass

        print("Table created successfully")

        conn.close()

    def submit(self): # Insert values into earning table
        try:
            sqliteConnection = sqlite3.connect('daily_earning.db')
            cursor = sqliteConnection.cursor()
            print("Successfully Connected to SQLite")

            sqlite_insert_query = "INSERT INTO DAILY_EARNING_CHART (DESCRIPTION,EARNING,TIME) VALUES ('" + self.description + "','"+ self.earning + "',datetime('now', 'localtime'))"

            count = cursor.execute(sqlite_insert_query)
            sqliteConnection.commit()
            print("Record inserted successfully into DAILY_EARNING_CHART table", cursor.rowcount)
            cursor.close()

        except sqlite3.Error as error:
            print("Failed to insert earning data into sqlite table", error)
        finally:
            if (sqliteConnection):
                sqliteConnection.close()
                print("The SQLite connection is closed")

The above class will allow us to create an object in which the main program will be able to use to call various methods that will then create the database, earning table and insert values into that table.

import tkinter as tk
from tkinter import ttk

from Input import Input

win = tk.Tk()

win.title("Earning Input")

def submit():
    if(description.get()!='' and earning.get()!=""):
        sub_mit = Input(description.get(), earning.get())
        sub_mit.setting()
        sub_mit.submit()
    else:
        print("You need to enter a value!")

#create label frame for ui
earn= ttk.Labelframe(win, text = "Daily Earning Input")
earn.grid(column=0, row=0, padx=4, pady=4)
# create label for description
dLabel = ttk.Label(earn, text="Description:").grid(column=0, row=0)
# create text box for description
description = tk.StringVar()
descriptionEntry = ttk.Entry(earn, width=13, textvariable=description)
descriptionEntry.grid(column=1, row=0)

# create label for earning
eLabel = ttk.Label(earn, text="Earning:").grid(column=2, row=0)
# create text box for earning
earning = tk.StringVar()
earningEntry = ttk.Entry(earn, width=13, textvariable=earning)
earningEntry.grid(column=3, row=0)
# create the action button
action = ttk.Button(earn, text="submit", command=submit)
action.grid(column=5, row=0)

win.resizable(0,0)

win.mainloop()

The Input class and the main python program has taken care of the followings:-

• If the table and database have not been created yet, then create them.
• If there is no input from the user then just skip the submit process.
• Always inform the programmer whether the program has successfully created a database or inserted values or not.

Let us start the Tkinter user interface and insert some values

The program has succeeded in creating a database, earning table and inserting the values into the earning table

Open DB Browser and view the outcome

DB Browser is a great tool and we will stick with this tool for a while, we will continue to modify our project in the next chapter before moving onward.

This week we welcome Sebastián Ramírez (@tiangolo) as our PyDev of the Week! Sebastián is the creator of the FastAPI Python web framework. He maintains his own website/blog which you should check out if you have some free time. You can also see his open source projects there. You can also see what projects he is contributing to over on Github.

Let’s take a few moments to get to know Sebastián better!

Can you tell us a little about yourself (hobbies, education, etc):

Hey! I’m Sebastián Ramírez, I’m from Colombia, and currently living in Berlin, Germany.

I was “homeschooled” since I was a kid, there wasn’t even a term for that, it wasn’t common. I didn’t go to school nor university, I studied everything at home. At about (I think) 14 I started fiddling with video edition and visual effects, some music production, and then graphic design to help with my parent’s business.

Then I thought that building a website should be almost the same …soon I realized I had to learn some of those scary “programming languages”. HTML, CSS, and JavaScript (“but!!! HTML and CSS are not…” I know, I know). But soon I was able to write a very short text, in a text file, and use it to make a browser show a button, that when clicked would show a pop-up saying “Hello world!”… I was so proud and excited about it, I guess it was a huge “I maked these” moment for me. I still feel that rush, that excitement from time to time. That’s what makes me keep loving code.

I also like to play videogames and watch movies, but many times I end up just coding in my free time too. I’m boring like that… 

Why did you start using Python?

At some point, I was taking several (too many) courses on Coursera, edX, and Udacity. I knew mainly frontend vanilla JavaScript (Node.js was just starting), so I did all the exercises for the Cryptography, Algorithms, and other courses with JavaScript running in a browser, it sounds a bit crazy now.

Then I took Andrew Ng’s ML course on Coursera, it used Octave (kinda Matlab) and it taught me enough Octave/Matlab for the course, and also that learning a new language was not so terrible. But then an AI course from Berkeley/edX required Python… so I took the Python crash course that was embedded (it was just like one page). And I went into the AI course with that. I loved the course, and with it, I started to love Python. I had to read a lot of Python docs, tutorials, StackOverflow, etc. just to be able to keep the pace, but I loved it. After that, I took an MIT/edX Python course and several others.

And I just kept learning and loving Python more and more.

What other programming languages do you know and which is your favorite?

I’m quite fond of JavaScript as it was my first language. I have also used some compile-to-JS languages like CoffeeScript, TypeScript. I have also ended up doing quite some Bash for Linux and Docker.

I really like TypeScript, and now I almost never do plain JS without TS, I love having autocompletion everywhere and type checks for free. I naturally got super excited when optional type hints for Python were released as a Christmas gift in 2016. And 2 years later FastAPI came to be, heavily based on them.

What projects are you working on now?

I put a lot of my free time to FastAPI and sibling projects, and also some of the other open source tools I’ve built.

Right now I’m working for Explosion AI. They are the creators of spaCy, the open source, industrial-strength, Natural Language Processing package.

At work, I’m currently on the team building the teams version of Prodigy, a commercial tool for radically efficient machine teaching, using Active (Machine) Learning.

But as open source is very important for the company (because they’re awesome like that), I also devote part of my working time to FastAPI and family.

Which Python libraries are your favorite (core or 3rd party)?

Core, I would say typing, as it’s relatively new and it deserves more attention, I think not many people know that those optional type hints are what powers autocompletion and automatic type checks for errors in editors. Most of the developers love those features, but a few know that type hints are what powers them.

3rd party, I think naturally Starlette and Pydantic, as they power FastAPI.

But I think Pydantic also deserves a lot more attention, even outside of FastAPI. It’s an amazing library, really easy to use, and saves a lot of time debugging, validating, documenting, and parsing data. It’s also great for managing application settings and just moving data around in an app. Imagine using deeply nested dicts and lists of values, but not having to remember what is what everywhere (“did I write ‘username’ or ‘user_name’ as the key in the other function?” ), just having autocomplete for everything and automatic error checks (type checks).

I recently built a GitHub action to help me manage issues, and most of the work ended up being done automatically by Pydantic. It also works great for data science, cleaning and structuring data.

This list could probably grow a lot, but some highlights:

* Dev utils: Poetry or Pipenv, Black, Isort, Flake8, Autoflake8, Mypy, Pytest, Pytest-cov

* For docs: Mkdocs with Mkdocs-material and Markdown-include

* Others: Cookiecutter, Requests or HTTPX, Uvicorn

* Data Science/Processing, ML: Keras with TensorFlow or PyTorch, Numpy, PyAV, Pandas, Numba, and of course, spaCy and Prodigy

Is there anything else you’d like to say?

I love the Python community, I think it’s a friendly ecosystem and I would like all of us to help it be even more welcoming, friendly, and inclusive. I think we all can help in achieving that.

New developers: don’t be shy, you can help too. Updating documentation of a new tool you are learning is a great start.

Maintainers: help us build a friendly ecosystem, it’s difficult for a new developer to come and try to help. Please be nice.

—————————————————————–

Here are a couple of others that you can answer if you want to, but if you don’t have the time, that’s ok:

How did your project, FastAPI, come about?

I had spent years finding the right tools and plug-ins (even testing other languages with their frameworks) to build APIs.

I wanted to have automatic docs; data validation, serialization, and documentation; I wanted it to use open standards like OpenAPI, JSON Schema, and OAuth2; I wanted it to be independent of other things, like database and ORM, etc.

I had somewhat achieved it with some components from several places, but it was difficult to use and somewhat brittle, as there were a lot of components and plug-ins, and I had to somehow make them interact well together.

I also discovered that having types as, in TypeScript, it was possible to have autocompletion and checks for many errors (type checks). But then Python added optional type hints! 

And after searching everywhere for a framework that used them and did all that, and finding that it didn’t exist yet, I used all the great ideas brought by previous tools with some of mine to integrate all those features in a single package.

I also wanted to provide a development experience as pleasant as possible, with as small/simple code as possible, while having great performance (powered by the awesome tools underneath, Starlette and Pydantic).

What top three things did you learn while creating the package?

First, that it was possible. I thought building a package that others found useful was reserved for some olympian-semi-god coders. It turns out that if there’s something to solve, and you solve it, and you help others use it to solve the same thing, that’s all that is needed.

Second, I learned a lot about how Python interacts with the web. FastAPI uses the new standard ASGI (the spiritual successor to WSGI), I learned a lot of it. Especially reading the beautiful and clean code of Starlette.

Third, I learned a lot about how Python works underneath by adding features to Pydantic. To be able to provide all its awesome features and the great simplicity while using it, its own internal code has to be, naturally, very complex. I even learned about undocumented features of Python’s internal typing parsing, that are needed to make everything work.

But I don’t think that a new developer needs to learn the last 2 things, the first one is the most important one. And as I was able to build FastAPI using the great tools and ideas provided by others, I hope FastAPI can provide a simple and easy way for others to build their ideas.

Do you have any advice for other aspiring package creators?

Write docs for your package. It doesn’t exist completely if it’s not well documented. And write them from the point of view of a new user, not of your own.

Also, building and publishing a new package is now extremely easy. Use Flit or Poetry if your project is simple enough to use them (i.e. pure Python, you are not building with Cython extensions, etc).

The post PyDev of the Week: Sebastián Ramírez appeared first on The Mouse Vs. The Python.

The post How to Save a Seaborn Plot as a File (e.g., PNG, PDF, EPS, TIFF) appeared first on Erik Marsja.

In this short post, we will learn how to save Seaborn plots to a range of different file formats. More specifically, we will learn how to use the plt.savefig method save plots made with Seaborn to:

1. Portable Network Graphics (PNG)
2. Portable Document Format (PDF)
3. Encapsulated Postscript (EPS)
4. Tagged Image File Format (TIFF)
5. Scalable Vector Graphics (SVG)

First, we will create some basic plots and work with Matplotlibs savefig method to export the files to the different file formats. There is more information on data visualization in Python using Seaborn and Matplotlib in the following posts:

• 9 Data Visualization Techniques you Need to Know in Python
• How to Make Seaborn Scatter Plots in Python
• Exploratory Data Analysis in Python (Scatter plots)
• How to Visualize Response Time Distributions using Python and Seaborn

Prerequisites: Python and Seaborn

Now, before learning how to save Seaborn plots (e.g., to .png files), we need to have both Python and Seaborn installed. There are two easy methods to install Seaborn. First, if we don’t have Python installed we can download and install a Python distribution packed with Seaborn (e.g., Anaconda). Second, if we already have Python installed we can install Seaborn using Pip. Of course, there are a number of mandatory dependencies (i.e., NumPy, SciPy, Matplotlib, & Pandas) but pip will installed them too. At times, we may need to update Seaborn and we will after we have installed Seaborn, learn how to update Seaborn to the latest version.

How to Install Seaborn with Pip

Now we’re ready to use pip to install Seaborn. It’s very simple. First, we open up a terminal window, or Windows command prompt, and type pip -m install seaborn.

How to Upgrade Seaborn using Pip and Conda

In this section, before creating and saving a Seaborn plot we will learn how to upgrade Seaborn using pip and conda. First, if we want to upgrade Seaborn with pip we just type the following code: pip install -upgrade seaborn.

If we, on the other hand, have the Anaconda Python distribution installed we will use conda to update Seaborn. Now, this is also very easy and we will open up a terminal window (or the Anaconda Prompt, if we use Windows) and type conda update seaborn.

Learn more about installing, using, and upgrading Python packages using pip, pipx, and conda in the following two posts:

• Installing, using. and upgrading Python packages using pip and conda
• Installing Python packages directly into virtual environments using pipx

When we are installing and upgrading packages, we may notice, that we also need to upgrade pip.

Using the plt.savefig Method

In this section, before we start saving a Seaborn plot as a file, we are going to learn a bit more about the plt.savefig method.

As can be seen in the image above, we have a number of arguments to work with.

In this post, we are going to work with some of them when saving Seaborn plots as a file (e.g., PDF). Specifically, we will use the fname, dpi, format, orientation, and transparent. Now, orientation can only be used in the format is postscript (e.g., eps). Thus, we will only use it when we are saving a Seaborn plot as a .eps file.

Concerning the other arguments, they will work with other formats as well but we will only use them when we save a Seaborn plot as a png file.

How to Save a Seaborn Plot as a File (i.e., png, eps, svg, pdf)

In this section, we are finally going to learn how to save a Seaborn plot. Now, in all the examples of saving Seaborn plots here we will start by creating a plot. First, we need to import Seaborn, matplotlib.pyplot, and Pandas. Here, we are following convention and import seaborn as sns, matplotlib.pyplot as plt, and pandas as pd. Note, we need to do this in all our Python scripts in which we are visualizing data and saving the plots to files.

import seaborn as sns
import matplotlib.pyplot as plt
import pandas as pd

Example Data

Next, we are going to import data to visualize. Here, we are using Pandas and the pd.read_csv method to load data from a CSV file.

data = 'https://vincentarelbundock.github.io/Rdatasets/csv/datasets/mtcars.csv'
df = pd.read_csv(data, index_col=0)
df.head(6)

How to Save a Seaborn Plot as png

In the first example, we are going to export the Seaborn plot as Portable Network Graphics (png). First, we need to create our plot and we are going to create a simple histogram using sns.distplot.

In the second line of code, we are using plt.savefig with only the fname argument. Note, we are saving the file as a png only by using a string as fname. That is, we add the file ending .png to tell plt.savefig that we want the Seaborn plot saved as a png.

sns.distplot(df['mpg'])
plt.savefig('save_as_a_png.png')

Saved PNG file (larger version)

Saving a High-Resolution PNG

Now, we will also use the dpi argument here. If we, for instance, wanted to save the histogram plot as a high-resolution image file. Most commonly is, however, that when we need a high-resolution image, we also want a different format. Anyway, if we want to save the histogram with 300 dpi, this is how we do it:

sns.distplot(df['mpg'])
plt.savefig('saving-a-high-resolution-seaborn-plot.png', dpi=300)

How to Save a Transparent PNG

In this example, we are going to save the Seaborn plot as a transparent png. That is, we will use the transparent argument and set it to true.

sns.distplot(df['mpg'])
plt.savefig('saving-a-seaborn-plot-as-png-file-transparent.png', 
           transparent=True)

Transparent PNG

Save a Seaborn Figure as a PDF File

In this section, we are going to learn how to save a Seaborn plot as a .pdf file. As when we learned how to save a histogram figure as a png, we first need to make a plot. Here, we are going to create a scatter plot using the scatterplot method from Seaborn.

sns.scatterplot(x='wt', y='drat', data=df)
plt.savefig('saving-a-seaborn-plot-as-pdf-file.pdf')

Saving a Seaborn Plot as a High-Resolution PDF file

In this section, we are going to use the dpi argument again. Many scientific journals requires image files to be in high-resolution images. For example, the PLOS journals (e.g., Plos One) requires figures to be in 300 dpi. Here’s how to save a Seaborn plot as a PDF with 300 dpi:

<pre><code class="lang-py">sns.scatterplot(x='wt', y='drat', data=df)
plt.savefig('saving-a-seaborn-plot-as-pdf-file-300dpi.pdf', 
           dpi=300)</code></pre>

How to Save Python Data Visualizations (e.g., Seaborn plots) to an EPS file

In this section, we will carry on and learn how to save a Seaborn plot as an Encapsulated Postscript file. First, we will change the file ending (the fname argument) to .eps to export the plot as an EPS file. Second, we will learn how to save the Seaborn plot as a high-resolution .eps file.

In this example, we are going to create a violin plot using Seaborn’s catplot method and save it as a file:

sns.catplot(x='cyl', y='drat', hue='am',

                data=df, kind='violin')

plt.savefig('saving-a-seaborn-plot-as-eps-file.eps')

Violin plot to save to eps

Saving a Seaborn Plot as a High-resolution EPS file

Now, we are already familiar with how to set the dpi but in the next example we are going to save the Seaborn violin plot as a high-resolution image file:

sns.catplot(x='cyl', y='drat', hue='am',
                data=df, kind='violin')
plt.savefig('saving-a-seaborn-plot-as-eps-file-300dpi.eps',
           dpi=300)

Saving plots as EPS, and also TIFF (as we will learn about later) is also formats that are required, or recommended (e.g., see the American Psychological Associations recommendations in ther submission guidelines).

Saving the Seaborn Plot in Landscape (EPS)

Now, in the final saving a plot to EPS example, we are going to use the orientation argument to save the Seaborn plot in landscape.

sns.catplot(x='cyl', y='drat', hue='am',
                data=df, kind='violin')
plt.savefig('saving-a-seaborn-plot-as-eps-file-300dpi.eps', orientation="landscape",
           dpi=300)

Seaborn plot saved in landscape orientation

How to Save Seaborn plots to TIFF files

In this section, we will use the final format to save the Seaborn plots. More specifically, we will save the plots as in the Tagged Image File Format (TIFF). Now, as with the other formats, we will change the file ending (the fname argument) to .tiff. This, will as we now know, save the plot as a TIFF file.

Swarm plot to save as TIFF

In the how to save to a TIFF file example, we are going to continue working with the catplot method but now we are going to create a swarm plot and save it as a TIFF file.

sns.catplot(x='vs', y='wt', hue='am',
                data=df, kind='swarm')
plt.savefig('saving-a-seaborn-plot-as-eps-file.tiff')

Saving a (Swarm) Plot as a High-Resolution TIFF File

In the final, how to save a TIFF file example, we are going to save the Seaborn plot as a high-resolution TIFF file. Why? Well, as previously mentioned a lot of scientific journals require, or recommend, that we use the PDF, EPS, or TIFF format when we submit our scientific papers.

Now, here’s how to explort the Seaborn plot as a TIFF file with high-resolution:

<pre><code class="lang-py">sns.catplot(x='vs', y='wt', hue='am',
                data=df, kind='swarm')
plt.savefig('saving-a-seaborn-plot-as-eps-file.tiff',
           dpi=300)</code></pre>

How to Save a Plot in Python Seaborn as a SVG File

In the final example, we are going to save a Seaborn plot as a SVG file. Now, we already know that we just change the file ending to accomplish this:

sns.catplot(x='vs', y='wt', hue='am',
                data=df, kind='swarm')
plt.savefig('saving-a-seaborn-plot-as-eps-file.svg')

Note, if we want to save the SVG file as a high-resolution SVG file, we’d just add the dpi=300 as we’ve learned in the previous examples.

There are, of course, a couple of other arguments that could’ve been used when working with plt.savefig. However, when working with Seaborn these are, most often, redundant. The formatting is already taken care of and we get quite nice-looking image files of our Seaborn plots.

Conclusion: Saving Plots in Python to Files

In this Python data visualization tutorial, we have learned how to save Python plots (made in Seaborn) as PNG, PDF, EPS, TIFF, and SVG files. It was quite simple, and we simply used the savefig method.

The post How to Save a Seaborn Plot as a File (e.g., PNG, PDF, EPS, TIFF) appeared first on Erik Marsja.

Let us continue with the final touch up of the python class uses to create SQL database and submits values to the SQL database’s earning table.

What we have modified from the previous Input class.

• The main program will only call the Input class once to create the SQL database and the earning table if they have not been created yet!
• We will pass the action button into the Input class and disable the action button if there is an ongoing committing job.
• The main program will pass the description and earning into the submit method of the Input class.

Below is the final revised version for both the main program and the Input class.

import tkinter as tk
from tkinter import ttk

from Input import Input

win = tk.Tk()

win.title("Earning Input")

def submit():
    if(description.get()!='' and earning.get()!=""):
        sub_mit.submit(description.get(), earning.get())
    else:
        print("You need to enter a value!")

#create label frame for ui
earn= ttk.Labelframe(win, text = "Daily Earning Input")
earn.grid(column=0, row=0, padx=4, pady=4)
# create label for description
dLabel = ttk.Label(earn, text="Description:").grid(column=0, row=0)
# create text box for description
description = tk.StringVar()
descriptionEntry = ttk.Entry(earn, width=13, textvariable=description)
descriptionEntry.grid(column=1, row=0)

# create label for earning
eLabel = ttk.Label(earn, text="Earning:").grid(column=2, row=0)
# create text box for earning
earning = tk.StringVar()
earningEntry = ttk.Entry(earn, width=13, textvariable=earning)
earningEntry.grid(column=3, row=0)
# create the action button
action = ttk.Button(earn, text="submit", command=submit)
action.grid(column=5, row=0)

win.resizable(0,0)

sub_mit = Input(action)
sub_mit.setting()

win.mainloop()

import sqlite3

class Input:
    def __init__(self, action):
        self.action = action

    def setting(self):
        self.action["state"] = "disabled"
        conn = sqlite3.connect('daily_earning.db')
        print("Opened database successfully")
        try:
            conn.execute('''CREATE TABLE DAILY_EARNING_CHART
                 (ID INTEGER PRIMARY KEY AUTOINCREMENT,
                 DESCRIPTION    TEXT (50)   NOT NULL,
                 EARNING    TEXT  NOT NULL,
                 TIME   TEXT NOT NULL);''')
        except:
            pass

        self.action["state"] = "enable"

        conn.close()

    def submit(self,description, earning): # Insert values into earning table
        self.action["state"] = "disabled"
        self.description = description
        self.earning = earning
        try:
            sqliteConnection = sqlite3.connect('daily_earning.db')
            cursor = sqliteConnection.cursor()
            print("Successfully Connected to SQLite")

            sqlite_insert_query = "INSERT INTO DAILY_EARNING_CHART (DESCRIPTION,EARNING,TIME) VALUES ('" + self.description + "','"+ self.earning + "',datetime('now', 'localtime'))"

            count = cursor.execute(sqlite_insert_query)
            sqliteConnection.commit()
            print("Record inserted successfully into DAILY_EARNING_CHART table", cursor.rowcount)
            cursor.close()

        except sqlite3.Error as error:
            print("Failed to insert earning data into sqlite table", error)
        finally:
            if (sqliteConnection):
                sqliteConnection.close()
            self.action["state"] = "enable"

The python program works alright!

More shoes have been sold!

That is all, in the next chapter we will include more features into the above python class file!