The week in numbers: 7 closed issues, 1 opened. 47 closed PRs, 4 more reviewed.

This week was almost entirely focused on iOS build support for CPython. I’m writing a blog post on my adventures with this. I also spent some time with the CLA bot. In terms of pull requests, I barely closed 13.

Overview

In a previous article, I talked about Jira, and if you are a Jira user, chances are that you also use Confluence as they often go hand in hand, Jira as a ticketing application and Confluence as an enterprise knowledge management system.

From the perspective of integrations, connecting to Confluence and invoking its APIs looks and feels practically the same as with Jira:

• You need an API token
• You fill out a form in Zato Dashboard
• You create a Python service that offers methods such as get_page_by_title, attach_file, update_page_property and similar.

Let’s go through it all step-by-step, starting off with the creation of an API token.

Creating an Atlassian API token

To invoke Confluence, you use an API token that can be shared with other Atlassian products, such as Jira. If you do not have one already, here is how to create it:

• Log in to Confluence or Jira
• Visit the address where API tokens can be managed: https://id.atlassian.com/manage-profile/security/api-tokens
• Click “Create API Token” and provide a name for the token, such as “Zato Integrations”
• Copy the token somewhere as, once it has been created, you will not be able to retrieve it later on. The only way to change a token is to revoke it and create a new one.

Creating a Confluence connection

In your Zato Dashboard, go to Cloud -> Atlassian -> Confluence:

Click “Create a new connection” and fill out the form below. The username is the same as the email address that you log in to Confluence or Jira with.

Now, click “Change API Token” and enter the token created in the previous section:

Invoking Confluence

Authoring a Zato service that invokes Confluence follows a pattern that will feel familiar no matter what kind of an API you integrate with:

• Obtain a connection to remote resource
• Invoke it
• Process the response the resource returned

In the case of the code below, we are merely logging the response from Confluence. In a bigger integration, we would process it accordingly, e.g. parts of the output could be synchronized with Jira or another system.

Note the ‘client.get_all_pages_from_space’ method below - the client will offer other methods as well, e.g. get_space, get_page_as_pdf or ways to run CQL (Confluence Query Language) directly. Use auto-completion in your IDE to discover all the methods available.

# -*- coding: utf-8 -*-# Zatofrom zato.common.typing_ import cast_
from zato.server.service import Service
# ###########################################################################if0:
from zato.server.connection.confluence_ import ConfluenceClient
# ###########################################################################classGetAllPages(Service):
defhandle(self):
# Name of the Confluence space that our pages are in        space ='ABC'# Name of the connection definition to use        conn_name ='My Confluence Connection'# .. create a reference to our connection definition ..        confluence = self.cloud.confluence[conn_name]
# .. obtain a client to Confluence ..with confluence.conn.client() as client: # type: ConfluenceClient# Cast to enable code completion            client = cast_('ConfluenceClient', client)
# Get all pages from our space            pages = client.get_all_pages_from_space(space)
            self.logger.info('Pages received ->%s', pages)
# ###########################################################################

That is all - you have create an Atlassian API token, a Zato Confluence connection and you have integrated with Confluence in Python!

Next steps

• Start the tutorial to learn how to integrate APIs and build systems. After completing it, you will have a multi-protocol service representing a sample scenario often seen in banking systems with several applications cooperating to provide a single and consistent API to its callers.

• Visit the support page if you need assistance.

• Para aprender más sobre las integraciones de Zato y API en español, haga clic aquí.

• Pour en savoir plus sur les intégrations API avec Zato en français, cliquez ici.

A common issue programmers have when they try to test real-world software is how to deal with third-party dependencies. Let’s examine an old, but counter-intuitive principle.

Django 4.1 beta 1 is now available. It represents the second stage in the 4.1 release cycle and is an opportunity for you to try out the changes coming in Django 4.1.

Django 4.1 has an profusion of new features which you can read about in the in-development 4.1 release notes.

Only bugs in new features and regressions from earlier versions of Django will be fixed between now and 4.1 final (also, translations will be updated following the "string freeze" when the release candidate is issued). The current release schedule calls for a release candidate in a month from now with the final release to follow about two weeks after that around August 3. Early and often testing from the community will help minimize the number of bugs in the release. Updates on the release schedule schedule are available on the django-developers mailing list.

As with all alpha and beta packages, this is not for production use. But if you'd like to take some of the new features for a spin, or to help find and fix bugs (which should be reported to the issue tracker), you can grab a copy of the beta package from our downloads page or on PyPI.

The PGP key ID used for this release is Carlton Gibson: E17DF5C82B4F9D00.

Introduction

Python has become an all-purpose language. It is especially commonly used in analytics and solving algorithmic problems within data science but is also popular in web development. This combination makes it a reasonable choice for various extract-transform-load (ETL) tasks.

However, many of these tasks are rather small and don’t require large frameworks such as Airflow or Luigi. When polling one or several web pages for data, a simple python script plus crontab is more than sufficient. Still, when a project gets a little bigger, managing multiple jobs using cron may become cumbersome. At the same time, bare installation of Airflow for “small jobs” needs at least 4GB RAM and 2 CPUs (here). Thinking about AWS costs, it is at least a t2.small instance running at all times.

Is there anything in between? Small enough to use, say t2.nano (very cheap) and fairly “maintainable” and “extendable”?

In this post, I would like to share with you a simple approach that uses python’s schedule package with a few modifications.

Python scheduler

Python schedule library offers simple task scheduling. It is installable using pip, and fairly easy to use. Unfortunately, the documentation doesn’t provide examples of using it within a larger project:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
importscheduleimporttimedefjob():print("I'm working...")# Run job every 3 second/minute/hour/day/week,# Starting 3 second/minute/hour/day/week from nowschedule.every(3).seconds.do(job)schedule.every(3).minutes.do(job)schedule.every(3).hours.do(job)schedule.every(3).days.do(job)schedule.every(3).weeks.do(job)# Run job every minute at the 23rd secondschedule.every().minute.at(":23").do(job)# Run job every hour at the 42rd minuteschedule.every().hour.at(":42").do(job)# Run jobs every 5th hour, 20 minutes and 30 seconds in.# If current time is 02:00, first execution is at 06:20:30schedule.every(5).hours.at("20:30").do(job)# Run job every day at specific HH:MM and next HH:MM:SSschedule.every().day.at("10:30").do(job)schedule.every().day.at("10:30:42").do(job)# Run job on a specific day of the weekschedule.every().monday.do(job)schedule.every().wednesday.at("13:15").do(job)schedule.every().minute.at(":17").do(job)whileTrue:schedule.run_pending()time.sleep(1)

As you can see, all functions are called at the level of the module, which is OK for placing it in a script. However, if you have several different jobs, the code quickly becomes cluttered, especially if different callables require different parameters.

In other words, it may preferable to take advantage of the object-oriented approach and define some “architecture” around it.

Using it in a project

Let’s say, for the sake of an argument, that we have a set of dedicated ETL tasks, modeled using the following abstract class:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
fromabcimportABC,abstractmethodfromtypingimportAny,Dict,TypeVarE=TypeVar("ETL")classBaseETL(ABC):def__init__(self,**kwargs:Dict)->None:self.raw_data=Noneself.transformed_data=None@abstractmethoddefextract(self,**kwargs:Dict)->E:...@abstractmethoddeftransform(self,**kwargs:Dict)->E:...@abstractmethoddefload(self,**kwargs:Dict)->Any:...defrun(self,**kwargs:Dict)->None:self.extract(**kwargs).transform(**kwargs).load(**kwargs)

Any class that would implement an ETL process would inherit from this base class. The extract method could, for example, fetch a website. Then transform would transform the raw HTML into a format acceptable by a database. Finally, the load would save the data to the database. All methods, executed in this order can be wrapped using the run method.

Now, after the ETL classes are defined, we would like to schedule each of them through the schedule module in a nice fashion.

Two example ETL tasks

For brevity, in the following examples, let’s skip the inheritance and only focus on the run method. Assume, that their extract, transform and load methods are implemented elsewhere.

etl.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
classDummyETL:# normally DummyETL(BaseETL)def__init__(self,init_param:int)->None:# super().__init__()  # - not needed hereself.init_param=init_paramdefrun(self,p1:int,p2:int)->None:print(f"{self.__class__.__name__}({self.init_param}, p1={p1}, p2={p1})")classEvenDummierETL:# same...def__init__(self,init_param:int)->None:# super().__init__()  # - sameself.init_param=init_paramdefrun(self,p1:int)->None:print(f"{self.__class__.__name__}({self.init_param}, p1={p1})")

The constructors’ parameters can, for instance, specify the URLs of the pages for scraping. The run methods’ parameters, for a change, can be used to pass secrets.

Now, that we have the ETL classes defined, let’s create a separate registry to associate the processes with some sort of schedule.

registry.py

1
2
3
4
5
6
7
8
9
10
11
12
13
importschedulefrometlimportDummyETL,EvenDummierETLdefget_registry():dummy_etl=DummyETL(init_param=13)dummier_etl=EvenDummierETL(init_param=15)return[(dummy_etl,schedule.every(1).seconds),(dummier_etl,schedule.every(1).minutes.at(":05")),]

The get_registry function is a place to define the schedule. Although the parameters’ values are hard-coded, you can think of a situation where the function loads them from a config file. Either way, it returns a list of tuples that matches the ETL objects with Jobs (from schedule). Note that this is our convention. The jobs are not yet associated with any particular Scheduler (again, from schedule). However, the convention allows us to do so in any other part of the project. We don’t have to bind them with the module-level object, as shown in the documentation example.

Our scheduler-based scheduler

Finally, let’s create a new class that will activate the whole mechanism.

scheduler.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
importtimefromtypingimportDict,List,Tuple,TypeVarfromscheduleimportJob,SchedulerfrometlimportDummyETL,EvenDummierETLfrometlimportE# we could do so from e.g. etl.baseS=TypeVar("Scheduler")classTaskScheduler:def__init__(self,registry:List[Tuple[E,Job]])->None:self.scheduler=Scheduler()self.registry=[]fortask,jobinregistry:self.registry.append(task)self.scheduler.jobs.append(job)defregister(self,run_params:Dict)->S:jobs=self.scheduler.get_jobs()fortask,jobinzip(self.registry,jobs):params=run_params.get(task.__class__.__name__)job.do(task.run,**params)returnselfdefrun(self,polling_seconds:int)->None:whileTrue:time.sleep(polling_seconds)self.scheduler.run_pending()

Our TaskScheduler uses composition to create a single Scheduler instance and add previously registered jobs to it. Although not enforced, we use typing to give a strong hint on what should be provided to the constructor to properly register the jobs. Then, the register method is a separate method that provides the binding. Last, but not least, run activates the machinery.

A script that uses this implementation would look like this:

run.py

1
2
3
4
5
6
7
8
9
10
11
12
13
fromregistryimportget_registryfromschedulerimportTaskSchedulerif__name__=="__main__":run_params={"DummyETL":dict(p1=1,p2=2),# e.g. from environmental variables"EvenDummierETL":dict(p1=3),}registry=get_registry()# e.g. from script's args or config filetask_scheduler=TaskScheduler(registry).register(run_params)task_scheduler.run()

Probably the weakest point of this solution is the convention that uses the __class__.__name__ as keys in the run_params dictionary. However, considering the simplicity of the approach, it may be OK, especially if these parameters would be defined at runtime. There are many alternatives, one of which could be creating an additional abstraction layer with e.g. objects like DummyTask that would serve as a bridge between ETL objects and the registry.

Another approach to TaskScheduler

Coming back to the TaskScheduler, we can also define it through inheritance as opposed to composition (as before). That would mean expanding the functionality of the schedule’s native Scheduler class. In this case, the TaskScheduler would be the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
classTaskScheduler(Scheduler):# <- heredef__init__(self,registry:List[Tuple[E,Job]])->None:super().__init__()# <- hereself.registry=[]fortask,jobinregistry:self.registry.append(task)self.jobs.append(job)# <- heredefregister(self,run_params:Dict)->S:jobs=self.get_jobs()# <- herefortask,jobinzip(self.registry,jobs):params=run_params.get(task.__class__.__name__)job.do(task.run,**params)returnselfdefrun(self,polling_seconds:int)->None:whileTrue:time.sleep(polling_seconds)self.run_pending()# <- and here

You decide which way is better if any ;).

Conclusion

In this brief article, we have shown how the simple schedule module can be expanded to create a small ETL working machine. Most importantly, the approach allows to better organize the code within a small project without having to fetch the big cannons.

PyPy is a fast, compliant alternative implementation of Python.
cPython is implemented in C.

PyPy is implemented in Python.
What does that mean?

And how do you test something as huge as an alternative implementation of Python?

Special Guest: Carl Friedrich Bolz-Tereick.

Sponsored By:

• Rollbar: With Rollbar, developers deploy better software faster.

Links:

• PyPy
• How is PyPy Tested?
• PyPy Speed
• Python Speed Center

The Zen of Python has a lot of wisdom to offer. One especially useful idea is that “There should be one—and preferably only one—obvious way to do it.” Yet there are multiple ways to do most things in Python, and often for good reason. For example, there are multiple ways to read a file in Python, including the rarely used mmap module.

Python’s mmap provides memory-mapped file input and output (I/O). It allows you to take advantage of lower-level operating system functionality to read files as if they were one large string or array. This can provide significant performance improvements in code that requires a lot of file I/O.

In this video course, you’ll learn:

• What kinds of computer memory exist
• What problems you can solve with mmap
• How use memory mapping to read large files faster
• How to change a portion of a file without rewriting the entire file
• How to use mmap to share information between multiple processes

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python’s AsyncIO web ecosystem continues to mature, but should you build your next production application with one of these shiny new frameworks such as FastAPI, Starlette, or Quart?

• A Brief History of Python Web Server Interfaces
• AsyncIO Package Ecosystem Overview
  • Servers
  • Frameworks
• Benefits of Using AsyncIO
• Obstacles to Using AsyncIO
• My Answer

A Brief History of Python Web Server Interfaces

Prior to PEP 333, Python web application frameworks, such as Zope, Quixote, and Twisted Web would each be written against specific web server APIs such as CGI or mod_python. PEP 333 created the v1.0 implementation of WSGI to give Python a standard similar to Java’s servlet API. WSGI created a common interface that promoted interchangeability between web frameworks and web servers. For example, Frameworks such as Django, Flask, and Pyramid are compatible with servers such as uWSGI and gunicorn.

Between 2003, when PEP 333 was published, and now, in 2022, WSGI has seen near universal adoption with all popular Python web frameworks and servers using WSGI. Since 2003, web application development and network protocols have evolved and changed. The RFC for the WebSocket protocol was finalized in 2013. The release of Python 3.6, in 2016, added the async and await keywords, which enable non-blocking I/O. WSGI accepts a request and returns a response, which doesn’t work for newer protocols such as WebSocket. ASGI continues the legacy of WSGI by allowing the same interchangeability in the new world of asynchronous Python. ASGI is a fundamental redesign of WSGI that enables support for newer protocols such as HTTP/2, HTTP/3, and WebSocket.

AsyncIO Package Ecosystem Overview

Servers

Uvicorn

Uvicorn supports the HTTP and WebSocket protocols. Encode OSS, the company started by Tom Christie, the creator of Django REST Framework, maintains Uvicorn. Uvicorn tends to be close to the top in popular performance benchmarks due to its use of uvloop, an alternative event loop, and httptools, Python bindings for the NodeJS HTTP parser.

Uvicorn implements a gunicorn worker, that allows you to turn gunicorn into an ASGI server. Gunicorn is a fantastic, battle-tested process manager and WSGI server for Python, and combining it with uvicorn gives you one of the best ASGI servers. Uvicorn also implements an alternative gunicorn worker with support for PyPy.

Code Example

Taken from the Uvicorn docs

import uvicorn
async def app(scope, receive, send):
...
if __name__ == "__main__":
uvicorn.run("example:app", host="127.0.0.1", port=5000, log_level="info")

Hypercorn

Hypercorn supports HTTP, HTTP/2, HTTP/3 (QUIC), and WebSocket protocols and utilizes the python hyper libraries and uvloop. Initially, hypercorn was a part of the Quart web framework but transitioned to a standalone ASGI server of quart. Hypercorn is maintained by Philip Jones, a member of the Pallets Project that maintains Flask.

Hypercorn stands out as a well maintained ASGI server that supports HTTP/2, HTTP/3, and Trio, an alternative implementation to the Python standard library’s AsyncIO package. As opposed to uvicorn, hypercorn works similar to gunicorn and operates as a process manager.

Code Example

Taken from the Hypercorn docs

import asyncio
from hypercorn.asyncio import serve
from hypercorn.config import Config
config = Config()
config.bind = ["localhost:8080"]
async def app():
...
asyncio.run(serve(app, config))

Frameworks

Starlette

Starlette can be used as a web framework or a toolkit and supports WebSocket, GraphQL, HTTP server push, in-process background tasks, and more. It falls somewhere in the middle between Django’s batteries include approach and Flask’s minimalism. Starlette is also maintained by Encode OSS. Starlette also tends to be near the top of popular performance benchmarks.

Starlette occupies a unique position in the current ASGI framework ecosystem since other popular projects such as FastAPI build on top of it. Tom Christie has a great track record of open source maintenance and development with Django REST Framework, and Encode has funding.

Code Example

Taken from the Starlette docs

from starlette.applications import Starlette
from starlette.responses import PlainTextResponse
from starlette.routing import (
Route,
Mount,
WebSocketRoute
)
from starlette.staticfiles import StaticFiles
def homepage(request):
return PlainTextResponse('Hello, world!')
def user_me(request):
username = "John Doe"
return PlainTextResponse('Hello, %s!' % username)
def user(request):
username = request.path_params['username']
return PlainTextResponse('Hello, %s!' % username)
async def websocket_endpoint(websocket):
await websocket.accept()
await websocket.send_text('Hello, websocket!')
await websocket.close()
def startup():
print('Ready to go')
routes = [
Route('/', homepage),
Route('/user/me', user_me),
Route('/user/{username}', user),
WebSocketRoute('/ws', websocket_endpoint),
Mount('/static', StaticFiles(directory="static")),
]
app = Starlette(
debug=True,
routes=routes,
on_startup=[startup]
)

FastAPI

FastAPI is a framework built on top of Starlette that adds Pydantic, a Python package that provides data validation and settings management using Python’s type annotations. By adding Pydantic, FastAPI endpoints validate input data and auto generate documentation. FastAPI is maintained by Sebastián Ramírez who has sponsor funding for FastAPI. By using Starlette under the hood, FastAPI’s performance is near the top of popular performance benchmarks.

As the name implies, FastAPI is intended for API applications, and this is where it excels. In the past, if you were creating a small API application, Flask would be the choice. Now, I would consider using FastAPI over Flask. The AsyncIO ecosystem isn’t as mature as the WSGI/Flask ecosystem, but FastAPI looks like one of the big future frameworks in the Python web ecosystem.

Code Example

Taken from the FastAPI docs

from typing import Union
from fastapi import FastAPI
from pydantic import BaseModel
app = FastAPI()
class Item(BaseModel):
name: str
price: float
is_offer: Union[bool, None] = None
@app.get("/")
def read_root():
return {"Hello": "World"}
@app.get("/items/{item_id}")
def read_item(item_id: int, q: Union[str, None] = None):
return {"item_id": item_id, "q": q}
@app.put("/items/{item_id}")
def update_item(item_id: int, item: Item):
return {"item_name": item.name, "item_id": item_id}

Quart

Quart bills itself as an AsyncIO reimplementation of the Flask microframework API and provides a migration guide. Additionally, some Flask extensions work with Quart. The author of Quart, Philip Jones, also maintains Hypercorn. Quart’s performance ranks lower than some of the previously mentioned frameworks. Quart stands out with its potential access to the large package ecosystem surrounding Flask.

Code Example

Taken from Quart docs

from dataclasses import dataclass
from datetime import datetime
from quart import Quart
from quart_schema import (
QuartSchema,
validate_request,
validate_response
)
app = Quart(__name__)
QuartSchema(app)
@dataclass
class TodoIn:
task: str
due: datetime | None
@dataclass
class Todo(TodoIn):
id: int
@app.post("/todos/")
@validate_request(TodoIn)
@validate_response(Todo)
async def create_todo(data: Todo) -> Todo:
return Todo(id=1, task=data.task, due=data.due)

Other Frameworks

• Sanic
  • Supports ASGI, WebSocket, and background tasks
• BlackSheep
  • Supports ASGI, WebSocket, and background tasks
• Aiohttp
  • AsyncIO Client and Server

Benefits of Using AsyncIO

• Improved Throughput
  • AsyncIO does not usually improve latency for I/O requests. In fact, if you look at the “latency” tab of the previously linked performance benchmark, AsyncIO frameworks perform worse in latency than Django and Flask. However, AsyncIO improves the throughput of your application, meaning the same server hardware can handle more requests per second using AsyncIO.
• Availability of New Protocols
  • Due to the limitations of WSGI, it doesn’t support newer protocols such as HTTP/2, HTTP/3 and WebSocket. With AsyncIO, Python web servers and frameworks can support these newer protocols.

Obstacles to Using AsyncIO

• Django and Flask’s Package Ecosystems
  • Large third-party package ecosystems developed around both Django and Flask. Taping into these ecosystems saves you development time. The same ecosystem has not yet developed around AsyncIO frameworks.
• Synchronous I/O Blocks AsyncIO
  • Let’s say I’m building a SaaS application that needs to accept payments. I choose Stripe, but Stripe’s python library doesn’t support AsyncIO. If I use Stripe’s library, whenever I make an API request to Stripe, it blocks the event loop, and you lose the benefits of AsyncIO.

My Answer

While Python’s AsyncIO web ecosystem has come a long way, I still choose Flask or Django for production applications that don’t require newer protocols such as HTTP/2, HTTP/3 and WebSocket. Django and Flask each have a robust set of third-party packages to help you quickly build your application. On top of that, Django and Flask are both mature, production-tested frameworks used by a lot of companies. I think the new AsyncIO frameworks will reach that point as well, but they’re still too new. If you’re looking at using the AsyncIO ecosystem, your first question should be “Do I need it?” My guess is most people don’t. WSGI servers and frameworks are usually performant enough.

If supporting HTTP/2 or HTTP/3 is a requirement, you don’t have much choice on the server side. I recommend Hypercorn. If you aren’t required to support HTTP/2 or HTTP/3, then Uvicorn becomes my preferred server option. When it comes to frameworks, I would reach for FastAPI for small-API focused applications. Tom Christie has a great track record of open source maintenance with Django REST Framework, and FastAPI adds nice additions on top of Starlette. If you’re building a larger application with some HTML template rendering, I would choose Starlette over FastAPI for the flexibility. If you need to migrate an existing Flask project to enable newer protocols, Quart is the clear choice.

I want to be clear that I think ASGI and AsyncIO are great developments for Python, and I applaud all the people putting work into the ecosystem. I think the ecosystem will continue to grow and mature, but Django and Flask are my choice right now.

#530 – JUNE 21, 2022
View in Browser »

ButterCMS Melts Into Your Python app. Try the #1 Headless CMS for Python and Django

ButterCMS is your content backend. Enable your marketing team to update website + app content without bothering you. Try the #1 rated Headless CMS for Python today. Free for 30 days →
BUTTERCMSsponsor

Discussions

Python Jobs

Senior Python Software Engineer (San Francisco, CA, USA)

MyOme

Senior Full-Stack Web Developer (Anywhere)

MonetizeMore

Backend Software Developer (Draper, UT, USA)

Canopy

Gameful Learning Developer (Ann Arbor, MI, USA)

University of Michigan

Python Technical Architect (USA)

Blenderbox

Software Engineer (Los Angeles or Dallas) (Los Angeles, CA, USA)

Causeway Capital Management LLC

DevOps Engineer (Ann Arbor, MI, USA)

University of Michigan

Academic Innovation Developer (Ann Arbor, MI, USA)

University of Michigan

Software Development Lead (Ann Arbor, MI, USA)

University of Michigan

Principal Python Engineer (100% Remote) (San Francisco, CA, USA)

Nira

More Python Jobs >>>

Articles & Tutorials

Connect, Integrate, & Automate Your Data—From Python, or Any Other Application or Tool

CData makes it easier to unlock the value of data — simplifying connectivity between applications and data sources. Our SQL-based connectors streamline data access making it easy to access real-time data from on-premise and cloud databases, SaaS, APIs, NoSQL and more. Visit cdata.com to learn more →
CDATA SOFTWAREsponsor

Projects & Code

Events

Happy Pythoning!
This was PyCoder’s Weekly Issue #530.
View in Browser »

[ Subscribe to 🐍 PyCoder’s Weekly 💌 – Get the best Python news, articles, and tutorials delivered to your inbox once a week >> Click here to learn more ]

CSV files are one of the most efficient tools to store structured, tabular data. Sometimes, we might need to append data to the CSV file from a python dictionary. In this article, we will discuss how we can append the values in a dictionary to a CSV file in python. 

Append Dictionary to CSV File Using csv.writer()

You can append a dictionary to CSV file using CSV.writer() method and CSV.writerow() method. For this, we will first open the CSV file in append mode using the open() function. The open() function takes the filename as its first input argument and the literal “a” as its second input argument to denote that the file is opened in the append mode.

After opening the file, we will create a CSV writer object using the CSV.writer() function. The CSV.writer() function takes the file object containing the CSV file as its input argument and returns a writer object. 

After creating the writer object, we will append the dictionary to the CSV file using the writerow() method. The writerow() method, when invoked on a writer object, takes the values in the dictionary as its input argument and appends it to the CSV file.

After execution of the writerow() method, you must close the CSV file using the close() method. Otherwise, the changes won’t be saved in the CSV file. The source code for this approach to append a dictionary to a CSV file is given below.

import csv

myFile = open('Demo.csv', 'r+')
print("The content of the csv file before appending is:")
print(myFile.read())
myDict = {'Roll': 4, 'Name': 'Joel', 'Language': 'Golang'}
print("The dictionary is:")
print(myDict)
writer = csv.writer(myFile)
writer.writerow(myDict.values())
myFile.close()
myFile = open('Demo.csv', 'r')
print("The content of the csv file after appending is:")
print(myFile.read())

Output:

The content of the csv file before appending is:
Roll,Name,Language
1,Aditya,Python
2,Sam, Java
3, Chris, C++

The dictionary is:
{'Roll': 4, 'Name': 'Joel', 'Language': 'Golang'}
The content of the csv file after appending is:
Roll,Name,Language
1,Aditya,Python
2,Sam, Java
3, Chris, C++
4,Joel,Golang

Append Dictionary to CSV File using csv.DictWriter()

Instead of using the csv.writer() method, we can use the csv.DictWriter() function along with the csv.writerow() method to append a python dictionary to csv file. The approach is almost similar to the approach using the csv.writer() method with the following differences.

• Instead of the csv.writer() method, we will use the csv.DictWriter() method. The DictWriter() method takes the file object containing the csv file as its input argument and returns a DictWriter object.
• When the writerow() method is executed on the DictWriter object, it takes a dictionary as the input argument instead of the values in the dictionary.

The python code to append a dictionary to a csv file using csv.DictWriter() is as follows.

import csv

myFile = open('Demo.csv', 'r+')
print("The content of the csv file before appending is:")
print(myFile.read())
myDict = {'Roll': 4, 'Name': 'Joel', 'Language': 'Golang'}
print("The dictionary is:")
print(myDict)
writer = csv.DictWriter(myFile, fieldnames=list(myDict.keys()))
writer.writerow(myDict)
myFile.close()
myFile = open('Demo.csv', 'r')
print("The content of the csv file after appending is:")
print(myFile.read())

Output:

The content of the csv file before appending is:
Roll,Name,Language
1,Aditya,Python
2,Sam, Java
3, Chris, C++


The dictionary is:
{'Roll': 4, 'Name': 'Joel', 'Language': 'Golang'}
The content of the csv file after appending is:
Roll,Name,Language
1,Aditya,Python
2,Sam, Java
3, Chris, C++

4,Joel,Golang

Conclusion

In this article, we have discussed two ways to append a dictionary to csv file in python. In these approaches, the dictionary will be appended irrespective of whether it has same number of items as compared to the columns in the csv file or it has the same keys as compared to the column names in the csv file. Thus it advised to make sure that the dictionary should have the same number of keys as compared to the columns in the csv file. Also You should make sure that the order of columns of the csv file should be same as the order of the keys present in the dictionary. Otherwise, the data appended to the csv file will become inconsistent and will lead to errors. 

I hope you enjoyed reading this article.To know more about dictionaries in python, you can read this article on dictionary comprehension in python. You might also write this article on list comprehension in python.

Stay tuned for more informative articles.

Happy Learning!

The post Append Dictionary to CSV File in Python appeared first on PythonForBeginners.com.

PS> python-mvenvvenvPS> .\venv\Scripts\activate(venv)PS> python-mpipinstallpytest

$ python -m venv venv
$ source venv/bin/activate
(venv)$ python -m pip install pytest

# test_with_unittest.pyfromunittestimportTestCaseclassTryTesting(TestCase):deftest_always_passes(self):self.assertTrue(True)deftest_always_fails(self):self.assertTrue(False)

(venv)$ python -m unittest discover
F.======================================================================FAIL: test_always_fails (test_with_unittest.TryTesting)----------------------------------------------------------------------Traceback (most recent call last):  File "...\effective-python-testing-with-pytest\test_with_unittest.py",  line 10, in test_always_fails    self.assertTrue(False)AssertionError: False is not true----------------------------------------------------------------------Ran 2 tests in 0.006sFAILED (failures=1)

Read the full article at https://realpython.com/pytest-python-testing/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python's match-case blocks are complex structural pattern matching tools that are often more hassle than they're worth. But they're great for parsing abstract syntax trees.

Read the full article: https://www.pythonmorsels.com/match-case-parsing-python/

We just released PyCharm 2022.1.3, which provides fixes for the important regressions that appeared in 2022.1 and 2022.1.1.

Please note: If you already have PyCharm 2022.1.2 installed, you will need to update to PyCharm 2022.1.3 manually via the Toolbox App or the website.

Download PyCharm 2022.1.3!

Here’s what we’ve done:

• Fixed a regression with the debug console that was truncating outputs [PY-53983].
• Fixed a regression causing .pth files to be ignored inside venv site-packages [PY-54321].
• Cmd+Click / Ctrl+Click in the Git Log panel has been fixed and now works as expected [IDEA-292405].
• The New Project button on the Welcome screen is once again working as intended [IDEA-276553].
• Fixed the issue causing a misleading error message when using $var in the calc() function in SCSS files [WEB-54056].
• Fixed the issue causing an Unexpected term error when using a variable in a min() and max() arguments list [WEB-52057].

For the full list of bug fixes, please take a look at the release notes. If you notice any bugs, please submit them to our issue tracker.

The PyCharm team

The AttributeError: ‘int’ object has no attribute ‘get’ mainly occurs when you try to call the get() method on the integer type. The attribute get() method is present in the dictionary and must be called on the dictionary data type.

In this tutorial, we will look at what exactly is AttributeError: ‘int’ object has no attribute ‘get’ and how to resolve this error with examples.

What is AttributeError: ‘int’ object has no attribute ‘get’?

If we call the get() method on the integer data type, Python will raise an AttributeError: ‘int’ object has no attribute ‘get’. The error can also happen if you have a method which returns an integer instead of a dictionary.

Let us take a simple example to reproduce this error.

# Method return integer instead of dict
def fetch_data():
    output = 100
    return output


data = fetch_data()
print(data.get("name"))

Output

AttributeError: 'int' object has no attribute 'get'

In the above example, we have a method fetch_data() which returns an integer instead of a dictionary.

Since we call the get() method on the integer type, we get AttributeError.

We can also check if the variable type using the type() method, and using the dir() method, we can also print the list of all the attributes of a given object.

# Method return integer instead of dict
def fetch_data():
    output = 100
    return output


data = fetch_data()
print("The type of the object is ", type(data))
print("List of valid attributes in this object is ", dir(data))

Output

The type of the object is  <class 'int'>

List of valid attributes in this object is  ['__abs__', '__add__', '__and__', '__bool__', '__ceil__', '__class__', '__delattr__', '__dir__', '__divmod__', '__doc__', '__eq__', '__float__', '__floor__', '__floordiv__', '__format__', '__ge__', '__getattribute__', '__getnewargs__', '__gt__', '__hash__', '__index__', '__init__', '__init_subclass__', '__int__', '__invert__', '__le__', '__lshift__', '__lt__', '__mod__', '__mul__', '__ne__', '__neg__', '__new__', '__or__', '__pos__', '__pow__', '__radd__', '__rand__', '__rdivmod__', '__reduce__', '__reduce_ex__', '__repr__', '__rfloordiv__', '__rlshift__', '__rmod__', '__rmul__', '__ror__', '__round__', '__rpow__', '__rrshift__', '__rshift__', '__rsub__', '__rtruediv__', '__rxor__', '__setattr__', '__sizeof__', '__str__', '__sub__', '__subclasshook__', '__truediv__', '__trunc__', '__xor__', 'as_integer_ratio', 'bit_count', 'bit_length', 'conjugate', 'denominator', 'from_bytes', 'imag', 'numerator', 'real', 'to_bytes']

How to fix AttributeError: ‘int’ object has no attribute ‘get’?

Let us see how we can resolve the error.

Solution 1 – Call the get() method on valid dictionary

We can resolve the error by calling the get() method on the valid dictionary object instead of the integer type.

The dict.get() method returns the value of the given key. The get() method will not throw KeyError if the key is not present; instead, we get the None value or the default value that we pass in the get() method.

# Method returns dict
def fetch_data():
    output = {"Name": "Audi", "Release_Date": "2022", "Price": "$45000"}
    return output

data = fetch_data()

# Get the price of the car
print(data.get("Price"))

Output

$45000

Solution 2 – Check if the object is of type dictionary using type

Another way is to check if the object is of type dictionary; we can do that using the type() method. This way, we can check if the object is of the correct data type before calling the get() method.

# Method returns dict
def fetch_data():
    output = {"Name": "Audi", "Release_Date": "2022", "Price": "$45000"}
    return output


data = fetch_data()

# Check if the object is dict
if (type(data) == dict):
    print(data.get("Price"))

Output

$45000

Solution 3 – Check if the object has get attribute using hasattr

Before calling the get() method, we can also check if the object has a certain attribute. Even if we call an external API which returns different data, using the hasattr() method, we can check if the object has an attribute with the given name.

# Method returns dict
def fetch_data():
    output = {"Name": "Audi", "Release_Date": "2022", "Price": "$45000"}
    return output


data = fetch_data()

# Check if the object has get attribute
if (hasattr(data, 'get')):
    print(data.get("Price"))

Output

$45000

Conclusion

The AttributeError: ‘int’ object has no attribute ‘get’ occurs when you try to call the get() method on the integer type. The error also occurs if the calling method returns an integer instead of a dictionary object.

We can resolve the error by calling the get() method on the dictionary object instead of an integer. We can check if the object is of type dictionary using the type() method, and also, we can check if the object has a valid get attribute using hasattr() before performing the get operation.

The AttributeError: ‘list’ object has no attribute ‘get’ mainly occurs when you try to call the get() method on the list data type. The attribute get() method is present in the dictionary and must be called on the dictionary data type.

In this tutorial, we will look at what exactly is AttributeError: ‘list’ object has no attribute ‘get’ and how to resolve this error with examples.

What is AttributeError: ‘list’ object has no attribute ‘get’?

If we call the get() method on the list data type, Python will raise an AttributeError: ‘list’ object has no attribute ‘get’. The error can also happen if you have a method which returns an list instead of a dictionary.

Let us take a simple example to reproduce this error.

# Method return list of dict 
def fetch_data():
    cars = [
        {'name': 'Audi', 'price': 45000},
        {'name': 'Ferrari', 'price': 450000},
        {'name': 'BMW', 'price': 55000},
    ]
    return cars


data = fetch_data()
print(data.get("name"))

Output

AttributeError: 'list' object has no attribute 'get'

In the above example, we have a method fetch_data() which returns an list of dictionary object instead of a dictionary.

Since we call the get() method directly on the list type, we get AttributeError.

We can also check if the variable type using the type() method, and using the dir() method, we can also print the list of all the attributes of a given object.

# Method return list of dict 
def fetch_data():
    cars = [
        {'name': 'Audi', 'price': 45000},
        {'name': 'Ferrari', 'price': 450000},
        {'name': 'BMW', 'price': 55000},
    ]
    return cars

data = fetch_data()
print("The type of the object is ", type(data))
print("List of valid attributes in this object is ", dir(data))

Output

The type of the object is  <class 'list'>

List of valid attributes in this object is  ['__add__', '__class__', '__class_getitem__', '__contains__', '__delattr__', '__delitem__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__gt__', '__hash__', '__iadd__', '__imul__', '__init__', '__init_subclass__', '__iter__', '__le__', '__len__', '__lt__', '__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__reversed__', '__rmul__', '__setattr__', '__setitem__', '__sizeof__', '__str__', '__subclasshook__', 'append', 'clear', 'copy', 'count', 'extend', 'index', 'insert', 'pop', 'remove', 'reverse', 'sort']

How to fix AttributeError: ‘list’ object has no attribute ‘get’?

Let us see how we can resolve the error.

Solution 1 – Call the get() method on valid dictionary

We can resolve the error by calling the get() method on the valid dictionary object instead of the list type.

Since the data has a valid dictionary object inside the list, we can loop through the list and use the get() method on the dictionary elements.

The dict.get() method returns the value of the given key. The get() method will not throw KeyError if the key is not present; instead, we get the None value or the default value that we pass in the get() method.

# Method return list of dict 
def fetch_data():
    cars = [
        {'name': 'Audi', 'price': 45000},
        {'name': 'Ferrari', 'price': 450000},
        {'name': 'BMW', 'price': 55000},
    ]
    return cars


data = fetch_data()
for item in data:
    print(item.get("name"))

Output

Audi
Ferrari
BMW

We can also use the generator expression, filter function to get specific dictionary element, or directly use the index of the list. Let us look at each of these with examples.

# Method return list instead of dict
def fetch_data():
    cars = [
        {'name': 'Audi', 'price': 45000},
        {'name': 'Ferrari', 'price': 450000},
        {'name': 'BMW', 'price': 55000},
    ]
    return cars


data = fetch_data()
# Generator expression to get specific element
car_obj = next(
    (x for x in data if x['name'] == 'BMW'),
    {}
)

print(car_obj)
print("Car Name is ", car_obj.get("name"))
print("Car Price is ", car_obj.get("price"))

# Directly access the dictionary using the index of list
print("The car name in index 0 is ", data[0].get("name"))

Output

{'name': 'BMW', 'price': 55000}
Car Name is  BMW
Car Price is  55000

The car name in index 0 is  Audi

Solution 2 – Check if the object is of type dictionary using type

Another way is to check if the object is of type dictionary; we can do that using the type() method. This way, we can check if the object is of the correct data type before calling the get() method.

# Method return list of dict
def fetch_data():
    cars = [
        {'name': 'Audi', 'price': 45000},
        {'name': 'Ferrari', 'price': 450000},
        {'name': 'BMW', 'price': 55000},
    ]
    return cars

# assigns the list of dict
data = fetch_data()
if (type(data) == dict):
    print(data.get("name"))
else:
    print("The object is not dictionary and it is of type ", type(data))

# assign the index 0 dict    
my_car =data[0]
if (type(my_car) == dict):
    print(my_car.get("name"))
else:
    print("The object is not dictionary and it is of type ", type(my_car))

Output

The object is not dictionary and it is of type  <class 'list'>
Audi

Solution 3 – Check if the object has get attribute using hasattr

Before calling the get() method, we can also check if the object has a certain attribute. Even if we call an external API which returns different data, using the hasattr() method, we can check if the object has an attribute with the given name.

# Method return list of dict
def fetch_data():
    cars = [
        {'name': 'Audi', 'price': 45000},
        {'name': 'Ferrari', 'price': 450000},
        {'name': 'BMW', 'price': 55000},
    ]
    return cars


# assigns the list of dict
data = fetch_data()
if (hasattr(data, 'get')):
    print(data.get("name"))
else:
    print("The object does not have get attribute")

# assign the index 0 dict
my_car = data[0]
if (hasattr(my_car, 'get')):
    print(my_car.get("name"))
else:
    print("The object does not have get attribute")

Output

The object does not have get attribute
Audi

Conclusion

The AttributeError: ‘list’ object has no attribute ‘get’ occurs when you try to call the get() method directly on the list data type. The error also occurs if the calling method returns an list instead of a dictionary object.

We can resolve the error by calling the get() method on the dictionary object by iterating the list instead of directly calling the get() method on an list. We can check if the object is of type dictionary using the type() method, and also, we can check if the object has a valid get attribute using hasattr() before performing the get operation.

The AttributeError: ‘float’ object has no attribute ‘get’ mainly occurs when you try to call the get() method on the float data type. The attribute get() method is present in the dictionary and must be called on the dictionary data type.

In this tutorial, we will look at what exactly is AttributeError: ‘float’ object has no attribute ‘get’ and how to resolve this error with examples.

What is AttributeError: ‘float’ object has no attribute ‘get’?

If we call the get() method on the float data type, Python will raise an AttributeError: ‘float’ object has no attribute ‘get’. The error can also happen if you have a method which returns an float instead of a dictionary.

Let us take a simple example to reproduce this error.

# Method return float instead of dict
def fetch_data():
    output = 44.55
    return output


data = fetch_data()
print(data.get("price"))

Output

AttributeError: 'float' object has no attribute 'get'

In the above example, we have a method fetch_data() which returns an float instead of a dictionary.

Since we call the get() method on the float type, we get AttributeError.

We can also check if the variable type using the type() method, and using the dir() method, we can also print the list of all the attributes of a given object.

# Method return float instead of dict
def fetch_data():
    output = "Toyota Car"
    return output


data = fetch_data()
print("The type of the object is ", type(data))
print("List of valid attributes in this object is ", dir(data))

Output

The type of the object is  <class 'float'>

List of valid attributes in this object is  ['__abs__', '__add__', '__bool__', '__ceil__', '__class__', '__delattr__', '__dir__', '__divmod__', '__doc__', '__eq__', '__float__', '__floor__', '__floordiv__', '__format__', '__ge__', '__getattribute__', '__getformat__', '__getnewargs__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__int__', '__le__', '__lt__', '__mod__', '__mul__', '__ne__', '__neg__', '__new__', '__pos__', '__pow__', '__radd__', '__rdivmod__', '__reduce__', '__reduce_ex__', '__repr__', '__rfloordiv__', '__rmod__', '__rmul__', '__round__', '__rpow__', '__rsub__', '__rtruediv__', '__set_format__', '__setattr__', '__sizeof__', '__str__', '__sub__', '__subclasshook__', '__truediv__', '__trunc__', 'as_integer_ratio', 'conjugate', 'fromhex', 'hex', 'imag', 'is_integer', 'real']

How to fix AttributeError: ‘float’ object has no attribute ‘get’?

Let us see how we can resolve the error.

Solution 1 – Call the get() method on valid dictionary

We can resolve the error by calling the get() method on the valid dictionary object instead of the float type.

The dict.get() method returns the value of the given key. The get() method will not throw KeyError if the key is not present; instead, we get the None value or the default value that we pass in the get() method.

# Method returns dict
def fetch_data():
    output = {"Name": "NordVPN",  "Price": "22.22"}
    return output


data = fetch_data()

# Get the Price
print(data.get("Price"))

Output

22.22

Solution 2 – Check if the object is of type dictionary using type

Another way is to check if the object is of type dictionary; we can do that using the type() method. This way, we can check if the object is of the correct data type before calling the get() method.

# Method returns dict
def fetch_data():
    output = {"Name": "NordVPN",  "Price": "22.22"}
    return output


data = fetch_data()

# Get the Price
if (type(data) == dict):
    print(data.get("Price"))
else:
    print("The object is not dictionary and it is of type ", type(data))

Output

22.22

Solution 3 – Check if the object has get attribute using hasattr

Before calling the get() method, we can also check if the object has a certain attribute. Even if we call an external API which returns different data, using the hasattr() method, we can check if the object has an attribute with the given name.

# Method returns dict
def fetch_data():
    output = {"Name": "NordVPN",  "Price": "22.22"}
    return output


data = fetch_data()

# Get the Price
if (hasattr(data, 'get')):
    print(data.get("Price"))
else:
    print("The object does not have get attribute")

Output

22.22

Conclusion

The AttributeError: ‘float’ object has no attribute ‘get’ occurs when you try to call the get() method on the floatdata type. The error also occurs if the calling method returns an float instead of a dictionary object.

We can resolve the error by calling the get() method on the dictionary object instead of an float. We can check if the object is of type dictionary using the type() method, and also, we can check if the object has a valid get attribute using hasattr() before performing the get operation.