@ced wrote:

adventure-backlit-cool-wallpaper-357891.jpg1280×847 77.4 KB

We are proud to announce the 5.4 release of Tryton.
In addition to my bug fixes and performance improvements, this release improves in many place the user experience. It also extends a lot the existing workflow to support more use cases. We see 8 new modules landing as official.

You can have a try it on the demo server, use the docker image or download it here.
As usual the migration from previous series is fully supported. Some manual operation may be required, see Migration from 5.2 to 5.4.

Here is the list of the most noticeable changes:
(For a more complete list, see the change log of each package)

Contents

• Changes for users
  • Desktop Client
  • Web Client
  • Accounting
  • Bank
  • Party
  • Product
  • Purchase
  • Sale
  • Stock
  • Timesheet
• New modules
  • Secondary Unit
  • Amendment
  • Purchase/Sale History
• New languages
• Changes for developers
  • Accounting
  • Country/Currency
  • Party
  • Stock

Changes For The User

We can show a visual context on the rows or cells. Those visual context can be muted , success , warning or danger . Many modules have been updated to use them like the payable and receivable today on the party or when an invoice is due etc.

Screenshot_2019-08-08%20Customer%20Invoices%20-%20Tryton.png2238×985 109 KB

In the search bar of the clients, we enabled the direct search on fields of relational field types, like One2Many, Many2Many and Many2One. This is done by appending a dot to the relational field name and then the name of the field in the relation model. E.g. On products filter you can use the search clause Variants.Code: PROD, to find all products, which variants have a code named PROD.
The search entry provides completion for such related fields.
By default only one level of completion is activated but customization can activate more. This feature also works on the keys of dictionary fields like the product attributes.

sao-search-variants-code.png793×505 29.3 KB

Now the clients display a more user-friendly error messages for domain validation error. The clients display the exact failing constraint using the same format of the search bar.

All the list views have been reviewed. The most important fields are now expanded to take advantage of available space. For main editable view, the creation of new record is done on top instead of bottom to avoid to load all the records.

For now, an CSV export created by a user will be only available for him by default. The administrator can make an export available to a group of users.

Desktop Client

Now, when drag&drop is available on a view, we show a draggable icon to notify the user but also to provide a handle to drag which is easier when the list is editable.

Web Client

The web client now supports drag and drop to order list and tree rows like in the desktop client. There is one small difference to insert a row inside a non-expanded row: The user must drop it below the row while pressing the CTRL key. Otherwise the row is dropped next to the row.

sao-dnd.gif1419×780 245 KB

The column size of the web client has been improved. Now columns have a minimal width (depending of the type) and a double scrollbar (top and bottom) is displayed if there is not enough space to show all the columns on the view-port.

Accounting

The constraint that prevented to use twice the same invoice sequence per fiscal year, has been relaxed. Now Tryton checks only that the sequence was not used to number an invoice with a later date.

Since 14 September 2019, Strong Customer Authentication (SCA) has been introduced by EU regulators to reduce online fraud and make the internet a safer place to transact. For that Stripe has introduced the Setup Intent and Payment Intent mechanism for credit card payment. Tryton now supports them in addition to the former mechanism (for SEPA, SOFORT etc.).

Some financial institute have precise requirement about which initiator identifier to use the SEPA message. For that we added a configuration options on the payment journal. The available options for now are: “SEPA Creditor Identifier”, “Belgian Enterprise Number”, “Spanish VAT Number”.

Until now, it was possible to cancel a posted supplier invoice but not one from a customer. This was because in many countries it is not allowed. But in order to be more flexible, we added an option on the company to allow cancel of customer invoice.

When creating manually an accounting move, we set the date to today by default if the current period is selected. Otherwise it is still the start date of the period.

The wizard that renew automatically the fiscal year, now update the sequence name if it contains the year.

When migrating to Tryton, the accountant needs to fill depreciation of existing asset which has already been deprecated. To ease the encoding and ensure a correct computation, we added a field to store the already depreciated amount for the asset. This amount will be deducted from the asset value before continuing the depreciation computation.

To ensure that any tax line will be reported in tax statement, the tax is always required on the tax line.

It is now possible to define default values for the customer and supplier tax rules. This can be useful to apply a local tax rule based on subdivision by default.

In addition to the country, the tax rules can now be written using the subdivisions of origin and/or destination. A child subdivision will match the rule based on an upper level subdivision. This is useful for countries that have different tax rates for some subdivisions.

The income statement is included in the balance sheet for the Spanish accounting (as it is done for other countries). So the running income of the current year is already included before the year closing.

Bank

The BIC of banks are now validated and formatted. This avoid encoding error and ease the research.

Party

Until now, the subdivision on an address was limited to the top-level subdivision of the country. It is now possible to define which types of subdivision are allowed to be used. Tryton comes with some configurations following the country rules for postal address format.

Product

Now it is possible to configure a sequence for the product code that will be used to be filled at creation time. This may be used to ensure to have a unique code per product, even when it is duplicated.

Like for parties, we added on product a list of identifiers. By default, Tryton supports and validates these numbers: EAN, ISAN, ISBN, ISIL, ISIN and ISMN. Non-standard identifiers are supported also. These identifiers are used for matching when searching products by name.

The product cost price can be used in the price list. It uses the cost price of the company set in the context. This allows to build price lists by defining a margin to apply on the cost.

You can now define which unit of measure is the basis for quantities used in a price list. In standard modules we support the default unit (the original one) and the sale unit.

Purchase

It is now possible to configure the customer code of the current company on the supplier party. The code will be displayed on the request for quotation .

The same processing delay for purchases is added to the requisitions. This allows to reset an approved requisition to draft if it was not yet automatically processed.

Sale

We added the same processing delay for sales to the sale complaints. So you can reset to draft a complaint after being approved or rejected if it was not yet automatically processed.

We added an option to deactivate a subscription service. This prevents to use these service for new subscriptions.

We allow now to finish a subscription line before the next consumption. This gives more flexibility about ending subscription.

Stock

Now users are able to set a default warehouse in the preferences.
This is useful for companies with multiple warehouses. It saves time for the users as they could have the warehouse filled-in for which they work.

You can use consumable products in an inventory if needed. There are still no requirements and the inventory is not automatically filled with products of this type.

When you are looking at the evolution of the stock quantity for a product, you can open the date to see the moves involved for the changes.

When opening the graph of product quantities by warehouse, if there was no move for the current date, the user can not see the current quantity. Now we add always an entry for the current date.

We force now to always have a minimal quantity for the order point. This avoid confusion for the case where it was not set. Now if you do not want to trigger the purchase or production for any quantity, the user must set an explicit negative quantity.

Timesheet

When using the “Enter Timesheet” wizard, now we display the date in the window name (next to the employee name). The shown date is the one selected in the first step of the wizard.

New Modules

Secondary Unit

These modules allow to define a different secondary unit and factor on the product for sale and for purchase.
The quantity of sale and purchase lines can be defined using the secondary unit fields (quantity and unit price), the main unit fields are automatically updated using the product factor.
On related documents like the invoice or shipment, the secondary fields are displayed using the factor stored on the sale or purchase.

Amendment

The amendment modules allow you to change sales and purchases that are being processed and keeping track of those changes. When an amendment is validated the document is updated and given a new revision. If needed, the invoices or shipments are also updated or recreated to match the new order.

Purchase and Sale History

These modules activate the history on sales and purchases but also add a revision number which is incremented each time the document is reset to draft. The revision number is appended to the document number to ensure parties are communicating about the same version.

New Languages

• Indonesian

Changes For The Developer

It is now possible to use SQL expressions as value with the create/write methods. The main purpose is to be able to use the time functions of the database server which are linked to the transaction instead of the one provided by the Tryton server.

The expand attribute has been changed from a Boolean (1 or 0) into an integer. The integer represents the proportion of available space which is taken among all expanded columns.

The format_date method on Report can now take an optional format parameter if you don’t want to use the default format of the language.
The Report receives also a new method format_timedelta. It uses the same representation as the clients to format duration field values.

There is now an environment variable to set the default logging level when running trytond as a WSGI application.

Now we have a lazy_gettext method which allows to defer the translation by using a LazyString . It can be used as label or help text of Fields. This is useful for base Model classes and Mixins to limit the duplication of the translation of the same string for each derived class.

Now we prevent to set a value for an unknown field in proteus scripts and in Tryton modules model definitions. For that we add __slots__ automatically on each model. A positive side effect is that it reduces also the memory consumption of each instance.

The PYSON Eval now supports the dotted notation. This feature is a common expectation from beginners. So we decided it is good to support it.

We have already a multiselection widget to use with a Many2Many field. But now we have also aMultiSelection field which stores a list of value as a JSON list in the database. This is useful when the selection has a few options. For now, the widget is also available on list views (but not editable). And the field is usable in the search bar of the client.

You can now define a different start date when using PYSON Date or DateTime with delta.

Now we give the possibility to define a different order (alphabetic) to the keys of a Dict field.

Even if cron jobs are relaunch, it is better to retry directly them few times when a DatabaseOperationalError is raised. This also avoid unnecessary errors in the logs.

Missing a depends on a method is a common mistake. We have improved the generic test to catch more cases like missing or empty parent or unknown field. All the modules have been checked and corrected against this new tests.

Accounting

The generic checkout page for Stripe has been updated to use Stripe.js and support setup and payment intent.
Until now, it was required to setup a webhook from Stripe to Tryton in order to receive the events to update asynchronously the workflow of the payments. Now if you do not setup such webhook, a cron task will fetch periodically the new events and process them. This is useful for testing or when Tryton can not be reached from outside.

We require now to have a fresh session to post a statement. If it is not the case, the client will request to re-enter the user password (or any other authentication method configured).

Country/Currency

The countries, subdivisions and currencies are no more loaded from XML at the module installation but using proteus scripts which use pycountry data: trytond_import_countries and trytond_import_currencies . The translations are also loaded by those scripts.
This reduces the maintenance load of each release and allows users to keep their database up to date without relying on Tryton releases.

Party

As the countries are no more managed as XML data in Tryton (but by import script). The address format are now using country and language code instead of Many2One. So a format can be created even if the country or the language do not yet exist.

Stock

As we now keep a link between the inventory moves and the outgoing moves, we simplified the synchronization algorithm to use this link. Another advantage is that if the product is changed on the inventory move, the outgoing move is also updated instead of creating a new move.

Posts: 2

Participants: 1

Read full topic

In this course, you’ll get a look into the newest version of Python. On October 14th, 2019 the first official version of Python 3.8 became ready.

You’ll learn about the following:

• Using assignment expressions to simplify some code constructs
• Enforcing positional-only arguments in your own functions
• Specifying more precise type hints
• Using f-strings for simpler debugging

With a few exceptions, Python 3.8 contains many small improvements over the earlier versions. Towards the end of the course, you’ll see many of these less attention-grabbing changes, as well as a discussion about some of the optimizations that make Python 3.8 faster than its predecessors.

If you want to learn more, additional resources will be referenced and linked to throughout the course.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Tutorials

Talks

Hatchery

StartUp Row

Introduction

Following this article by Muhammad Junaid Khalid, where basic OpenGL concepts and setup was explained, now we'll be looking at how to make more complex objects and how to animate them.

OpenGL is very old, and you won't find many tutorials online on how to properly use it and understand it because all the top dogs are already knee-deep in new technologies.

To understand modern OpenGL code, you have to first understand the ancient concepts that were written on stone tablets by the wise Mayan game developers.

In this article, we'll jump into several fundamental topics you'll need to know:

• Basic Matrix Operations
• Composite Transformations
• Transformations that Involve a Referral Point
• Modeling Demonstration

In the last section we'll take a look at how to actually use OpenGL with the Python libraries PyGame and PyOpenGL.

In the next article (coming soon!) we'll take a deeper look at how to use OpenGL with Python and the libraries mentioned above.

Basic Matrix Operations

To properly be able to use many of the functions in OpenGL, we'll need some geometry.

Every single point in space can be represented with Cartesian coordinates. Coordinates represent any given point's location by defining it's X, Y and Z values.

We'll be practically using them as 1x3 matrices, or rather 3-dimensional vectors (more on matrices later on).

Here are examples of some coordinates:

a and b being points in space, their x-coordinates being 5 and 9 respectively, y-coordinates being 3 and 1, and so on.

In computer graphics, more often than not, homogeneous coordinates are utilized instead of regular old Cartesian coordinates. They're basically the same thing, only with an additional utility parameter, which for the sake of simplicity we'll say is always 1.

So if the regular coordinates of a are (5,3,4), the corresponding homogeneous coordinates would be (5,3,4,1). There's a lot of geometric theory behind this, but it isn't really necessary for this article.

Next, an essential tool for representing geometric transformations are matrices. A matrix is basically a two-dimensional array (in this case of size n*n, it's very important for them to have the same number of rows and columns).

Now matrix operations are, more often than not, pretty straightforward, like addition, subtraction, etc. But of course the most important operation has to be the most complicated one - multiplication. Let's take a look at basic matrix operation examples:

Now, as all math tends to do, it gets relatively complicated when you actually want something practical out of it.

The formula for matrix multiplication goes as follows:

$$
c[i,j] = \sum_{k=1}^{n}a[i,k]*b[k,j]
$$

c being the resulting matrix, a and b being the multiplicand and the multiplier.

There's a simple explanation for this formula, actually. Every element can be constructed by summing the products of all the elements in the i-th row and the j-th column. This is the reason why in a[i,k], the i is fixed and the k is used to iterate through the elements of the corresponding row. Same principle can be applied to b[k,j].

Knowing this, there's an additional condition that needs to be fulfilled for us to be able to use matrix multiplication. If we want to multiply matrices A and B of dimensions a*b and c*d. The number of elements in a single row in the first matrix (b) has to be the same as the number of elements in a column in the second matrix (c), so that the formula above can be used properly.

A very good way of visualizing this concept is highlighting the rows and columns who's elements are going to be utilized in the multiplication for a given element. Imagine the two highlighted lines over each other, as if they're in the same matrix.

The element where they intercept is the position of the resulting element of the summation of their products:

Matrix multiplication is so important because if we want to explain the following expression in simple terms: A*B (A and B being matrices), we would say:

We are transforming A using B.

This is why matrix multiplication is the quintessential tool for transforming any object in OpenGL or geometry in general.

The last thing you need to know about matrix multiplication is that it has a neutral. This means there is a unique element (matrix in this case) E which when multiplied with any other element A doesn't change A's value, that is:

$$
(!\exists{E}\ \ \forall{A})\ E*A=A
$$

The exclamation point in conjunction with the exists symbol means: A unique element E exists which...

In case of multiplication with normal integers, E has the value of 1. In case of matrices, E has the following values in normal Cartesian (E₁) and homogeneous coordinates (E₂) respectively:

Every single geometric transformation has it's own unique transformation matrix that has a pattern of some sort, of which the most important are:

• Translation
• Scaling
• Reflection
• Rotation
• Sheering

Translation

Translation is the act of literally moving an object by a set vector. The object that's affected by the transformation doesn't change its shape in any way, nor does it change its orientation - it's just moved in space (that's why translation is classified as a movement transformation).

Translation can be described with the following matrix form:

The t-s represents by how much the object's x,y and z location values will be changed.

So, after we transform any coordinates with the translation matrix T, we get:

$$
[x,y,z]*T=[t_x+x,t_y+y,t_z+z]
$$

Translation is implemented with the following OpenGL function:

void glTranslatef(GLfloat tx, GLfloat ty, GLfloat tz);

As you can see, if we know the form of the translation matrix, understanding the OpenGL function is very straightforward, this is the case with all OpenGL transformations.

Don't mind the GLfloat, it's just a clever data type for OpenGL to work on multiple platforms, you can look at it like this:

typedef float GLfloat;
typedef double GLdouble;
typedef someType GLsomeType;

This is a necessary measure because not all systems have the same storage space for a char, for example.

Rotation

Rotation is bit more complicated transformation, because of the simple fact it's dependent on 2 factors:

• Pivot: Around what line in 3D space (or point in 2D space) we'll be rotating
• Amount: By how much (in degrees or radians) we'll be rotating

Because of this, we first need to define rotation in a 2D space, and for that we need a bit of trigonometry.

Here's a quick reference:

These trigonometric functions can only be used inside a right-angled triangle (one of the angles has to be 90 degrees).

The base rotation matrix for rotating an object in 2D space around the vertex (0,0) by the angle A goes as follows:

Again, the 3rd row and 3rd column are just in case we want to stack translation transformations on top of other transformations (which we will in OpenGL), it's ok if you don't fully grasp why they're there right now. Things should clear up in the composite transformation example.

This was all in 2D space, now let's move on to 3D space. In 3D space we need to define a matrix that can rotate an object around any line.

As a wise man once said: "Keep it simple and stupid!" Fortunately, math magicians did for once keep it simple and stupid.

Every single rotation around a line can be broken down into a few transformations:

• Rotation around the x axis
• Rotation around the y axis
• Rotation around the z axis
• Utility translations (which will be touched upon later)

So, the only three things we need to construct for any 3D rotation are matrices that represent rotation around the x, y, and z axis by an angle A:

3D rotation is implemented with the following OpenGL function:

void glRotatef(GLfloat angle, GLfloat x, GLfloat y, GLfloat z);

• angle: angle of rotation in degrees (0-360)
• x,y,z: vector around which the rotation is executed

Scaling

Scaling is the act of multiplying any dimension of the target object by a scalar. This scalar can be <1 if we want to shrink the object, and it can be >1 if we want to enlarge the object.

Scaling can be described with the following matrix form:

s_x, s_y, s_z are the scalars that are multiplied with the x, y, and z values of the target object.

After we transform any coordinates with the scaling matrix S we get:

This transformation is particularly useful when scaling an object by factor k (this means the resulting object is two times bigger), this is achieved by setting s_x=s_y=s_z=k:

A special case of scaling is known as reflection. It's achieved by setting either s_x, s_y, or s_z to -1. This just means we invert the sign of one of the object's coordinates.

In simpler terms, we put the object on the other side of the x, y, or z axis.

This transformation can be modified to work for any plain of reflection, but we don't really need it for now.

void glScalef(GLfloat sx, GLfloat sy, GLfloat sz);

Composite Transformations

Composite transformations are transformations which consist of more than 1 basic transformation (listed above). Transformations A and B are combined by matrix multiplying the corresponding transformation matrices M_a and M_b.

This may seem like very straightforward logic, however there are some things that can be confusing. For example:

• Matrix multiplication is not commutable:

• Every single one of these transformations has an inverse transformation. An inverse transformation is a transformation that cancels out the original one:

• When we want to make an inverse of a composite transformation, we have to change the order of elements utilized:

The point is - the topological order of matrix utilization is very important, just like ascending to a certain floor of a building.

If you're on the first floor, and you want to get to the fourth floor, first you need to go to the third floor and then to the fourth.

But if you want to descend back to the second floor, you would then have to go to the third floor and then to the second floor (in reverse topological order).

Transformations that Involve a Referral Point

As previously mentioned, when a transformation has to be done relative to a specific point in space, for example rotating around a referral point A=(a,b,c) in 3D space, not the origin O=(0,0,0), we need to turn that referral point A into O by translating everything by T(-a,-b,-c).

Then we can do any transformation we need to do, and when we're done, translate everything back by T(a,b,c), so that the original origin O again has the coordinates (0,0,0).

The matrix form of this example is:

Where M is the transformation we wish to do on an object.

The whole point to learning these matrix operations is so that you can fully understand how OpenGL works.

Modeling Demonstration

With all of that out of the way, let's take a look at a simple modeling demonstration.

In order to do anything with OpenGL through Python, we'll use two modules - PyGame and PyOpenGL:

$ python3 -m pip install -U pygame --user
$ python3 -m pip install PyOpenGL PyOpenGL_accelerate

Because it's redundant to unload 3 books worth of graphics theory on yourself, we'll be using the PyGame library. It will essentially just shorten the process from project initialization to actual modeling and animating.

To start off, we need to import everything necessary from both OpenGL and PyGame:

import pygame as pg
from pygame.locals import *

from OpenGL.GL import *
from OpenGL.GLU import *

In the following example, we can see that to model unconventional object, all we need to know is how the complex object can be broken down into smaller and simpler pieces.

Because we still don't know what some of these functions do, I'll give some surface level definitions in the code itself, just so you can see how OpenGL can be used. In the next article, all of these will be covered in detail - this is just to give you a basic idea of how working with OpenGL looks like:

def draw_gun():
    # Setting up materials, ambient, diffuse, specular and shininess properties are all
    # different properties of how a material will react in low/high/direct light for
    # example.
    ambient_coeffsGray = [0.3, 0.3, 0.3, 1]
    diffuse_coeffsGray = [0.5, 0.5, 0.5, 1]
    specular_coeffsGray = [0, 0, 0, 1]
    glMaterialfv(GL_FRONT, GL_AMBIENT, ambient_coeffsGray)
    glMaterialfv(GL_FRONT, GL_DIFFUSE, diffuse_coeffsGray)
    glMaterialfv(GL_FRONT, GL_SPECULAR, specular_coeffsGray)
    glMateriali(GL_FRONT, GL_SHININESS, 1)

    # OpenGL is very finicky when it comes to transformations, for all of them are global,
    # so it's good to seperate the transformations which are used to generate the object
    # from the actual global transformations like animation, movement and such.
    # The glPushMatrix() ----code----- glPopMatrix() just means that the code in between
    # these two functions calls is isolated from the rest of your project.
    # Even inside this push-pop (pp for short) block, we can use nested pp blocks,
    # which are used to further isolate code in it's entirety.
    glPushMatrix()

    glPushMatrix()
    glTranslatef(3.1, 0, 1.75)
    glRotatef(90, 0, 1, 0)
    glScalef(1, 1, 5)
    glScalef(0.2, 0.2, 0.2)
    glutSolidTorus(0.2, 1, 10, 10)
    glPopMatrix()

    glPushMatrix()
    glTranslatef(2.5, 0, 1.75)
    glScalef(0.1, 0.1, 1)
    glutSolidCube(1)
    glPopMatrix()

    glPushMatrix()
    glTranslatef(1, 0, 1)
    glRotatef(10, 0, 1, 0)
    glScalef(0.1, 0.1, 1)
    glutSolidCube(1)

    glPopMatrix()

    glPushMatrix()
    glTranslatef(0.8, 0, 0.8)
    glRotatef(90, 1, 0, 0)
    glScalef(0.5, 0.5, 0.5)
    glutSolidTorus(0.2, 1, 10, 10)
    glPopMatrix()

    glPushMatrix()
    glTranslatef(1, 0, 1.5)
    glRotatef(90, 0, 1, 0)
    glScalef(1, 1, 4)
    glutSolidCube(1)
    glPopMatrix()

    glPushMatrix()
    glRotatef(8, 0, 1, 0)
    glScalef(1.1, 0.8, 3)
    glutSolidCube(1)
    glPopMatrix()

    glPopMatrix()

def main():
    # Initialization of PyGame modules
    pg.init()
    # Initialization of Glut library
    glutInit(sys.argv)
    # Setting up the viewport, camera, backgroud and display mode
    display = (800,600)
    pg.display.set_mode(display, DOUBLEBUF|OPENGL)
    glClearColor(0.1,0.1,0.1,0.3)
    gluPerspective(45, (display[0]/display[1]), 0.1, 50.0)
    gluLookAt(5,5,3,0,0,0,0,0,1)

    glTranslatef(0.0,0.0, -5)
    while True:
        # Listener for exit command
        for event in pg.event.get():
            if event.type == pg.QUIT:
                pg.quit()
                quit()

        # Clears the screen for the next frame to be drawn over
        glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)
        ############## INSERT CODE FOR GENERATING OBJECTS ##################
        draw_gun()
        ####################################################################
        # Function used to advance to the next frame essentially
        pg.display.flip()
        pg.time.wait(10)

This whole bunch of code yields us:

Conclusion

OpenGL is very old, and you won't find many tutorials online on how to properly use it and understand it because all the top dogs are already knee-deep in new technologies.

To properly use OpenGL, one needs to grasp the basic concepts in order to understand the implementations through OpenGL functions.

In this article, we've covered basic matrix operations (translation, rotation, and scaling) as well as composite transformations and transformations that involve a referral point.

In the next article, we'll be using PyGame and PyOpenGL to initialize a project, draw objects, animate them, etc.

#393 – NOVEMBER 5, 2019
View in Browser »

Become a Python Guru With PyCharm

PyCharm is the Python IDE for Professional Developers by JetBrains providing a complete set of tools for productive Python, Web and scientific development. Be more productive and save time while PyCharm takes care of the routine →
JETBRAINSsponsor

Python Jobs

Django Full Stack Web Developer (Austin, TX, USA)

Zeitcode

Full Stack Developer (Toronto, ON, Canada)

Beanfield Metroconnect

Full Stack Software Developer (Remote)

Cybercoders

Full-Stack Python/Django Developer (Remote)

Kimetrica, LLC

Sr. Python Data Engineer (Remote)

TEEMA Solutions Goup

More Python Jobs >>>

Articles & Tutorials

Python Developers Are in Demand on Vettery

Vettery is an online hiring marketplace that’s changing the way people hire and get hired. Ready for a bold career move? Make a free profile, name your salary, and connect with hiring managers from top employers today →
VETTERYsponsor

Projects & Code

Events

Happy Pythoning!
This was PyCoder’s Weekly Issue #393.
View in Browser »

[ Subscribe to 🐍 PyCoder’s Weekly 💌 – Get the best Python news, articles, and tutorials delivered to your inbox once a week >> Click here to learn more ]

This article introduces features built into Zato that let one take advantage of publish/subscribe topics and message queues in communication between Zato services, API clients and backend systems.

Overview

Let's start by recalling the basic means through which services can invoke each other.

• Using self.invoke will invoke another service directly, in a blocking manner, with the calling service waiting for a response from the target one.

self.invoke('my.service','my.request')

• With self.invoke_async, the calling service will invoke another one in background, without waiting for a response from the one being called.

self.invoke_async('my.service','my.request')

There are other variants too, likeasyncpatterns, and all of them work great but what they all have in common is that the entire communication between services takes place in RAM. In other words, as long as Zato servers are up and running, services will be able to communicate.

What happens, though, when a service invokes another one and the server the other one is running on is abruptly stopped? The answer is that, without publish/subscribe, a given request will be lost irrevocably - after all, it was in RAM only so there is no way for it to be available across server restarts.

Sometimes this is desirable and sometimes it is not - publish/subscribe topics and message queues focus on scenarios of the latter kind. Let's discuss publish/subscribe, then.

Introducing publish/subscribe

In its essence, publish/subscribe is about building message-based workflows and processes revolving around asynchronous communication.

Publishers send messages to topics, subscribers have queues for data from topics and Zato ensures that messages will be delivered to subscribers even if servers are not running or if subscribers are not currently available.

Publish/subscribe and Zato services

Publish/subscribe, as it pertains to Zato services, is an extension of the idea of message topics and queues. In this case, it is either internal or user-defined services that topics and queues are used by. Whereas previously, publishers and subscribers were external applications, here both of these roles are fulfilled by services running in Zato servers.

In the diagram above, an external API client invokes a channel service, for instance a REST one. Now, instead of using self.invoke or self.invoke_async the channel service will use self.pubsub.publish to publish the message received to a backend service which will in turn deliver it to external, backend systems.

The nice part of it is that, given how Zato publish/subscribe works, even if all servers are stopped and even if some of the recipients are not available, the message will be delivered eventually. That is the cornerstone of the design - if everything works smoothly, the message will be delivered immediately, but if anything goes wrong along the way, the message is retained and attempts are made periodically to deliver it to its destination.

Python code

As usual in Zato, the Python code needed is straightforward. Below, one service publishes a message to another - the programmer does not need to think about the inner details of publish/subscribe, about locations of servers, re-deliveries or guaranteed delivery. Merely using self.publish.publish suffices for everything to work out of the box.

# -*- coding: utf-8 -*-from__future__importabsolute_import,division,print_function,unicode_literals# Zatofromzato.server.serviceimportService# ################################################################################classMyService(Service):name='pub.sub.source.1'defhandle(self):# What service to publish the message totarget='pub.sub.target.1'# Data to invoke the service with, here, we are just taking as-is# what we were given on input.data=self.request.raw_request# An optional correlation ID to assign to the published message,# if givenm it can be anything as long as it is unique.cid=self.cidself.pubsub.publish(target,data=data,cid=cid)# Return the correlation ID to our callerself.response.payload=cid# ################################################################################classPubSubTarget(Service):name='pub.sub.target.1'defhandle(self):# This is how the incoming message can be accessedmsg=self.request.raw_request# Now, the message can be processed accordingly# The actual code is skipped here - it will depend# on one's particular needs.# ################################################################################

Asynchronous message flows

The kind of message flows that publish/subscribe topics promote are called asynchronous because, seeing as in a general case it is not possible to guarantee that communication will be instantaneous, the initial callers (e.g. API clients) and possibly other participants too should only submit their requests without expectations that responses, if any, will appear immediately.

Consider a simple case of topping up a pay-as-you go mobile phone. Such a process will invariably require participation from at least several backend systems, all of which can be coordinated by Zato.

Let's say that the initial caller, the API client initiating the process, is a cell phone itself, sending a text message with a top-up code from a gift card.

Clearly, there is no need for the phone itself to actively wait for the response. With several backend systems involved, it may take anything between seconds to minutes before the card is actually recharged and there is no need to keep all of the systems involved, including the cell phone, occupied.

At the same time, in case some of the backend systems are down and the initial request is lost, we cannot expect that the end user will keep purchasing more cards - we need some kind of a guaranteed delivery mechanism, which is precisely where Zato topics are useful with their ability to retain messages in case immediate delivery is not possible.

With topics, if a response is needed, instead of waiting in a synchronous manner, API callers can be given a correlation ID (CID) on output when they submit a request. A CID is just a random piece of string, uniquely identifying the request.

In the Python code example, self.cid is used for the CID. It is convenient to use it because it already available for each service and Zato knows how to use it in other parts of the platform - for instance, if the request is via HTTP (REST or SOAP), the correlation ID will be saved in Apache-style HTTP access logs. This facilitates answering of typical support questions, such as 'What happened to this or that message, when was it processed or when was the response produced?'

We have a CID but why is it useful? It is because it may be used an ID to key messages of two kinds:

• API callers may save it and then be notified later on by Zato-based services that such and such request, one with a particular CID, has been already processed

• API callers may save it and then periodically query Zato-based services if a given request is already processed

Which style to use ultimately depends on the overall business and technical processes that Zato publish/subscribe and services support - sometimes it is more desirable to receive notifications yet sometimes it is not possible at all, e.g. if the recipients are rarely accessible, for instance, if they join networks irregularly.

Web-admin GUI

In parting words, it needs to be mentioned that a very convenient aspect of Zato services' being part of the bigger publish/subscribe mechanism is that web-admin GUI treats them just like any other endpoint and we can browse their topics, inspect last messages published, consult web-admin to check how many messages were published, or carry out other tasks that it is capable of, like in the screenshots below:

If you need to configure Zato for Oracle DB connections and you want to ensure the highest performance possible, this is the post which goes through the process step-by-step. Read on for details.

Overview

Note that Zato 3.1+ is required. The tasks involved are:

• Installing Zato
• Installing an Oracle client
• Greenifying the Oracle client
• Starting servers
• Creating database connection definitions
• Confirming the installation

Installing Zato

• Choose your preferred operating system and follow the general installation instructions - Oracle DB connections will work the same no matter the system Zato runs on

• Create a Zato environment with as many servers as required. Again, there are no Oracle-specific steps at this point yet.

Installing an Oracle client

• Download an Oracle client and install it on all the systems with Zato servers

• Add the client's installation path to LD_LIBRARY_PATH. For instance, if the client is installed to /opt/zato/instantclient_19_3, add the following to ~/.bashrc:

  export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/zato/instantclient_19_3

Greenifying the Oracle client

• This step is crucial for achieving the highest performance of SQL queries

• For each Zato server installed, open its server.conf file and find stanza [greenify]

• If there is no such stanza in server.conf, create it yourself

• Modify the stanza to greenify the libclntsh.so library - this is a library that the Oracle client ships with. For instance, if the client's installation path is /opt/zato/instantclient_19_3, the full path to the library is /opt/zato/instantclient_19_3/libclntsh.so

• The stanza should read as below, assuming the installation path as above:

  [greenify]
  /opt/zato/instantclient_19_3/libclntsh.so=True

• Note that entries in this stanza are followed by =True, which is a signal to Zato that a particular library should be processed

Starting servers

• Start Zato servers

• For each, make sure that an entry such as below is saved in server logs

  INFO - Greenified library `/opt/zato/instantclient_19_3/libclntsh.so.19.1`
  

Creating database connection definitions

• Go to web-admin and create a new Oracle DB connection via Connections -> Outgoing -> SQL, as below:

• Update the newly created connection's password by clicking Change password

• Click Ping to confirm the connectivity to the remote server

• This concludes the process, the connection is ready for use in Zato services now

We have more ways to manage dependencies in Python applications than ever. But how do they fare in production? Unfortunately this topic turned out to be quite polarizing and was at the center of a lot of heated debates. This is my attempt at an opinionated review through a DevOps lens.

Python is famous for allowing you to write code that’s elegant, easy to write, and almost as easy to read as plain English. One of the language’s most distinctive features is the list comprehension, which you can use to create powerful functionality within a single line of code. However, many developers struggle to fully leverage the more advanced features of a list comprehension in Python. Some programmers even use them too much, which can lead to code that’s less efficient and harder to read.

By the end of this tutorial, you’ll understand the full power of Python list comprehensions and how to use their features comfortably. You’ll also gain an understanding of the trade-offs that come with using them so that you can determine when other approaches are more preferable.

In this tutorial, you’ll learn how to:

• Rewrite loops and map() calls as a list comprehension in Python
• Choose between comprehensions, loops, and map() calls
• Supercharge your comprehensions with conditional logic
• Use comprehensions to replace filter()
• Profile your code to solve performance questions

How to Create Lists in Python

There are a few different ways you can create lists in Python. To better understand the trade-offs of using a list comprehension in Python, let’s first see how to create lists with these approaches.

Using for Loops

The most common type of loop is the for loop. You can use a for loop to create a list of elements in three steps:

1. Instantiate an empty list.
2. Loop over an iterable or range of elements.
3. Append each element to the end of the list.

If you want to create a list containing the first ten perfect squares, then you can complete these steps in three lines of code:

>>> squares=[]>>> foriinrange(10):... squares.append(i*i)>>> squares[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

Here, you instantiate an empty list, squares. Then, you use a for loop to iterate over range(10). Finally, you multiply each number by itself and append the result to the end of the list.

Using map() Objects

map() provides an alternative approach that’s based in functional programming. You pass in a function and an iterable, and map() will create an object. This object contains the output you would get from running each iterable element through the supplied function.

As an example, consider a situation in which you need to calculate the price after tax for a list of transactions:

>>> txns=[1.09,23.56,57.84,4.56,6.78]>>> TAX_RATE=.08>>> defget_price_with_tax(txn):... returntxn*(1+TAX_RATE)>>> final_prices=map(get_price_with_tax,txns)>>> list(final_prices)[1.1772000000000002, 25.4448, 62.467200000000005, 4.9248, 7.322400000000001]

Here, you have an iterable txns and a function get_price_with_tax(). You pass both of these arguments to map(), and store the resulting object in final_prices. You can easily convert this map object into a list using list().

Using List Comprehensions

List comprehensions are a third way of making lists. With this elegant approach, you could rewrite the for loop from the first example in just a single line of code:

>>> squares=[i*iforiinrange(10)]>>> squares[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

Rather than creating an empty list and adding each element to the end, you simply define the list and its contents at the same time by following this format:

new_list = [expression for member in iterable]

Every list comprehension in Python includes three elements:

1. expression is the member itself, a call to a method, or any other valid expression that returns a value. In the example above, the expression i * i is the square of the member value.
2. member is the object or value in the list or iterable. In the example above, the member value is i.
3. iterable is a list, set, sequence, generator, or any other object that can return its elements one at a time. In the example above, the iterable is range(10).

Because the expression requirement is so flexible, a list comprehension in Python works well in many places where you would use map(). You can rewrite the pricing example with its own list comprehension:

>>> txns=[1.09,23.56,57.84,4.56,6.78]>>> TAX_RATE=.08>>> defget_price_with_tax(txn):... returntxn*(1+TAX_RATE)>>> final_prices=[get_price_with_tax(i)foriintxns]>>> final_prices[1.1772000000000002, 25.4448, 62.467200000000005, 4.9248, 7.322400000000001]

The only distinction between this implementation and map() is that the list comprehension in Python returns a list, not a map object.

Benefits of Using List Comprehensions

List comprehensions are often described as being more Pythonic than loops or map(). But rather than blindly accepting that assessment, it’s worth it to understand the benefits of using a list comprehension in Python when compared to the alternatives. Later on, you’ll learn about a few scenarios where the alternatives are a better choice.

One main benefit of using a list comprehension in Python is that it’s a single tool that you can use in many different situations. In addition to standard list creation, list comprehensions can also be used for mapping and filtering. You don’t have to use a different approach for each scenario.

This is the main reason why list comprehensions are considered Pythonic, as Python embraces simple, powerful tools that you can use in a wide variety of situations. As an added side benefit, whenever you use a list comprehension in Python, you won’t need to remember the proper order of arguments like you would when you call map().

List comprehensions are also more declarative than loops, which means they’re easier to read and understand. Loops require you to focus on how the list is created. You have to manually create an empty list, loop over the elements, and add each of them to the end of the list. With a list comprehension in Python, you can instead focus on what you want to go in the list and trust that Python will take care of how the list construction takes place.

How to Supercharge Your Comprehensions

In order to understand the full value that list comprehensions can provide, it’s helpful to understand their range of possible functionality. You’ll also want to understand the changes that are coming to the list comprehension in Python 3.8.

Using Conditional Logic

Earlier, you saw this formula for how to create list comprehensions:

new_list = [expression for member in iterable]

While this formula is accurate, it’s also a bit incomplete. A more complete description of the comprehension formula adds support for optional conditionals. The most common way to add conditional logic to a list comprehension is to add a conditional to the end of the expression:

new_list = [expression for member in iterable (if conditional)]

Here, your conditional statement comes just before the closing bracket.

Conditionals are important because they allow list comprehensions to filter out unwanted values, which would normally require a call to filter():

>>> sentence='the rocket came back from mars'>>> vowels=[iforiinsentenceifiin'aeiou']>>> vowels['e', 'o', 'e', 'a', 'e', 'a', 'o', 'a']

In this code block, the conditional statement filters out any characters in sentence that aren’t a vowel.

The conditional can test any valid expression. If you need a more complex filter, then you can even move the conditional logic to a separate function:

>>> sentence='The rocket, who was named Ted, came back \... from Mars because he missed his friends.'>>> defis_consonant(letter):... vowels='aeiou'... returnletter.isalpha()andletter.lower()notinvowels>>> consonants=[iforiinsentenceifis_consonant(i)]['T', 'h', 'r', 'c', 'k', 't', 'w', 'h', 'w', 's', 'n', 'm', 'd', \'T', 'd', 'c', 'm', 'b', 'c', 'k', 'f', 'r', 'm', 'M', 'r', 's', 'b', \'c', 's', 'h', 'm', 's', 's', 'd', 'h', 's', 'f', 'r', 'n', 'd', 's']

Here, you create a complex filter is_consonant() and pass this function as the conditional statement for your list comprehension. Note that the member value i is also passed as an argument to your function.

You can place the conditional at the end of the statement for simple filtering, but what if you want to change a member value instead of filtering it out? In this case, it’s useful to place the conditional near the beginning of the expression:

new_list = [expression (if conditional) for member in iterable]

With this formula, you can use conditional logic to select from multiple possible output options. For example, if you have a list of prices, then you may want to replace negative prices with 0 and leave the positive values unchanged:

>>> original_prices=[1.25,-9.45,10.22,3.78,-5.92,1.16]>>> prices=[iifi>0else0foriinoriginal_prices]>>> prices[1.25, 0, 10.22, 3.78, 0, 1.16]

Here, your expression i contains a conditional statement, if i > 0 else 0. This tells Python to output the value of i if the number is positive, but to change i to 0 if the number is negative. If this seems overwhelming, then it may be helpful to view the conditional logic as its own function:

>>> defget_price(price):... returnpriceifprice>0else0>>> prices=[get_price(i)foriinoriginal_prices]>>> prices[1.25, 0, 10.22, 3.78, 0, 1.16]

Now, your conditional statement is contained within get_price(), and you can use it as part of your list comprehension expression.

Using Set and Dictionary Comprehensions

While the list comprehension in Python is a common tool, you can also create set and dictionary comprehensions. A set comprehension is almost exactly the same as a list comprehension in Python. The difference is that set comprehensions make sure the output contains no duplicates. You can create a set comprehension by using curly braces instead of brackets:

>>> quote="life, uh, finds a way">>> unique_vowels={iforiinquoteifiin'aeiou'}>>> unique_vowels{'a', 'e', 'u', 'i'}

Your set comprehension outputs all the unique vowels it found in quote. Unlike lists, sets don’t guarantee that items will be saved in any particular order. This is why the first member of the set is a, even though the first vowel in quote is i.

Dictionary comprehensions are similar, with the additional requirement of defining a key:

>>> squares={i:i*iforiinrange(10)}>>> squares{0: 0, 1: 1, 2: 4, 3: 9, 4: 16, 5: 25, 6: 36, 7: 49, 8: 64, 9: 81}

To create the squares dictionary, you use curly braces ({}) as well as a key-value pair (i: i * i) in your expression.

Using the Walrus Operator

Python 3.8 will introduce the assignment expression, also known as the walrus operator. To understand how you can use it, consider the following example.

Say you need to make ten requests to an API that will return temperature data. You only want to return results that are greater than 100 degrees Fahrenheit. Assume that each request will return different data. In this case, there’s no way to use a list comprehension in Python to solve the problem. The formula expression for member in iterable (if conditional) provides no way for the conditional to assign data to a variable that the expression can access.

The walrus operator solves this problem. It allows you to run an expression while simultaneously assigning the output value to a variable. The following example shows how this is possible, using get_weather_data() to generate fake weather data:

>>> importrandom>>> defget_weather_data():... returnrandom.randrange(90,110)>>> hot_temps=[tempfor_inrange(20)if(temp:=get_weather_data())>=100]>>> hot_temperatures[107, 102, 109, 104, 107, 109, 108, 101, 104]

You won’t often need to use the assignment expression inside of a list comprehension in Python, but it’s a useful tool to have at your disposal when necessary.

When Not to Use a List Comprehension in Python

List comprehensions are useful and can help you write elegant code that’s easy to read and debug, but they’re not the right choice for all circumstances. They might make your code run more slowly or use more memory. If your code is less performant or harder to understand, then it’s probably better to choose an alternative.

Watch Out for Nested Comprehensions

Comprehensions can be nested to create combinations of lists, dictionaries, and sets within a collection. For example, say a climate laboratory is tracking the high temperature in five different cities for the first week of June. The perfect data structure for storing this data could be a Python list comprehension nested within a dictionary comprehension:

>>> cities=['Austin','Tacoma','Topeka','Sacramento','Charlotte']>>> temps={city:[0for_inrange(7)]forcityincities}>>> temps{    'Austin': [0, 0, 0, 0, 0, 0, 0],    'Tacoma': [0, 0, 0, 0, 0, 0, 0],    'Topeka': [0, 0, 0, 0, 0, 0, 0],    'Sacramento': [0, 0, 0, 0, 0, 0, 0],    'Charlotte': [0, 0, 0, 0, 0, 0, 0]}

You create the outer collection temps with a dictionary comprehension. The expression is a key-value pair, which contains yet another comprehension. This code will quickly generate a list of data for each city in cities.

Nested lists are a common way to create matrices, which are often used for mathematical purposes. Take a look at the code block below:

>>> matrix=[[iforiinrange(5)]for_inrange(6)]>>> matrix[    [0, 1, 2, 3, 4],    [0, 1, 2, 3, 4],    [0, 1, 2, 3, 4],    [0, 1, 2, 3, 4],    [0, 1, 2, 3, 4],    [0, 1, 2, 3, 4]]

The outer list comprehension [... for _ in range(6)] creates six rows, while the inner list comprehension [i for i in range(5)] fills each of these rows with values.

So far, the purpose of each nested comprehension is pretty intuitive. However, there are other situations, such as flattening nested lists, where the logic arguably makes your code more confusing. Take this example, which uses a nested list comprehension to flatten a matrix:

matrix = [... [0,0,0],... [1,1,1],... [2,2,2],... ]>>> flat=[numforrowinmatrixfornuminrow]>>> flat[0, 0, 0, 1, 1, 1, 2, 2, 2]

The code to flatten the matrix is concise, but it may not be so intuitive to understand how it works. On the other hand, if you were to use for loops to flatten the same matrix, then your code will be much more straightforward:

>>> matrix=[... [0,0,0],... [1,1,1],... [2,2,2],... ]>>> flat=[]>>> forrowinmatrix:... fornuminrow:... flat.append(num)...>>> flat[0, 0, 0, 1, 1, 1, 2, 2, 2]

Now you can see that the code traverses one row of the matrix at a time, pulling out all the elements in that row before moving on to the next one.

While the single-line nested list comprehension might seem more Pythonic, what’s most important is to write code that your team can easily understand and modify. When you choose your approach, you’ll have to make a judgment call based on whether you think the comprehension helps or hurts readability.

Choose Generators for Large Datasets

A list comprehension in Python works by loading the entire output list into memory. For small or even medium-sized lists, this is generally fine. If you want to sum the squares of the first one-thousand integers, then a list comprehension will solve this problem admirably:

>>> sum([i*iforiinrange(1000)])332833500

But what if you wanted to sum the squares of the first billion integers? If you tried then on your machine, then you may notice that your computer becomes non-responsive. That’s because Python is trying to create a list with one billion integers, which consumes more memory than your computer would like. Your computer may not have the resources it needs to generate an enormous list and store it in memory. If you try to do it anyway, then your machine could slow down or even crash.

When the size of a list becomes problematic, it’s often helpful to use a generator instead of a list comprehension in Python. A generator doesn’t create a single, large data structure in memory, but instead returns an iterable. Your code can ask for the next value from the iterable as many times as necessary or until you’ve reached the end of your sequence, while only storing a single value at a time.

If you were to sum the first billion squares with a generator, then your program will likely run for a while, but it shouldn’t cause your computer to freeze. The example below uses a generator:

>>> sum(i*iforiinrange(1000000000))333333332833333333500000000

You can tell this is a generator because the expression isn’t surrounded by brackets or curly braces. Optionally, generators can be surrounded by parentheses.

The example above still requires a lot of work, but it performs the operations lazily. Because of lazy evaluation, values are only calculated when they’re explicitly requested. After the generator yields a value (for example, 567 * 567), it can add that value to the running sum, then discard that value and generate the next value (568 * 568). When the sum function requests the next value, the cycle starts over. This process keeps the memory footprint small.

map() also operates lazily, meaning memory won’t be an issue if you choose to use it in this case:

>>> sum(map(lambdai:i*i,range(1000000000)))333333332833333333500000000

It’s up to you whether you prefer the generator expression or map().

Profile to Optimize Performance

So, which approach is faster? Should you use list comprehensions or one of their alternatives? Rather than adhere to a single rule that’s true in all cases, it’s more useful to ask yourself whether or not performance matters in your specific circumstance. If not, then it’s usually best to choose whatever approach leads to the cleanest code!

If you’re in a scenario where performance is important, then it’s typically best to profile different approaches and listen to the data. timeit is a useful library for timing how long it takes chunks of code to run. You can use timeit to compare the runtime of map(), for loops, and list comprehensions:

>>> importrandom>>> importtimeit>>> TAX_RATE=.08>>> txns=[random.randrange(100)for_inrange(100000)]>>> defget_price(txn):... returntxn*(1+TAX_RATE)...>>> defget_prices_with_map():... returnlist(map(get_price,txns))...>>> defget_prices_with_comprehension():... return[get_price(txn)fortxnintxns]...>>> defget_prices_with_loop():... prices=[]... fortxnintxns:... prices.append(get_price(txn))... returnprices...>>> timeit.timeit(get_prices_with_map,number=100)2.0554370979998566>>> timeit.timeit(get_prices_with_comprehension,number=100)2.3982384680002724>>> timeit.timeit(get_prices_with_loop,number=100)3.0531821520007725

Here, you define three methods that each use a different approach for creating a list. Then, you tell timeit to run each of those functions 100 times each. timeit returns the total time it took to run those 100 executions.

As the code demonstrates, the biggest difference is between the loop-based approach and map(), with the loop taking 50% longer to execute. Whether or not this matters depends on the needs of your application.

Conclusion

In this tutorial, you learned how to use a list comprehension in Python to accomplish complex tasks without making your code overly complicated.

Now you can:

• Simplify loops and map() calls with declarative list comprehensions
• Supercharge your comprehensions with conditional logic
• Create set and dictionary comprehensions
• Determine when code clarity or performance dictates an alternative approach

Whenever you have to choose a list creation method, try multiple implementations and consider what’s easiest to read and understand in your specific scenario. If performance is important, then you can use profiling tools to give you actionable data instead of relying on hunches or guesses about what works the best.

Remember that while Python list comprehensions get a lot of attention, your intuition and ability to use data when it counts will help you write clean code that serves the task at hand. This, ultimately, is the key to making your code Pythonic!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

When you're reading this blog post Sentry which I have been working on for the last few years has undergone a license change. Making money with Open Source has always been a complex topic and over the years my own ideas of how this should be done have become less and less clear. The following text is an attempt to summarize my thoughts on it an to put some more clarification on how we ended up picking the BSL license for Sentry.

I’ve recently appeared on a whole bunch of podcasts about Python, freelancing, and even (believe it or not) learning Chinese! If you’re interested in any or all of these subjects, then you might want to catch my interviews:

• Talk Python to Me: I spoke with Michael Kennedy (and Casey Kinsen) about freelancing in Python — and things to consider when you’re thinking of freelancing.
• Programming Leadership: I spoke with Marcus Blankenship about why companies offer training to their employees, how they should look for training, and how best to take advantage of a course.
• Profitable Python: I spoke with Ben McNeill about the world of Python training — how training works (for me, companies that invite me to train, and the people in my courses), how to build up an online business, and the difference between B2C vs. B2B. You can watch the video on YouTube, or listen to the audio version of the podcast!
• Teaching Python: I spoke with Kelly Paredes and Sean Tibor about what it’s like to teach adults vs. children, and what tricks I use to help keep my students engaged. I learned quite a bit about how they teach Python to middle-school students!
• You Can Learn Chinese: I’ve been studying Chinese for a few years, and spent some time chatting with Jared Turner about my experience, how I continue to improve, and how my Chinese studies have affected my work teaching Python. The entire episode is great, and my interview starts about halfway through.

In related news, you might know that I’ve been a co-panelist on the Freelancers Show podcast for the last few years. The entire panel (including me) recently left the show, and we’re currently discussing how/when/where we’ll restart.

I’ll be sure to post to my blog here when there are updates — but if you’re a freelancer of any level (new or experienced) who might be interested in sharing your stories with us, please contact me, so we can speak with you when we re-start in our new format.

The post Podcasts, podcasts, and even more podcasts appeared first on Reuven Lerner.

The post How to Handle Coroutines with asyncio in Python appeared first on Erik Marsja.

When a program becomes very long and complex, it is convenient to divide it into subroutines, each of which implements a specific task. However, subroutines cannot be executed independently, but only at the request of the main program, which is responsible for coordinating the use of subroutines.

In this post, we introduce a generalization of the concept of subroutines, known as coroutines: just like subroutines, coroutines compute a single computational step, but unlike subroutines, there is no main program to coordinate the results. The coroutines link themselves together to form a pipeline without any supervising function responsible for calling them in a particular order. 

This post is taken from the book Python Parallel Programming Cookbook  (2nd Ed.) by Giancarlo Zaccone. In this book, you will implement effective programming techniques in Python to build scalable software that saves time and memory. 

In a coroutine, the execution point can be suspended and resumed later, since the coroutine keeps track of the state of execution. Having a pool of coroutines, it is possible to interleave the computations: the first one runs until it yields control back, then the second runs and goes on down the line.

Read Also: Introducing Coil, an open-source Android image loading library backed by Kotlin Coroutines

The interleaving is managed by the event loop. It keeps track of all the coroutines and schedules when they will be executed.

Other important aspects of coroutines are as follows:

• Coroutines allow for multiple entry points that can yield multiple times.
• Coroutines can transfer execution to any other coroutine.

The term yield is used here to describe a coroutine pausing and passing the control flow to another coroutine.

Getting ready to work with coroutines

We will use the following notation to work with coroutines:

import asyncio 

@asyncio.coroutine
def coroutine_function(function_arguments):
    ............
    DO_SOMETHING
    ............

Coroutines use the yield from syntax introduced in PEP 380 (read more at https://www.python.org/dev/peps/pep-0380/) to stop the execution of the current computation and suspends the coroutine’s internal state.

In particular, in the case of yield from future, the coroutine is suspended until future is done, then the result of future will be propagated (or raise an exception); in the case of yield from coroutine, the coroutine waits for another coroutine to produce a result that will be propagated (or raise an exception).

As we shall see in the next example, in which the coroutines will be used to simulate a finite state machine, we will use the yield from coroutine notation.

More on coroutines with asyncio are available at https://docs.python.org/3.5/library/asyncio-task.html.

Using coroutines to simulate a finite state machine

In this example, we see how to use coroutines to simulate a finite state machine with five states.

A finite state machine or finite state automaton is a mathematical model that is widely used in engineering disciplines, but also in sciences such as mathematics and computer science.

The automaton that we want to simulate the behavior of using coroutines is as follows:

The states of the system are S0, S1, S2, S3, and S4, with 0 and 1: the values for which the automaton can pass from one state to the next state (this operation is called a transition). So, for example, state S0 can pass to state S1, but only for the value 1, and S0 can pass to state S2, but only for the value 0.

The following Python code simulates a transition of the automaton from state S0 (the start state), up to state S4 (the end state):

1) The first step is obviously to import the relevant libraries:

import asyncio
import time
from random import randint

2) Then, we define the coroutine relative to start_state. The input_value parameter is evaluated randomly; it can be 0 or 1. If it is 0, then the control goes to coroutinestate2; otherwise, it changes to coroutine state1:

@asyncio.coroutine
def start_state():
    print('Start State called\n')
    input_value = randint(0, 1)
    time.sleep(1)
    if input_value == 0:
        result = yield from state2(input_value)
    else:
        result = yield from state1(input_value)
    print('Resume of the Transition:\nStart State calling'+ result)

3) Here is the coroutine for state1. The input_value parameter is evaluated randomly; it can be 0 or 1. If it is 0, then the control goes tostate2; otherwise, it changes to state1:

@asyncio.coroutine
def state1(transition_value):
    output_value ='State 1 with transition value = %s\n'% \
                                             transition_value
    input_value = randint(0, 1)
    time.sleep(1)
    print('...evaluating...')
    if input_value == 0:
        result = yield from state3(input_value)
    else:
        result = yield from state2(input_value)
    return output_value + 'State 1 calling %s' % result

4) The coroutine for state1 has the transition_value argument that allowed the passage of the state. Also, in this case, input_value is randomly evaluated. If it is 0, then the state transitions to state3; otherwise, the control changes to state2:

@asyncio.coroutine
def state2(transition_value):
    output_value = 'State 2 with transition value = %s\n' %\
                                             transition_value
    input_value = randint(0, 1)
    time.sleep(1)
    print('...evaluating...')
    if input_value == 0:
        result = yield from state1(input_value)
    else:
        result = yield from state3(input_value)
    return output_value + 'State 2 calling %s' % result

5) The coroutine for state3 has the transition_value argument, which allowed the passage of the state. input_value is randomly evaluated. If it is 0, then the state transitions to state1; otherwise, the control changes to end_state:

@asyncio.coroutine
def state3(transition_value):
    output_value = 'State 3 with transition value = %s\n' %\
                                                 transition_value
    input_value = randint(0, 1)
    time.sleep(1)
    print('...evaluating...')
    if input_value == 0:
        result = yield from state1(input_value)
    else:
        result = yield from end_state(input_value)
    return output_value + 'State 3 calling %s' % result

end_state prints out the transition_value argument, which allowed the passage of the state, and then stops the computation:

@asyncio.coroutine
def end_state(transition_value):
    output_value = 'End State with transition value = %s\n'%\
                                                transition_value
    print('...stop computation...')
    return output_value

7) In the __main__ function, the event loop is acquired, and then we start the simulation of the finite state machine, calling the automaton’s start_state:

if __name__ == '__main__':
    print('Finite State Machine simulation with Asyncio Coroutine')
    loop = asyncio.get_event_loop()
    loop.run_until_complete(start_state())

How coroutines simulate a finite state machine

Each state of the automaton has been defined by using the decorator:

@asyncio.coroutine

For example, state S0 is defined here:

@asyncio.coroutine
def StartState():
    print ("Start State called \n")
    input_value = randint(0,1)
    time.sleep(1)
    if (input_value == 0):
        result = yield from State2(input_value)
    else :
        result = yield from State1(input_value)

The transition to the next state is determined by input_value, which is defined by the randint(0,1) function of Python’s random module. This function randomly provides a value of 0 or 1.

In this manner, randintrandomly determines the state to which the finite state machine will pass:

input_value = randint(0,1)

After determining the values to pass, the coroutine calls the next coroutine using the yield from command:

if (input_value == 0):
        result = yield from State2(input_value)
    else :
        result = yield from State1(input_value)

The result variable is the value that each coroutine returns. It is a string, and, at the end of the computation, we can reconstruct the transition from the initial state of the automaton, start_state, up to end_state.

The main program starts the evaluation inside the event loop:

if __name__ == "__main__":
    print("Finite State Machine simulation with Asyncio Coroutine")
    loop = asyncio.get_event_loop()
    loop.run_until_complete(StartState())

Running the code, we have an output like this:

Finite State Machine simulation with Asyncio Coroutine
Start State called
...evaluating...
...evaluating...
...evaluating...
...evaluating...
...stop computation...
Resume of the Transition : 
Start State calling State 1 with transition value = 1
State 1 calling State 2 with transition value = 1
State 2 calling State 1 with transition value = 0
State 1 calling State 3 with transition value = 0
State 3 calling End State with transition value = 1

Handling coroutines with asyncio in Python 3.5

Before Python 3.5 was released, the asyncio module used generators to mimic asynchronous calls and, therefore, had a different syntax than the current version of Python 3.5.

Python 3.5 introduced the async and await keywords. Notice the lack of parentheses around the await func() call.

The following is an example of “Hello, world!“, using asyncio with the new syntax introduced by Python 3.5+:

import asyncio
 
async def main():
    print(await func())
 
async def func():
    # Do time intensive stuff...
    return "Hello, world!"
 
if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())

In this post, we learned how to handle coroutines with asyncio. To learn more features of asynchronous programming in Python, you may go through the book Python Parallel Programming Cookbook  (2nd Ed.) by Packt Publishing.

The post How to Handle Coroutines with asyncio in Python appeared first on Erik Marsja.

I gave two talks at BayPIGgies. Here are the slides:

• How to Give a Talk
• Building Video Games for Fun with PyGame