In this course, you’ll learn about working with lists and tuples. Lists and tuples are arguably Python’s most versatile, useful data types. You’ll find them in virtually every non-trivial Python program.

Here’s what you’ll learn in this tutorial: You’ll cover the important characteristics of lists and tuples. You’ll learn how to define them and how to manipulate them. When you’re finished, you’ll have a good feel for when and how to use these object types in a Python program.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

In this article, we will use a Python-based messaging client to connect and subscribe to a topic with a durable subscription in the Apache ActiveMQ Artemis broker. We will use the text-based STOMP protocol to connect and subscribe to the broker. STOMP clients can communicate with any STOMP message broker to provide messaging interoperability among many languages, platforms, and brokers.

If you need to brush up on the difference between persistence and durability in messaging, check Mary Cochran’s article on developers.redhat.com/blog.

A similar process can be used with Red Hat AMQ 7. The broker in Red Hat AMQ 7 is based on the Apache ActiveMQ Artemis project. See the overview on developers.redhat.com for more information.

Setting Up the Project

In the following example, we are using one client, both to publish and subscribe to a topic. You can find the code at my personal GitHub repo. We have two receiver_queue.py and receiver_topic.py Python messaging clients. While receiver_queue.py is a Python client based on the STOMP protocol for point-to-point (queue) connection to the broker, receiver_topic.py is a Python client based on the STOMP protocol for durable subscription against a topic to the broker.

Here is the code:

import time
import sys

import stomp

class MyListener(stomp.ConnectionListener):
 def on_error(self, headers, message):
 print('received an error "%s"' % message)
 def on_message(self, headers, message):
 print('received a message "%s"' % message)
hosts = [('localhost', 61616)]

conn = stomp.Connection(host_and_ports=hosts)
conn.set_listener('', MyListener())
conn.start()
conn.connect('admin', 'admin', wait=True,headers = {'client-id': 'clientname'} )
conn.subscribe(destination='A.B.C.D', id=1, ack='auto',headers = {'subscription-type': 'MULTICAST','durable-subscription-name':'someValue'})

conn.send(body=' '.join(sys.argv[1:]), destination='A.B.C.D')

time.sleep(2)
conn.disconnect()

The following are tasks performed by this code:

• To receive messages from the messaging system, we need to set up a listener on a connection, and then later subscribe to the destination.
• We are establishing a connection to the broker available locally on port 61616. The first parameter to a Connection is host_and_ports. This contains an IP address and the port where the message broker is listening for STOMP connections.
• The start method creates a socket connection to the broker.
• Then we use the connect method with credentials to access the broker and we use the headers client-id to ensure that the subscription that is created is durable.
• Once a connection is established to the broker with subscribe method, we are subscribing to destination A.B.C.D using acknowledgment mode auto. Also, we must provide the headers subscription-type as MULTICAST and durable-subscription-name as some text value.
• To create a durable subscription, the client-id header must be set on the CONNECT frame and the durable-subscription-name must be set on the SUBSCRIBE frame. The combination of these two headers will form the identity of the durable subscription.
• After a connection is established to the broker, we can use the send method to send/produce messages to the destination A.B.C.D. Here the first argument is to accept the text/string value from the command line, and the second argument is destination name or topic name.

How to Execute the Python Client

• Make sure the  Apache ActiveMQ Artemis broker is configured to support the STOMP protocol. By default, port 61616 is configured to support almost all messaging protocols.

<acceptor name="artemis">tcp://0.0.0.0:61616?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=CORE,AMQP,STOMP,HORNETQ,MQTT,OPENWIRE;useEpoll=true;amqpCredits=1000;amqpLowCredits=300</acceptor>

• To run the client using the STOMP protocol, we first need the stomp module so that components of the STOMP API, such as connect, start,  send, subscribe, and disconnect, are available. So install the stomp module first.

pip install stomp.py

• Once the stomp module is installed, we can easily run the client in the following way:

[cpandey@vm254-231 python_stomp_example]$ python receiver_topic.py "Hello World"
received a message "Hello World"
[cpandey@vm254-231 python_stomp_example]$

• We can check the results using the following commands from the Apache ActiveMQ Artemis broker:

[cpandey@vm254-231 bin]$ ./artemis address show

A.B.C.D
DLQ

[cpandey@vm254-231 bin]$ ./artemis queue stat --user admin --password admin --url tcp://localhost:61616
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
|NAME |ADDRESS |CONSUMER_COUNT |MESSAGE_COUNT |MESSAGES_ADDED |DELIVERING_COUNT |MESSAGES_ACKED |
|DLQ |DLQ |0 |0 |0 |0 |0 |
|ExpiryQueue |ExpiryQueue |0 |0 |0 |0 |0 |
|clientname.someValue |A.B.C.D |0 |0 |1 |0 |1 |
[cpandey@vm254-231 bin]$

Note: A.B.C.D is the Address created and the durable subscription is created as queue clientname.someValue.

• If we read the network dumps using Wireshark, the following is the complete stream:

STOMP
accept-version:1.1
client-id:clientname
login:admin
passcode:admin

.CONNECTED
version:1.1
session:4c98c896
server:ActiveMQ-Artemis/2.4.0.amq-711002-redhat-1 ActiveMQ Artemis Messaging Engine

.
SUBSCRIBE
ack:auto
destination:A.B.C.D
durable-subscription-name:someValue
id:1
subscription-type:MULTICAST

.SEND
content-length:4
destination:A.B.C.D

abcd.MESSAGE
subscription:1
content-length:4
message-id:30
destination:A.B.C.D
expires:0
redelivered:false
priority:4
persistent:false
timestamp:1528858440363

abcd.
DISCONNECT
receipt:6a8bc1fd-0c8b-4e13-871f-fbc9c8c4df9d

.RECEIPT
receipt-id:6a8bc1fd-0c8b-4e13-871f-fbc9c8c4df9d

That’s it. I hope this helps you to have a basic understanding of using the STOMP protocol with the Apache ActiveMQ Artemis or Red Hat AMQ 7.

The post Using the STOMP Protocol with Apache ActiveMQ Artemis Broker appeared first on Red Hat Developer.

Red Hat OpenShift is part of the Cloud Native Computing Foundation (CNCF) Certified Program, ensuring portability and interoperability for your container workloads. This also allows you to use Kubernetes tools to interact with an OpenShift cluster, like kubectl, and you can rest assured that all the APIs you know and love are right there at your fingertips.

The Kubernetes Python client is another great tool for interacting with an OpenShift cluster, allowing you to perform actions on Kubernetes resources with Python code. It also has applications within a cluster. We can configure a Python application running on OpenShift to consume the OpenShift API, and list and create resources. We could then create containerized batch jobs from the running application, or a custom service monitor, for example. It sounds a bit like “OpenShift inception,” using the OpenShift API from services created using the OpenShift API.

In this article, we’ll create a Flask application running on OpenShift. This application will use the Kubernetes Python client to interact with the OpenShift API, list other pods in the project, and display them back to the user.

You’ll need a couple of things to follow along:

• An OpenShift cluster
• A working knowledge of Python

Let’s get started!

Setup

I’ve created a template to alleviate a lot of the boilerplate, so let’s clone it down:

git clone https://github.com/shaneboulden/openshift-client-demo
cd openshift-client-demo

You can create a new app on your OpenShift cluster using the provided template and see the application running:

oc new-app openshift_deploy/ocd.yaml

If you do an oc get routes, you’ll be able to see the route that’s been created. For now, if you select the Pods menu item you’ll just get some placeholder text. We’ll fix this shortly

Configure the Kubernetes Python client

Listing pods is trivial once we have our client configured, and, fortunately, we can use a little Kubernetes Python client magic to configure this easily with the correct service account token.

Usually, we’d configure a Kubernetes client using a kubeconfig file, which has the required token and hostname to create API requests. The Kubernetes Python client also provides a method load_incluster_config(), which replaces the kubeconfig file in a running pod, instead using the available environment variables and mount points to find the service account token and build API URLs from the information available within the pod.

There’s another huge benefit to using load_incluster_config()—our code is now portable. We can take this same application to any Kubernetes cluster, assume nothing about hostnames or network addresses, and easily construct API requests using this awesome little method.

Let’s configure our application to use the load_incluster_config() method. First, we need to import the client and config objects, you can verify this in the ocd.py file:

from kubernetes import client, config

We can now use that magic method to configure the client:

config.load_incluster_config()
v1 = client.CoreV1Api()

That’s it! This is all of the code we need to be able to interact with the OpenShift API from running pods.

Use the Kubernetes Downward API

I’m going to introduce something new here, and yes, it’s another “OpenShift-inception” concept. We’re going to use the list_namespaced_pod method to list pod details; you can find all of the methods available in the documentation. To use this method, we need to pass the current namespace (project) to the Kubernetes client object. But wait, how do we get the namespace for our pod, from inside the running pod?

This is where another awesome Kubernetes API comes into play. It’s called the Downward API and allows us to access metadata about our pod from inside the running pod. To expose information from the Downward API to our pod, we can use environment variables. If you look at the template, you’ll see the following in the ‘env’ section:

- name: POD_NAMESPACE
  valueFrom:
    fieldRef:
      apiVersion: v1
      fieldPath: metadata.namespace

Bring it all together

Now let’s get back to our /pods route in the ocd.py file. The last thing we need to do is to pass the namespace of the app to the Kubernetes client. We have our environment variable configured to use the downward API already, so let’s pass it in:

pods = v1.list_namespaced_pod(namespace=os.environ["POD_NAMESPACE"])

Ensure you’re in the top-level project directory (i.e., you can see the README) and start a build from the local directory:

oc start-build openshift-client-demo --from-dir=.

When you next visit the route and select the Pods menu, you’ll be able to see all of the pods for the current namespace:

I hope you’ve enjoyed this short introduction to the Kubernetes Python client. If you want to explore a little deeper, you can look at creating resources. There’s an example here that looks at creating containerized batch jobs from API POSTs.

The post Use the Kubernetes Python client from your running Red Hat OpenShift pods appeared first on Red Hat Developer.

This article explains how to configure a Python application running within an OpenShift pod to communicate with the Red Hat OpenShift cluster via openshift-restclient-python, the OpenShift Python client.

tl;dr

The code sample below is a sample application running within a pod, which connects to an OpenShift cluster and prints a list of projects:

import os
import sys
import yaml
from kubernetes import client, config
from openshift.dynamic import DynamicClient

config.load_incluster_config()
k8s_config = client.Configuration()
k8s_client = client.api_client.ApiClient(configuration=k8s_config)
dyn_client = DynamicClient(k8s_client)

v1_projects = dyn_client.resources.get(api_version="project.openshift.io/v1", kind="Project")

print(v1_projects.get())

Running locally

Using openshift-restclient-python from your laptop is relatively easy. This OpenShift dynamic client extends the standard Kubernetes Python client. The first step is to install openshift-restclient-python, which will pull in the kubernetes dependency:

$ pip install openshift

Next, we load a Kube configuration. The Kubernetes function new_client_from_config() searches for the file ~/.kube/config. The new_client_from_config() function is similar to the load_kube_config() function, but also returns an ApiClient to be used with any API object. This task allows the caller to concurrently talk with multiple clusters.

The code example below uses the OpenShift dynamic client to list every project the user can access:

#!/usr/bin/env python3

from kubernetes import client, config
from openshift.dynamic import DynamicClient

k8s_client = config.new_client_from_config()
dyn_client = DynamicClient(k8s_client)

v1_projects = dyn_client.resources.get(api_version='project.openshift.io/v1', kind='Project')

project_list = v1_projects.get()

for project in project_list.items:
    print(project.metadata.name)

Running locally after logging into OpenShift works as expected:

oc login -u user https://ocp.lab.example.com

./cmdlineclient.py
ProjectA
ProjectB

Running within Red Hat OpenShift

However, running the same code with an OpenShift pod will result in a TypeError, as shown below:

oc rsh api-gateway-dfs3
cd /opt/app-root/src/
./cmdlineclient.py

Traceback (most recent call last):
  File "./cmdlineclient.py", line 6, in <module>
    k8s_client = config.new_client_from_config()
  File "/opt/app-root/lib/python3.6/site-packages/kubernetes/config/kube_config.py", line 667, in new_client_from_config
    persist_config=persist_config)
  File "/opt/app-root/lib/python3.6/site-packages/kubernetes/config/kube_config.py", line 645, in load_kube_config
    persist_config=persist_config)
  File "/opt/app-root/lib/python3.6/site-packages/kubernetes/config/kube_config.py", line 613, in _get_kube_config_loader_for_yaml_file
    **kwargs)
  File "/opt/app-root/lib/python3.6/site-packages/kubernetes/config/kube_config.py", line 153, in __init__
    self.set_active_context(active_context)
  File "/opt/app-root/lib/python3.6/site-packages/kubernetes/config/kube_config.py", line 173, in set_active_context
    context_name = self._config['current-context']
  File "/opt/app-root/lib/python3.6/site-packages/kubernetes/config/kube_config.py", line 495, in __getitem__
    v = self.safe_get(key)
  File "/opt/app-root/lib/python3.6/site-packages/kubernetes/config/kube_config.py", line 491, in safe_get
    key in self.value):
TypeError: argument of type 'NoneType' is not iterable

Unfortunately, the current documentation provided by openshift-restclient-python is lacking. It does not explain how to connect to OpenShift or Kubernetes from within a pod.

After much searching, I found a section within the Kubernetes documentation which states that when accessing the Kube API from a pod, locating and authenticating the API server are somewhat different. They recommend using one of the official client libraries, which I was already doing. These libraries should automatically discover the API server and authenticate.

The Kubernetes config library also has the function load_incluster_config(). This function uses a combination of environment variables and a token to authenticate the API server. The recommended way to do this is to associate a pod with a service account. When the pod is started, a token for the service account is placed into the filesystem tree of each container in that pod, at /var/run/secrets/kubernetes.io/serviceaccount/token.

That all sounds simple enough. But, before updating the cmdlineclient, we need to create a service account, assign it a role, and then associate it with a pod (via a deployment config). The following instructions outline how to use the oc client to achieve this:

oc create serviceaccount robot

oc policy add-role-to-user admin -z robot

oc patch dc/api-gw --patch '{"spec":{"template":{"spec":{"serviceAccountName": "robot"}}}}'

oc rsh api-gw-9-kzrhn
(app-root) sh-4.2$ ls -al /var/run/secrets/kubernetes.io/serviceaccount/token
lrwxrwxrwx. 1 root root 12 Jul 14 06:13 /var/run/secrets/kubernetes.io/serviceaccount/token -> ..data/token

Now that we confirmed that a token is being injected within the pod, we need to update our function to use load_incluster_config(). However, remember that new_client_from_config() returns an ApiClient. We need to ensure we do this update before passing the ApiClient to the OpenShift dynamic client. Another undocumented step relates to the OpenShift dynamic client requiring a client.configuration object within the Kubernetes ApiClient object.

Lastly, we should also ensure that our code will work both within OpenShift and on our laptop. The updated cmdlineclientv2.py (below) determines if the client is running within OpenShift before calling the load_incluster_config(). It will also fall back to reading ~/.kube/config, which enables the program to run locally:

#!/usr/bin/env python3

import os
import sys

import yaml
from kubernetes import client, config
from openshift.dynamic import DynamicClient

# Check if code is running in OpenShift
if "OPENSHIFT_BUILD_NAME" in os.environ:
    config.load_incluster_config()
    file_namespace = open(
        "/run/secrets/kubernetes.io/serviceaccount/namespace", "r"
    )
    if file_namespace.mode == "r":
        namespace = file_namespace.read()
        print("namespace: %s\n" %(namespace))
else:
    config.load_kube_config()

# Create a client config
k8s_config = client.Configuration()

k8s_client = client.api_client.ApiClient(configuration=k8s_config)
dyn_client = DynamicClient(k8s_client)

v1_projects = dyn_client.resources.get(api_version="project.openshift.io/v1", kind="Project")

project_list = v1_projects.get()

for project in project_list.items:
    print("Project Name: %s" % (project.metadata.name))

When running cmdlineclientv2, notice that although we have assigned the admin role to the service account, it is only admin within the ProjectA namespace:

./cmdlineclientv2.py
namespace: ProjectA

Project Name: ProjectA

I hope you found this post helpful. Comments and questions are welcome.

The post Controlling Red Hat OpenShift from an OpenShift pod appeared first on Red Hat Developer.

#384 – SEPTEMBER 3, 2019
View in Browser »

Become a Python Guru With PyCharm

PyCharm is the Python IDE for Professional Developers by JetBrains providing a complete set of tools for productive Python, Web and scientific development. Be more productive and save time while PyCharm takes care of the routine →
JETBRAINSsponsor

Discussions

Python Jobs

Python Web Developer (Remote)

Premiere Digital Services

Senior Backend Software Engineer (Remote)

Close

Senior Python Developer (Austin, TX)

InQuest

Backend and DataScience Engineers (London, Relocation & Visa Possible)

Citymapper Ltd

Senior Software Developer (Edmonton, Canada)

Levven Electronics Ltd.

More Python Jobs >>>

Articles & Tutorials

Eliminate Python Package Management

Tired of dealing with package dependency conflicts? Sign up for a free ActiveState Platform account to download ActivePython pre-bundled with top packages, or customize your own Python build with automated dependency resolution.
ACTIVESTATEsponsor

Projects & Code

Events

Happy Pythoning!
This was PyCoder’s Weekly Issue #384.
View in Browser »

[ Subscribe to 🐍 PyCoder’s Weekly 💌 – Get the best Python news, articles, and tutorials delivered to your inbox once a week >> Click here to learn more ]

In this guest post Mridu shares with us how she got into public speaking, conquering fear and imposter syndrome. What did she learn? And what does she recommend for people starting out? Let's hear from Mridu!

I am everyday thankful for having crossed paths with individuals who are passionate to share their experience with others. Inspiration came from a teacher who loves to teach with energy and enthusiasm, making sure the student actually learned. Oftentimes I felt our teacher's enthusiasm was more than the collective enthusiasm of our batch of 200 students.

How it started

In one of the first community workshops, I crossed paths with another person who loves to teach probably every skill he has.

His positive influence helped me figure out a list of traits I wanted to possess. Traits, which neither a book, nor a podcast or school would ever be able to teach me. They could only be learned by practice and experience. Hence I needed to learn from him!

He shared a video on what might be holding you back. It's actually always our own thoughts that do, self-made stories of what we can do, and (above all) what we cannot do. I learned an important lesson that day!

Follow your heart. Have no apprehensions, leap

This was a quote that was shared as well. It was on me though to actually do something with it! It was August 2018 when I truly felt I should give back to the community by becoming a speaker at community events. I could do that by putting small amounts of efforts in to bridge the gap between academics and the corporate world. I could contribute to something I had seen my professor doing. Feeling like doing something is one thing, doing it though is a totally different ball game. We all have our fears. Even imagining being on the other side talking about tech was nothing less than a big leap!

Imposter Syndrome?

I am no different. The haunting feeling that every other person around me knows more than me, new tech concepts being overwhelming, stupid self-made stories, the list goes on and on. It's good to accept this, talk about it with someone you trust. Listen to a different perspective. To make sure fear was not going to hold me back, I repeated the same statement over and over again: "Have no apprehensions, leap." ... "Have no apprehensions, leap."

Talk Proposal

Just for a moment, leaving all "what if" questions aside, exploring the communities around me, I sent a talk proposal titled "Introduction to APIs" to one of them. Little did I know about how to write a proposal so I just jotted down the ideas I had. While doing so again this feeling nagged me: "This topic seems vast, what should I talk about?". I procrastinated.

A couple of weeks before the deadline I was opening a talk issue on GitHub for the PyData Delhi Meetup, I did a write-up on Virtual Environment in Python. I was fortunate that a colleague reached out to me that I must also include virtualenvwrapper, and a cousin recommended I'd cover Pipenv. With Introduction to APIs left unfinished, I opened a new issue for this topic. The talk got scheduled, so now the challenge was to come up with the full presentation on the subject!

A part of the initial write-up would be part of the presentation together with the ideas I received from my readers. Things were still not very intuitive though. All I knew was, when stuck, ask. So, I reached out to the person I met at a community workshop, sharing my ideas. I took his inputs and finished preparing the presentation. He reviewed the flow and gave me tips on how I needed to practice.

My first tech talk

The day came, December 2018, my first tech talk at a community meetup! Luckily positive influence won over the feeble voice deep inside of me which said I should fear the stage. I knew this would be the opportunity to pay it forward. This talk would teach me the traits I always wanted to have. This knowledge was thousand times more powerful than letting fear hold me back. My energy, enthusiasm and happiness during the event was different than usual. I had fun and from then on I started volunteering as a speaker every weekend, grabbing almost every opportunity I could.

8-9 months of weekend community meetups in, I submitted my first proposal to a technical conference (PyData Delhi 2019). And I got selected. It was not my goal from the start to deliver a talk at such a big conference. It happened the way it did though.

Why is it worth getting over the fear and just get started?

1. Climbing our own imaginary mountains is a never ending pursuit. Bringing others along is what makes the journey beautiful. Being able to help others even in a very small way makes you feel good about yourself.

2. Befriend the imposter syndrome. No one knows it all, I managed to learn and accept that. I can share my learning with others and learn from others.

3. You will be valued for your contribution. Out of the blue you will get some random messages of goodwill.

4. 40 minutes are all yours to experiment. Talk about something you would love to share. Expect smiling faces after your session, thank you’s and happiness overall.

5. You cannot make an impact on everyone in the classroom. This wisdom somebody I look up to shared with me. Can you make an impact on at least one person who actually needs it? This alone brings great satisfaction.

6. Feedback always helps improving yourself, and make you prepare better for your next presentation.

7. When you make an attempt in explaining a concept to the rest you end up simplifying it in every possible manner so that it is easier to understand. And in that process you will actually learn a lot yourself.

8. Questions are all about visualising one concept from different angles. They make you understand the knowledge gaps you have. This is something that is often overlooked when working in isolation.

Answers to questions that might hold you back

1. What if someone asks me a question and I don’t know the answer?

   • Accept it. No one knows it all.

2. How will I manage the ones who have come to attend?

   • Here is the truth. One question that I am often asked is: "Are you really able to manage the class?". The answer is, there is nothing I actually need to manage. Unlike the traditional learning environments, meetups are places where only the ones enthusiastic to learn hang out.

3. I have never spoken before. What if I am unable to do it?

   • "For everything there is a first time. But you need to get started!" - PyBites' founders Bob and Julian told me this the first time I was hesitant to write my first guest post for them. I have followed this piece of advice ever since. If a full-blown 40 minutes presentation seems intimidating, maybe a lightning talk of 10-15 minutes may be a better place to start. Remember, the community will be supportive, no one is perfect, and everyone is there to learn!

4. What if the topic of my talk is too obvious, something everyone already knows?

   • I read somewhere "What might be obvious to you, might be alien to someone else". So, don’t decide on your own, get feedback from others and then decide.

5. I have no friends, or a group to go to a conference/meetup with? Will going alone be awkward?

   • Your interest and your friend’s interest might be in sync, or not. Each one of us has different priorities. Go and meet like-minded people.

Conclusion

Looking back, all I can say is that my journey so far is the result of guidance, encouragement and different perspectives I received throughout the process.

I have been blessed to be surrounded by people who loved to teach. Try to listen to these kind of people, then practice, learn and teach!

Moving forward, I will be working on the feedback I get, perform more speaking engagements, and help organizing Pyladies Delhi and DjangoGirls Indore. I hope to bring smiles on a couple of more faces and of course to learn more tech. :o)

Keep Calm and Code in Python!

-- Mridu

Want to improve your Python fluency? There’s no better way than doing exercises, practicing your Python skills and ensuring you understand how to solve problems in the best possible way.

My book, “Python Workout,” not only contains 50 of my favorite exercises from my Python training business (at some of the world’s best-known companies), but is on sale today as Manning’s “Deal of the Day,” together with two other Python related books.

Don’t miss this opportunity to improve your Python skills at a discount! Go to https://www.manning.com/dotd to take advantage of the sale price.

The post “Python Workout” is today’s Manning “Deal of the day” appeared first on Reuven Lerner.

The post QuantumBlack joins NumFOCUS Corporate Sponsors appeared first on NumFOCUS.

We are pleased to announce the first batch of cut videos from EuroPython 2019 in Basel, Switzerland.

EuroPython 2019 YouTube Channel

In this batch, we have included all videos for Wednesday, July 10 2019, the first conference day.

In the coming two weeks we will publish videos for the next two conference days. In total, we will have more than 130 videos available for you to watch.

All EuroPython videos, including the ones from previous conferences, are available on our EuroPython YouTube Channel.

Enjoy,
–
EuroPython 2019 Team
https://ep2019.europython.eu/
https://www.europython-society.org/

Sometimes, when you look at a function definition in Python, you might see that it takes two strange arguments: *args and **kwargs. If you’ve ever wondered what these peculiar variables are, or why your IDE defines them in main(), then this article is for you. You’ll learn how to use args and kwargs in Python to add more flexibility to your functions.

By the end of the article, you’ll know:

• What *args and **kwargs actually mean
• How to use *args and **kwargs in function definitions
• How to use a single asterisk (*) to unpack iterables
• How to use two asterisks (**) to unpack dictionaries

This article assumes that you already know how to define Python functions and work with lists and dictionaries.

Passing Multiple Arguments to a Function

*args and **kwargs allow you to pass multiple arguments or keyword arguments to a function. Consider the following example. This is a simple function that takes two arguments and returns their sum:

defmy_sum(a,b):returna+b

This function works fine, but it’s limited to only two arguments. What if you need to sum a varying number of arguments, where the specific number of arguments passed is only determined at runtime? Wouldn’t it be great to create a function that could sum all the integers passed to it, no matter how many there are?

Using the Python args Variable in Function Definitions

There are a few ways you can pass a varying number of arguments to a function. The first way is often the most intuitive for people that have experience with collections. You simply pass a list or a set of all the arguments to your function. So for my_sum(), you could pass a list of all the integers you need to add:

# sum_integers_list.pydefmy_sum(my_integers):result=0forxinmy_integers:result+=xreturnresultlist_of_integers=[1,2,3]print(my_sum(list_of_integers))

This implementation works, but whenever you call this function you’ll also need to create a list of arguments to pass to it. This can be inconvenient, especially if you don’t know up front all the values that should go into the list.

This is where *args can be really useful, because it allows you to pass a varying number of positional arguments. Take the following example:

# sum_integers_args.pydefmy_sum(*args):result=0# Iterating over the Python args tupleforxinargs:result+=xreturnresultprint(my_sum(1,2,3))

In this example, you’re no longer passing a list to my_sum(). Instead, you’re passing three different positional arguments. my_sum() takes all the parameters that are provided in the input and packs them all into a single iterable object named args.

Note that args is just a name. You’re not required to use the name args. You can choose any name that you prefer, such as integers:

# sum_integers_args_2.pydefmy_sum(*integers):result=0forxinintegers:result+=xreturnresultprint(my_sum(1,2,3))

The function still works, even if you pass the iterable object as integers instead of args. All that matters here is that you use the unpacking operator (*).

Bear in mind that the iterable object you’ll get using the unpacking operator * is not a list but a tuple. A tuple is similar to a list in that they both support slicing and iteration. However, tuples are very different in at least one aspect: lists are mutable, while tuples are not. To test this, run the following code. This script tries to change a value of a list:

# change_list.pymy_list=[1,2,3]my_list[0]=9print(my_list)

The value located at the very first index of the list should be updated to 9. If you execute this script, you will see that the list indeed gets modified:

$ python change_list.py
[9, 2, 3]

The first value is no longer 0, but the updated value 9. Now, try to do the same with a tuple:

# change_tuple.pymy_tuple=(1,2,3)my_tuple[0]=9print(my_tuple)

Here, you see the same values, except they’re held together as a tuple. If you try to execute this script, you will see that the Python interpreter returns an error:

$ python change_tuple.py
Traceback (most recent call last):  File "change_tuple.py", line 3, in <module>    my_tuple[0] = 9TypeError: 'tuple' object does not support item assignment

This is because a tuple is an immutable object, and its values cannot be changed after assignment. Keep this in mind when you’re working with tuples and *args.

Using the Python kwargs Variable in Function Definitions

Okay, now you’ve understood what *args is for, but what about **kwargs? **kwargs works just like *args, but instead of accepting positional arguments it accepts keyword (or named) arguments. Take the following example:

# concatenate.pydefconcatenate(**kwargs):result=""# Iterating over the Python kwargs dictionaryforarginkwargs.values():result+=argreturnresultprint(concatenate(a="Real",b="Python",c="Is",d="Great",e="!"))

When you execute the script above, concatenate() will iterate through the Python kwargs dictionary and concatenate all the values it finds:

$ python concatenate.py
RealPythonIsGreat!

Like args, kwargs is just a name that can be changed to whatever you want. Again, what is important here is the use of the unpacking operator (**).

So, the previous example could be written like this:

# concatenate_2.pydefconcatenate(**words):result=""forarginwords.values():result+=argreturnresultprint(concatenate(a="Real",b="Python",c="Is",d="Great",e="!"))

Note that in the example above the iterable object is a standard dict. If you iterate over the dictionary and want to return its values, like in the example shown, then you must use .values().

In fact, if you forget to use this method, you will find yourself iterating through the keys of your Python kwargs dictionary instead, like in the following example:

# concatenate_keys.pydefconcatenate(**kwargs):result=""# Iterating over the keys of the Python kwargs dictionaryforarginkwargs:result+=argreturnresultprint(concatenate(a="Real",b="Python",c="Is",d="Great",e="!"))

Now, if you try to execute this example, you’ll notice the following output:

$ python concatenate_keys.py
abcde

As you can see, if you don’t specify .values(), your function will iterate over the keys of your Python kwargs dictionary, returning the wrong result.

Ordering Arguments in a Function

Now that you have learned what *args and **kwargs are for, you are ready to start writing functions that take a varying number of input arguments. But what if you want to create a function that takes a changeable number of both positional and named arguments?

In this case, you have to bear in mind that order counts. Just as non-default arguments have to precede default arguments, so *args must come before **kwargs.

To recap, the correct order for your parameters is:

1. Standard arguments
2. *args arguments
3. **kwargs arguments

For example, this function definition is correct:

# correct_function_definition.pydefmy_function(a,b,*args,**kwargs):pass

The *args variable is appropriately listed before **kwargs. But what if you try to modify the order of the arguments? For example, consider the following function:

# wrong_function_definition.pydefmy_function(a,b,**kwargs,*args):pass

Now, **kwargs comes before *args in the function definition. If you try to run this example, you’ll receive an error from the interpreter:

$ python wrong_function_definition.py
  File "wrong_function_definition.py", line 2    def my_function(a, b, **kwargs, *args):                                    ^SyntaxError: invalid syntax

In this case, since *args comes after **kwargs, the Python interpreter throws a SyntaxError.

Unpacking With the Asterisk Operators: *& **

You are now able to use *args and **kwargs to define Python functions that take a varying number of input arguments. Let’s go a little deeper to understand something more about the unpacking operators.

The single and double asterisk unpacking operators were introduced in Python 2. As of the 3.5 release, they have become even more powerful, thanks to PEP 448. In short, the unpacking operators are operators that unpack the values from iterable objects in Python. The single asterisk operator * can be used on any iterable that Python provides, while the double asterisk operator ** can only be used on dictionaries.

Let’s start with an example:

# print_list.pymy_list=[1,2,3]print(my_list)

This code defines a list and then prints it to the standard output:

$ python print_list.py
[1, 2, 3]

Note how the list is printed, along with the corresponding brackets and commas.

Now, try to prepend the unpacking operator * to the name of your list:

# print_unpacked_list.pymy_list=[1,2,3]print(*my_list)

Here, the * operator tells print() to unpack the list first.

In this case, the output is no longer the list itself, but rather the content of the list:

$ python print_unpacked_list.py
1 2 3

Can you see the difference between this execution and the one from print_list.py? Instead of a list, print() has taken three separate arguments as the input.

Another thing you’ll notice is that in print_unpacked_list.py, you used the unpacking operator * to call a function, instead of in a function definition. In this case, print() takes all the items of a list as though they were single arguments.

You can also use this method to call your own functions, but if your function requires a specific number of arguments, then the iterable you unpack must have the same number of arguments.

To test this behavior, consider this script:

# unpacking_call.pydefmy_sum(a,b,c):print(a+b+c)my_list=[1,2,3]my_sum(*my_list)

Here, my_sum() explicitly states that a, b, and c are required arguments.

If you run this script, you’ll get the sum of the three numbers in my_list:

$ python unpacking_call.py
6

The 3 elements in my_list match up perfectly with the required arguments in my_sum().

Now look at the following script, where my_list has 4 arguments instead of 3:

# wrong_unpacking_call.pydefmy_sum(a,b,c):print(a+b+c)my_list=[1,2,3,4]my_sum(*my_list)

In this example, my_sum() still expects just three arguments, but the * operator gets 4 items from the list. If you try to execute this script, you’ll see that the Python interpreter is unable to run it:

$ python wrong_unpacking_call.py
Traceback (most recent call last):  File "wrong_unpacking_call.py", line 6, in <module>    my_sum(*my_list)TypeError: my_sum() takes 3 positional arguments but 4 were given

When you use the * operator to unpack a list and pass arguments to a function, it’s exactly as though you’re passing every single argument alone. This means that you can use multiple unpacking operators to get values from several lists and pass them all to a single function.

To test this behavior, consider the following example:

# sum_integers_args_3.pydefmy_sum(*args):result=0forxinargs:result+=xreturnresultlist1=[1,2,3]list2=[4,5]list3=[6,7,8,9]print(my_sum(*list1,*list2,*list3))

If you run this example, all three lists are unpacked. Each individual item is passed to my_sum(), resulting in the following output:

$ python sum_integers_args_3.py
45

There are other convenient uses of the unpacking operator. For example, say you need to split a list into three different parts. The output should show the first value, the last value, and all the values in between. With the unpacking operator, you can do this in just one line of code:

# extract_list_body.pymy_list=[1,2,3,4,5,6]a,*b,c=my_listprint(a)print(b)print(c)

In this example, my_list contains 6 items. The first variable is assigned to a, the last to c, and all other values are packed into a new list b. If you run the script, print() will show you that your three variables have the values you would expect:

$ python extract_list_body.py
1[2, 3, 4, 5]6

Another interesting thing you can do with the unpacking operator * is to split the items of any iterable object. This could be very useful if you need to merge two lists, for instance:

# merging_lists.pymy_first_list=[1,2,3]my_second_list=[4,5,6]my_merged_list=[*my_first_list,*my_second_list]print(my_merged_list)

The unpacking operator * is prepended to both my_first_list and my_second_list.

If you run this script, you’ll see that the result is a merged list:

$ python merging_lists.py
[1, 2, 3, 4, 5, 6]

You can even merge two different dictionaries by using the unpacking operator **:

# merging_dicts.pymy_first_dict={"A":1,"B":2}my_second_dict={"C":3,"D":4}my_merged_dict={**my_first_dict,**my_second_dict}print(my_merged_dict)

Here, the iterables to merge are my_first_dict and my_second_dict.

Executing this code outputs a merged dictionary:

$ python merging_dicts.py
{'A': 1, 'B': 2, 'C': 3, 'D': 4}

Remember that the * operator works on any iterable object. It can also be used to unpack a string:

# string_to_list.pya=[*"RealPython"]print(a)

In Python, strings are iterable objects, so * will unpack it and place all individual values in a list a:

$ python string_to_list.py
['R', 'e', 'a', 'l', 'P', 'y', 't', 'h', 'o', 'n']

The previous example seems great, but when you work with these operators it’s important to keep in mind the seventh rule of The Zen of Python by Tim Peters: Readability counts.

To see why, consider the following example:

# mysterious_statement.py*a,="RealPython"print(a)

There’s the unpacking operator *, followed by a variable, a comma, and an assignment. That’s a lot packed into one line! In fact, this code is no different from the previous example. It just takes the string RealPython and assigns all the items to the new list a, thanks to the unpacking operator *.

The comma after the a does the trick. When you use the unpacking operator with variable assignment, Python requires that your resulting variable is either a list or a tuple. With the trailing comma, you have actually defined a tuple with just one named variable a.

While this is a neat trick, many Pythonistas would not consider this code to be very readable. As such, it’s best to use these kinds of constructions sparingly.

Conclusion

You are now able to use *args and **kwargs to accept a changeable number of arguments in your functions. You have also learned something more about the unpacking operators.

You’ve learned:

• What *args and **kwargs actually mean
• How to use *args and **kwargs in function definitions
• How to use a single asterisk (*) to unpack iterables
• How to use two asterisks (**) to unpack dictionaries

If you still have questions, don’t hesitate to reach out in the comments section below! To learn more about the use of the asterisks in Python, have a look at Trey Hunner’s article on the subject.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

The post Lucid Programming Podcast – Writing Books About Python appeared first on The Mouse Vs. The Python.

We are pleased to announce that the September 2019 release of the Python Extension for Visual Studio Code is now available. You can download the Python extension from the Marketplace, or install it directly from the extension gallery in Visual Studio Code. If you already have the Python extension installed, you can also get the latest update by restarting Visual Studio Code. You can learn more about  Python support in Visual Studio Code in the documentation.  

This was a short release where we closed 35 issues, including improvements to the Python Language Server and to Jupyter Notebook cell debugging, as well as detection of virtual environment creation. The full list of enhancements is listed in our changelog. 

Improvements to Python Language Server 

The Python Language Server now has linting capabilities, and its latest release includes new linting messages and a variety of additional general improvements, which are listed in the section “Other changes and enhancements” below.   

The linting messages provided by the Python Language Server include detecting unresolved imports, undefined variables, too many arguments in a function call, unknown keyword arguments and inheriting from something that is not a class. To see the full detailed list of linting messages, you can check the documentation in the Language Server GitHub repo or the settings reference page within the Python for Visual Studio Code docs. 

We’ve also added general #noqa support, so linting messages can be disabled on a case by case basis. Lines with a #noqa comment will have their diagnostic output suppressed. For more information, you can check the documentation. 

Improvements to Jupyter Notebook cell debugging  

In the August release, we added the ability to debug Jupyter notebook cells where you can step into user code. In this release, this feature is enhanced with the option to also step into non-user code if needed. To enable, open the settings page (File > Preferences > Settings), search for “Data Science: Debug Just My Code” and uncheck the option. 

Once the setting is disabled, you’ll be able to step into function calls and, for example, inspect the non-user code behavior and how variables change when it’s being executed.  

Detection of virtual environment creation  

The Python interpreter displayed on the status bar indicates which environment the Python extension is using for running Python code (using the Python: Run Python File in Terminal command, for example), and to provide language services such as auto-completion, syntax checking, linting, formatting, etc:

In this release, when a new virtual environment is created, a prompt will be displayed asking if you’d like to select its interpreter for the workspace:  

This will add the path to the Python interpreter from the new virtual environment to your workspace settings, and therefore that environment will be used when installing packages and running code through the Python extension.  

Other Changes and Enhancements 

We have also added small enhancements and fixed issues requested by users that should improve your experience working with Python in Visual Studio Code. Some notable changes include: 

• Update Jedi to 0.15.1 and parso to 0.5.1. (#6294) 
• Bump version of PTVSD to 4.3.2. 
• Added a setting to allow Python code to be executed when the interactive window is loading. (#6842) 
• Add debug command code lenses when in debug mode. (#6672) 
• General Improvements for the Python Language Server: 

• • Improved handling of generic classes in inheritance chains (#1278) 
  • Added support for TypeVar bound and generic self (#1242) 
  • Added support for forward references in type strings (#1186) 
  • Added goto definition for members in class bases (#1356, #1443) 
  • Improved assignment handling (#1457, #1494, #411, #1382)  

We are continuing to A/B test new features. If you see something different that was not announced by the team, you may be part of an experiment! To see if you are part of an experiment, you can check the first lines in the Python extension output channel. If you wish to opt-out from A/B testing, disable telemetry in Visual Studio Code.  

Be sure to download the Python extension for Visual Studio Code now to try out the above improvements. If you run into any problems, please file an issue on the Python VS Code GitHub page. 

The post Python in Visual Studio Code – September 2019 Release appeared first on Python.

Introduction

Object detection is a technology that falls under the broader domain of Computer Vision. It deals with identifying and tracking objects present in images and videos. Object detection has multiple applications such as face detection, vehicle detection, pedestrian counting, self-driving cars, security systems, etc.

The two major objectives of object detection include:

• To identify all objects present in an image
• Filter out the object of attention

In this article, you will see how to perform object detection in Python with the help of the ImageAI library.

Deep Learning for Object Detection

Deep learning techniques have been proven state of the art for various object detection problems. The following are some of the commonly used deep learning approaches for object detection:

• ImageAI
• Single Shot Detectors
• YOLO (You only look once)
• Region-based Convolutional Neural Networks

In the rest of this article, we will see what exactly ImageAI is and how to use it to perform object detection.

ImageAI

ImageAI is a Python library built to empower developers to build applications and systems with self-contained deep learning and Computer Vision capabilities using a few lines of straight forward code. ImageAI contains a Python implementation of almost all of the state-of-the-art deep learning algorithms like RetinaNet, YOLOv3, and TinyYOLOv3.

ImageAI makes use of several APIs that work offline - it has object detection, video detection, and object tracking APIs that can be called without internet access. ImageAI makes use of a pre-trained model and can easily be customized.

The ObjectDetection class of the ImageAI library contains functions to perform object detection on any image or set of images, using pre-trained models. With ImageAI, you can detect and recognize 80 different kinds of common, everyday objects.

Setting up your Environment

In this part of the tutorial, we will work through the installation of ImageAI.

To use ImageAI you need to install a few dependencies. The first step is to have Python installed on your computer. Download and install Python 3 from the official Python website.

Once you have Python installed on your computer, install the following dependencies using pip:

TensorFlow

$ pip install tensorflow

OpenCV

$ pip install opencv-python

Keras

$ pip install keras

ImageAI

$ pip install imageAI

Now download the TinyYOLOv3 model file that contains the classification model that will be used for object detection.

Performing Object Detection with ImageAI

Now let's see how to actually use the ImageAI library. I'll explain step by step how you can build your first object detection model with ImageAI.

Step 1

Our first task here is to create the necessary folders. For this tutorial we need the following folders:

• Object detection: root folder
• models: stores pre-trained model
• input: stores image file on which we want to perform object detection
• output: stores image file with detected objects

After you have created your folders, your Object detection folder should have the following sub-folders:

├── input
├── models
└── output

3 directories, 0 files

Step 2

Open your preferred text editor for writing Python code and create a new file detector.py.

Step 3

Import ObjectDetection class from the ImageAI library.

from imageai.Detection import ObjectDetection

Step 4

Now that you have imported imageAI library and the ObjectDetection class , the next thing is to create an instance of the class ObjectDetection, as shown here:

detector = ObjectDetection()

Step 5

Let's specify the path from our input image, output image, and model.

model_path = "./models/yolo-tiny.h5"
input_path = "./input/test45.jpg"
output_path = "./output/newimage.jpg"

Step 6

After instantiating the ObjectDetection class we can now call various functions from the class. The class contains the following functions to call pre-trained models: setModelTypeAsRetinaNet(), setModelTypeAsYOLOv3(), and setModelTypeAsTinyYOLOv3().

For the purpose of this tutorial, I'll be using the pre-trained TinyYOLOv3 model, and hence we will use the setModelTypeAsTinyYOLOv3() function to load our model.

detector.setModelTypeAsTinyYOLOv3()

Step 7

Next, we are going to call the function setModelPath(). This function accepts a string which contains the path to the pre-trained model:

detector.setModelPath(model_path)

Step 8

This step calls the function loadModel() from the detector instance. It loads the model from the path specified above using the setModelPath() class method.

detector.loadModel()

Step 9

To detect objects in the image, we need to call the detectObjectsFromImage function using the detector object that we created in the previous section.

This function requires two arguments: input_image and output_image_path. input_image is the path where the image we are detecting is located, while the output_image_path parameter is the path to store the image with detected objects. This function returns a dictionary which contains the names and percentage probabilities of all the objects detected in the image.

detection = detector.detectObjectsFromImage(input_image=input_path, output_image_path=output_path)

Step 10

The dictionary items can be accessed by traversing through each item in the dictionary.

for eachItem in detection:
    print(eachItem["name"] , " : ", eachItem["percentage_probability"])

Complete Code for Object Detection

Here is the complete code for the image detection:

from imageai.Detection import ObjectDetection

detector = ObjectDetection()

model_path = "./models/yolo-tiny.h5"
input_path = "./input/test45.jpg"
output_path = "./output/newimage.jpg"

detector.setModelTypeAsTinyYOLOv3()
detector.setModelPath(model_path)
detector.loadModel()
detection = detector.detectObjectsFromImage(input_image=input_path, output_image_path=output_path)

for eachItem in detection:
    print(eachItem["name"] , " : ", eachItem["percentage_probability"])

In the output, you can see the name of each detected object along with its percentage probability as shown below:
Output

car  :  54.72719073295593
car  :  58.94589424133301
car  :  62.59384751319885
car  :  74.07448291778564
car  :  91.10507369041443
car  :  97.26507663726807
car  :  97.55765795707703
person  :  53.6459743976593
person  :  56.59831762313843
person  :  72.28181958198547

Original image:

The original image, i.e. "test45", looked like this:

Image with Object Detection:

After the object detection, the resulting image looks like this:

You can see that ImageAI has successfully identified cars and persons in the image.

Conclusion

Object detection is one of the most common computer vision tasks. This article explains how to perform object detection in Python using the ImageAI library with the help of an example.

References

• https://imageai.readthedocs.io/en/latest/detection/index.html