import sys
from PyQt4 import QtGui
from PyQt4 import QtCore

class fileDialogSample(QtGui.QWidget):
    def __init__(self):
        QtGui.QWidget.__init__(self)

        #make text area and button
        self.my_text_edit = QtGui.QTextEdit()
        self.my_button = QtGui.QPushButton('Select File', self)

        #open the showDialog
        self.my_button.clicked.connect(self.showDialog)

        #put all into application area
        vbox = QtGui.QVBoxLayout()
        vbox.addWidget(self.my_text_edit)
        vbox.addWidget(self.my_button)
        self.setLayout(vbox)

        #set title and geometry of application
        self.setWindowTitle('File Dialog example')
        self.setGeometry(50, 50, 300, 300)

    #make a function with seeting for my QFileDialog
    def showDialog(self):
        file_name = QtGui.QFileDialog.getOpenFileName(self, 'Open file', 'C://')
        my_file_open = open(file_name)
        data = my_file_open.read()
        self.my_text_edit.setText(data)
        my_file_open.close()

#run the application 
app = QtGui.QApplication(sys.argv)
my_dialog = fileDialogSample()
my_dialog.show()
sys.exit(app.exec_())

So one of the things I've been working on recently is creating a docker-based container to host a complex application for video processing that requires a lot of OS-level setup. I, of course, work on a number of Ubuntu 17.04 machines for development, and docker there has proven to be exactly what you expect. You create a Dockerfile, and if it works here, it works there, no muss, no fuss. It does what it says on the box. Honestly it's been a pleasure to work with.

Then I went to run the containers on Centos7 hosts.

Note that this is about 90% of the value proposition of Docker, portable containers you can ship to a different host.

For those who haven't done this the default Centos7 install is an XFS filesystem. Yay for progress and all that. Oh, and the default on Centos7 for docker is to use overlayfs which is compatible with XFS, except it's only compatible with XFS if you happened to setup the non-default ftype=1 flag when creating the filesystem, but it doesn't check for that flag. Queue up much gnashing of teeth as the docker build process becomes a wonderful world of pain and suffering with RPM db corruptions, random failures to delete directories, and generally ridiculously non-predictable behaviour.

But, you say, overlayfs + XFS ftype=0 is an unsupported configuration. True, I might answer, but dang it, it's the default install of the tool on a default install of the OS. At the very least, refuse to run if you're on a configuration which is just going to fall over and die in exotic and hard-to-debug ways. Make the default the slow-as-dirt loopback fs if you are running against XFS with the wrong ftype, better that things run slow than you violate the entire *point* of the framework. We use docker to have a stable, reliable way to replicate work across environments, if that's not possible, error out and report the failure, or choose a reliable-but-slow alternative.

Apparently the solution is to have a separate set of block devices that you use for docker, and because that's a royal PITA to configure the default is to do silly things. Sigh.

C:\Python27\Scripts>pip install speech
Collecting speech
  Downloading speech-0.5.2.tar.gz
Installing collected packages: speech
  Running setup.py install for speech ... done
Successfully installed speech-0.5.2

>>> dir(speech)
['Listener', '_ListenerBase', '_ListenerCallback', '__builtins__', '__doc__', '__file__', '__name__', '__package__'
, '_constants', '_ensure_event_thread', '_eventthread', '_handlerqueue', '_listeners', '_recognizer', 
'_startlistening', '_voice', 'gencache', 'input', 'islistening', 'listenfor', 'listenforanything', 'pythoncom',
 'say', 'stoplistening', 'thread', 'time', 'win32com']>>> help(speech)
Help on module speech:

NAME
    speech - speech recognition and voice synthesis module.

FILE
    c:\python27\lib\site-packages\speech.py

DESCRIPTION
    Please let me know if you like or use this module -- it would make my day!

    speech.py: Copyright 2008 Michael Gundlach  (gundlach at gmail)
    License: Apache 2.0 (http://www.apache.org/licenses/LICENSE-2.0)

    For this module to work, you'll need pywin32 (http://tinyurl.com/5ezco9
    for Python 2.5 or http://tinyurl.com/5uzpox for Python 2.4) and
    the Microsoft Speech kit (http://tinyurl.com/zflb).


    Classes:
        Listener: represents a command to execute when phrases are heard.

    Functions:
        say(phrase): Say the given phrase out loud.
        input(prompt, phraselist): Block until input heard, then return text.
        stoplistening(): Like calling stoplistening() on all Listeners.
        islistening(): True if any Listener is listening.
        listenforanything(callback): Run a callback when any text is heard.
        listenfor(phraselist, callback): Run a callback when certain text is heard.

>>> speech.say('Hello Catalin George')

Bodhi mash

This is mostly Fedora release engineering work specific post. Feel free to skip if you are not into Fedora land.

I started packaging for Fedora from 2006, back in Fedora Extras repository days. Now, we have only one big repo. Packagers can just submit their new packages or updates to this repository. Now, to do so, first, we will have to get the package reviewed (in the case of new packages). Later, after getting a git repo for the spec file, one can build the package in Koji, this will create the RPMs and the SRPM in the koji server. Next, we mark the package for a push to the testing repository (or after certain criteria, mark them to push to the stable repository). For rawhide (the latest of everything) repository, we don’t have to do this, everything we build for rawhide will be automatically pushed.

What happens after we mark the package ready for push?

Every day, someone from the Fedora release engineering team is on push duty. This person calls bodhi-push commands, which in turn generates a fedmsg, which then consumed by a the consumer, defined in bodhi masher.py to generate new repos from the updates, and also mark the packages properly in koji. It also composes the ostree tree. We will now dig into the source of this command to see what all happens.

The work starts at line 271 At line 290, it initiates the state. There are use cases when it resumes from the middle of a previous mash, the if-else block at line 301 helps to load or save the state of the mash accordingly. To load or save it actually reads up a JSON file and a few other steps.

Next, at line 306, it calls a function load_updates. This function talks to the database and finds out the list of packages which need to be pushed (the final repo will be generated from all the packages in that particular tag) for updates. Now all of this work happens based on a unique tag, like f25-updates. The verify_updates function makes sure that list of the updates has the right updates for the current tag (it can happen that someone tries to mark an F24 build as the F25 update).

We have to do some more checks if this push is for stable repositories, like if the build has enough positive karma, or has it spent 7 days in the testing repository. At line 310, the perform_gating function does these gating checks.

At line 312, in the determine_and_perform_tag_actions functions, we talk to Koji and update the tags in Koji as required. Either it adds a new tag or moves the build to a new tag. Like, from candidate to the testing. After that we update the bugzilla bug entries if there are any update with a bug associated, and in the next few lines we remove any extra tags, and also update the comps files (from the git repo). At line 321, we create a new thread which does the real mash work using the mash command from the package with the same name. While we wait for this thread to finish on line 330, we have already created some digest mail information, and other updateinfo (in the uinfo variable, which is the content of updateinfo.xml). In line 337 we check if we are supposed to do build ostree repos for atomic, if yes, then we call the compose_atomic_trees function.

Then we sync the newly generated repo with the master mirror, and wait for it to finish (on line 345). After that it sends out fedmsg notifications, modifies the bugs in the bugzilla, adds comments to the koji updates, and also does the announcement emails (the function calls are documented).

The main reason I wanted to write this post is the upcoming change where we will stop using the mash command, and instead call pungi to do the work. As pungi can also do ostree tree compose.

I want to specially thank Patrick who helped me to understand this piece of code (as he does for many other things).

# dumb recursive factorial
def fact(n):
    if (n <= 1):
        return 1
    else:
        return n * fact(n - 1)

print(fact(6))

# find primes using a for-else construct
for n in range(2, 10):
    x_range = range(2, n)
    for x in x_range:
        if n % x == 0:
            break
    else:
        # loop fell through without finding a factor
        print(n)

prefix = "Hello "

n1 = raw_input("Enter your name")

n2 = raw_input("Enter another name")

res = prefix + n1 + " and " + n2
print(res)

Part 6 in the 2017 edition of our annual PyCon Must-See Series, highlighting the talks our staff especially loved at PyCon. While there were many great talks, this is our team's shortlist.

"Readability Counts" was a good talk about why your code should be readable and how you get it there. One of the things I appreciated was that while it was very developer-focused, it was human-oriented rather than technical.

In his presentation, Trey Hunner shared four reasons why code should be readable:

• It makes your life easier
• Code is more often read than written
• It is easier to maintain readable code
• It’s easier to onboard new team members

He also shared a few best practices to achieve this, including usage of white space, line breaks, and code structure; descriptive naming; and choosing the right construct and coding idioms.

How to Reverse a List in Python

A step-by-step tutorial on the three main ways to reverse a Python list or array: in-place reversal, list slicing, and reverse iteration.

Reversing a list is a common operation in Python programming.

For example, imagine you had a sorted list of customer names that your program displays in alphabetical (A-Z) order. Some of your users would like to view the customer list so that the names are in reverse alphabetical order. How are you going to flip the order of this existing list on its head? Or in other words:

What’s the best way to reverse the order of a list in Python?

In this article you’ll see three different ways to achieve this result in “plain vanilla” Python, meaning without the use of any third-party libraries:

1. Reversing a list in-place with the list.reverse() method
2. Using the “[::-1]” list slicing trick to create a reversed copy
3. Creating a reverse iterator with the reversed() built-in function

All examples I’m using here will be based on the following list object containing the numbers 1 through 5:

# You have this:[1,2,3,4,5]# And you want that:[5,4,3,2,1]

Ready? Let’s reverse some lists together!

Option #1: Reversing a List In-Place With the list.reverse() Method

Every list in Python has a built-in reverse() method you can call to reverse the contents of the list object in-place. Reversing the list in-place means won’t create a new list and copy the existing elements to it in reverse order. Instead, it directly modifies the original list object.

Here’s an example:

>>>mylist=[1,2,3,4,5]>>>mylist[1,2,3,4,5]>>>mylist.reverse()None>>>mylist[5,4,3,2,1]

As you can see, calling mylist.reverse() returned None, but modified the original list object. This implementation was chosen deliberately by the developers of the Python standard library:

The reverse() method modifies the sequence in place for economy of space when reversing a large sequence. To remind users that it operates by side effect, it does not return the reversed sequence. (Source: Python 3 Docs)

In-place reversal has some benefits and some downsides. On the plus side, it’s a fast operation—shuffling the list elements around doesn’t require much extra memory, as we’re not creating a full copy of the list.

However, reversing a list in-place overwrites the original sort order. This could be a potential downside. (Of course, to restore the original order you coud simply reverse the same list again.)

From a code readability standpoint, I like this approach. The syntax is clear and easy to understand, even for developers new to Python or someone who comes from another language background.

Option #2: Using the “[::-1]” Slicing Trick to Reverse a Python List

Python’s list objects have an interesting feature called slicing. You can view it as an extension of the square-brackets indexing syntax. It includes a special case where slicing a list with “[::-1]” produces a reversed copy:

>>>mylist[1,2,3,4,5]>>>mylist[::-1][5,4,3,2,1]

Reversing a list this way takes up a more memory compared to an in-place reversal because it creates a (shallow) copy of the list. And creating the copy requires allocating enough space to hold all of the existing elements.

Note that this only creates a “shallow” copy where the container is duplicated, but not the individual list elements. Instead of duplicating the list elements themselves, references to the original elements are reused in the new copy of the container. If the elements are mutable, modifying an element in the original list will also be reflected in the copy.

The biggest downside to reversing a list with the slicing syntax is that it uses a more advanced Python feature that some people would say is “arcane.” I don’t blame them—list slicing is fast, but also a little difficult to understand the first time you encounter its quirky syntax.

When I’m reading Python code that makes use of list slicing I often have to slow down and concentrate to “mentally parse” the statement, to make sure I understand what’s going on. My biggest gripe here is that the “[::-1]” slicing syntax does not communicate clearly enough that it creates a reversed copy of the original list.

Using Pythons’s slicing feature to reverse a list is a decent solution, but it can be a difficult to read to the uninitiated. Be sure to remember the wise words of master Yoda: With great power, great responsibility comes🙂

>>>mylist[start:end:step]>>>mylist[1,2,3,4,5]>>>mylist[1:3][2,3]

>>>mylist[::][1,2,3,4,5]

>>>mylist[::2][1,3,5]

>>>mylist[::-1][5,4,3,2,1]

Option #3: Creating a Reverse Iterator With the reversed() Built-In Function

Reversing a list using reverse iteration with the reversed() built-in is another option. It neither reverses a list in-place, nor does it create a full copy. Instead we get a reverse iterator we can use to cyle through the elements of the list in reverse order:

>>>mylist=[1,2,3,4,5]>>>foriteminreversed(mylist):...print(item)54321>>>mylist>>>[1,2,3,4,5]

Using reversed() does not modify the original list. In fact all we get is a “view” into the existing list that we can use to look at all the elements in reverse order. This is a powerful technique that takes advantage of Python’s iterator protocol.

So far, all we did was iterating over the elements of a list in reverse order. But how can you create a reversed copy of a list using Python’s reversed() function?

Here’s how:

>>>mylist=[1,2,3,4,5]>>>list(reversed(mylist))[5,4,3,2,1]

Notice how I’m calling the list() constructor on the result of the reversed() function?

Using the list constructor built-in keeps iterating until the (reverse) iterator is exhausted, and puts all the elements fetched from the iterator into a new list object. And this gives us the desired result: A reversed shallow copy of the original list.

I really like this reverse iterator approach for reversing lists in Python. It communicates clearly what is going on, and even someone new to the language would intuitively understand we’re creating a reversed copy of the list. And while understanding how iterators work at a deeper level is helpful, it’s not absolutely necessary to use this technique.

Summary: Reversing Lists in Python

List reversal is a fairly common operation in programming. In this tutorial we covered three different approaches for reversing a list or array in Python. Let’s do a quick recap on each of those approaches before I’ll give you my final verdict on which option I recommend the most:

>>>lst=[1,2,3,4,5]>>>lst.reverse()>>>lst[5,4,3,2,1]

>>>lst=[1,2,3,4,5]>>>lst[::-1][5,4,3,2,1]

>>>lst=[1,2,3,4,5]>>>list(reversed(lst))[5,4,3,2,1]

If you’re wondering what the “best” way is to reverse a list in Python my answer will be: “It depends.” Personally, I like the first and third approach:

• The list.reverse() method is fast, clear and speaks for itself. Whenever you have a situation where you want to reverse a list in-place and don’t want a copy and it’s okay to modify the original, then I would go with this option.

• If that isn’t possible, I would lean towards the reversed iterator approach where you call reversed() on the list object and you either cycle through the elements one by one, or you call the list() function to create a reversed copy. I like this solution because it’s fast and clearly states its intent.

I don’t like the list slicing trick as much. It feels “arcane” and it can be difficult to see at a glance what’s going on. I try to avoid using it for this reason.

Note that there are other approaches like implementing list reversal from scratch or reversing a list using a recursive algorithm that are common interview questions, but not very good solutions for Python programming in the “real world.” That’s why I didn’t cover them in this tutorial.

If you’d like to dig deeper into the subject, be sure to watch my YouTube tutorial on list reversal in Python. It’s also embedded at the top of the article. Happy Pythoning!

After 16 years with Vi(m), a week back I switched to Emacs as my primary editor. I used Emacs for a few days in 2010, when it was suggested to me by the #lisp channel. But, neither I continued, nor I understood the keystrokes well.

But, why now?

I was trying out Org Mode in Vim. Looking at the strange key-combinations, I felt it is not for me. I decided to give Emacs a proper try for the first time. If you know about our summer training, or saw me discussing about GNU/Linux world in any college, you would have seen me to suggest vim to everyone as a starting point. Mostly because of two reasons.

• It is there on default Fedora installation.
• I still think it is easy enough for the beginners to start with.

For me, to learn anything new, I prefer to use it regularly. Let it be a programming language, or a particular tool. I wanted to see how Org Mode works in Emacs, but, to do so well I had to use it myself. That also means I will have to use Emacs regularly.

How did I start?

I used a few different sources to start reading, and also configuring my init.el file. Shakthi Kannan has some excellent articles on Emacs. I also found another site which introduces Emacs and related configurations well. I am going to suggest both sites to anyone starting with Emacs. Shakthi also has a very good reference card for the keystrokes.

In the #dgplug IRC channel, maxking suggested to start using eshell and magit inside of Emacs. I am using the presentation from Shakthi to learn magit.

The last few blog posts (including this one), and also a few commits in random places were done using this setup (inside of Emacs).

The problems

I am not going to say this was very smooth. Remembering the keystrokes is always difficult. Getting them into the muscle memory is even more time consuming. But, there are few things which I found really difficult (for me).

• Marking for cut/copy/paste.
• Sometimes I see the same buffer in two split areas, just can not close them easily.
• Spell check keystrokes

All the other regular Emacs users I know, are using it for more than 10 years. I am not trying to hurry, I will slowly learn the various facts and HOWTOs. The famous XKCD post fits well in this regard.

The Python subprocess module is a powerful swiss-army knife for launching and interacting with child processes. It comes with several high-level APIs like call, check_output and (starting with Python 3.5) run that are focused at child processes our program runs and waits to complete.

In this post I want to discuss a variation of this task that is less directly addressed - long-running child processes. Think about testing some server - for example an HTTP server. We launch it as a child process, then connect clients to it and run some testing sequence. When we're done we want to shut down the child process in an orderly way. This would be difficult to achieve with APIs that just run a child process to completion synchronously, so we'll have to look at some of the lower-level APIs.

Sure, we could launch a child process with subprocess.run in one thread and interact with it (via a known port, for example) in another thread. But this would make it tricky to cleanly terminate the child process when we're done with it. If the child process has an orderly termination sequence (such as sending some sort of "quit" command), this is doable. But most servers do not, and will just spin forever until killed. This is the use-case this post addresses.

defmain():proc=subprocess.Popen(['python3','-u','-m','http.server','8070'],stdout=subprocess.PIPE,stderr=subprocess.STDOUT)try:time.sleep(0.2)resp=urllib.request.urlopen('http://localhost:8070')assertb'Directory listing'inresp.read()finally:proc.terminate()try:outs,_=proc.communicate(timeout=0.2)print('== subprocess exited with rc =',proc.returncode)print(outs.decode('utf-8'))exceptsubprocess.TimeoutExpired:print('subprocess did not terminate in time')

$ python3.6 interact-http-server.py
== subprocess exited with rc = -15
Serving HTTP on 0.0.0.0 port 8070 (http://0.0.0.0:8070/) ...
127.0.0.1 - - [05/Jul/2017 05:48:34] "GET / HTTP/1.1" 200 -

defoutput_reader(proc):forlineiniter(proc.stdout.readline,b''):print('got line: {0}'.format(line.decode('utf-8')),end='')defmain():proc=subprocess.Popen(['python3','-u','-m','http.server','8070'],stdout=subprocess.PIPE,stderr=subprocess.STDOUT)t=threading.Thread(target=output_reader,args=(proc,))t.start()try:time.sleep(0.2)foriinrange(4):resp=urllib.request.urlopen('http://localhost:8070')assertb'Directory listing'inresp.read()time.sleep(0.1)finally:proc.terminate()try:proc.wait(timeout=0.2)print('== subprocess exited with rc =',proc.returncode)exceptsubprocess.TimeoutExpired:print('subprocess did not terminate in time')t.join()

defoutput_reader(proc,outq):forlineiniter(proc.stdout.readline,b''):outq.put(line.decode('utf-8'))

outq=queue.Queue()t=threading.Thread(target=output_reader,args=(proc,outq))t.start()

try:line=outq.get(block=False)print('got line from outq: {0}'.format(line),end='')exceptqueue.Empty:print('could not get line from queue')

defmain():proc=subprocess.Popen(['python3','-i'],stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE)# To avoid deadlocks: careful to: add \n to output, flush output, use# readline() rather than read()proc.stdin.write(b'2+2\n')proc.stdin.flush()print(proc.stdout.readline())proc.stdin.write(b'len("foobar")\n')proc.stdin.flush()print(proc.stdout.readline())proc.stdin.close()proc.terminate()proc.wait(timeout=0.2)

defsocket_reader(sockobj,outq,exit_event):whilenotexit_event.is_set():try:buf=sockobj.recv(1)iflen(buf)<1:breakoutq.put(buf)exceptsocket.timeout:continueexceptOSErrorase:break

try:v=outq.get(block=False)print(v)exceptqueue.Empty:break

One of our jobs here in Shopkick’s infrastructure team is to save everyone time, without sacrificing security. During a recent hackathon, we decided it would be fun to replace some of our internal systems using http basic authentication with something a bit more - how shall I put it - from this decade. Fortunately, Bitly’s oauth2_proxy provides a fairly easy way to do just that, allowing us to leverage Google authentication with OAuth, which we were already using elsewhere.

We had a few issues to contend with in starting this.

• We use nginx for SSL termination and reverse proxying our internal sites.
• We want to make things more, not less, secure.
• The transition had to be as seamless as possible.
• We had to accommodate the WebSocket protocol for Jupyter notebook.
• We had to be able to finish the work in a single 24-hour hackathon.

You only need two things to make this work: oauth2_proxy and nginx (1.5.4+ for auth_request support). Building a simple Go application like oauth2_proxy is fairly easy, but if you prefer to grab the precompiled binary you can find one on the GitHub releases page. Either way, all you need is the oauth2_proxy binary. In the examples here, we are running oauth2_proxy on the same hosts as nginx, but that is not a requirement.

The configuration file for oauth2_proxy is fairly simple:

email_domains =["shopkick.com"]
upstreams = ["http://<FOO>.shopkick.com"]
cookie_secret = "<REDACTEDCOOKIESECRET>"
cookie_secure = true
provider = "google"
client_id = “<REDACTED_CLIENT_ID>.apps.googleusercontent.com”

Certain configuration values are better left set as flags at runtime. Here are some that we set:

• -set-xauthrequest - sets the X-Auth-Request-User and X-Auth-Request-Email headers which can be passed through by nginx
• -client-secret - sets our client-secret at runtime, rather than in the config file
• -authenticated-emails-file - sets the path to a newline-delimited list of external email addresses that are permitted to authenticate

Once you have your config in place, run oauth2_proxy like this:

$ oauth2_proxy -set-xauthrequest\-config<CONFIG_FILE>\-client-secret<REDACTED_SECRET>\-authenticated-emails-file<EMAIL_LIST_FILE>

The rest of your configuration is in nginx. As mentioned before, make sure you are using 1.5.4+ or something with auth_request support. Following are some example configurations.

Here is an example with WebSocket support, for Jupyter notebook in our case.

server {
    listen 0.0.0.0:443 ssl;
    server_name foo.shopkick.com;
    location /oauth2/auth {
        internal;
        proxy_pass http://127.0.0.1:4180;
    }
    location /oauth2/ {
        proxy_pass http://127.0.0.1:4180;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Scheme $scheme;
        proxy_set_header X-Auth-Request-Redirect $request_uri;
    }
    location / {
        auth_request /oauth2/auth;
        error_page 401= https://foo.shopkick.com/oauth2/sign_in;
        proxy_pass http://foo:8080/;
        proxy_set_header Host $host;
        # Jupyter requires WebSockets, so we have to add these lines
        #in order for terminals and notebooks to function.
        proxy_http_version 1.1;
        proxy_set_header Upgrade "websocket";
        proxy_set_header Connection "Upgrade";
        proxy_read_timeout    86400;
    }
}

Example passing X-Forwarded-User and X-Email headers, based on the information provided by oauth2_proxy's -set-xauthrequest command-line flag.

server {
    listen  0.0.0.0:443 ssl;
    server_name bar.shopkick.com;
    proxy_headers_hash_max_size 2048;
    proxy_headers_hash_bucket_size 128;
    location / {
        auth_request /oauth2/auth;
        error_page 401= https://bar.shopkick.com/oauth2/sign_in;
        proxy_set_header Host $host;
        proxy_set_header HTTP_X_FORWARDED_SSL on;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        auth_request_set $user   $upstream_http_x_auth_request_user;
        auth_request_set $email  $upstream_http_x_auth_request_email;
        proxy_set_header X-Forwarded-User  $user;
        proxy_set_header X-Email $email;
        proxy_pass http://bar:80;
    }
    location /oauth2/auth {
        internal;
        proxy_pass http://127.0.0.1:4180;
    }
    location /oauth2/ {
        proxy_pass http://127.0.0.1:4180;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Scheme $scheme;
        proxy_set_header X-Auth-Request-Redirect $request_uri;
    }
}

And finally, an example using a non-standard port for https.

server {
    listen  0.0.0.0:4443 ssl;
    server_name baz.shopkick.com;

    # Redirect to HTTPS when HTTP request comes in
    error_page 497 https://$host:4443$request_uri;
    location / {
        error_page 401= https://baz.shopkick.com/oauth2/sign_in;
        auth_request /oauth2/auth;
        proxy_pass http://baz:4443;# Note that here we are passing $http_host instead of $host.
        #$http_host contains the port information, which in this
        #case is critical for the redirect URL that google hands
        # back after a successful authentication.
        proxy_set_header Host $http_host;
    }
    location /oauth2/auth {
        internal;
        proxy_pass http://127.0.0.1:4180;
    }
    location /oauth2/ {
        proxy_pass http://127.0.0.1:4180;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Scheme $scheme;
        proxy_set_header X-Auth-Request-Redirect $request_uri;
    }
}

While all of the above uses Google as the OAuth provider, everything here should work with any other supported provider: Azure, Facebook, Github, Gitlab, LinkedIn, or MyUSA. Just remember to set your provider flag or config setting.

Happy authenticating! And don't forget to subscribe!

Context managers allow programmers to reuse common setup/cleanup code in multiple places.

We'll talk about when context managers are useful, where you usually find them, and how to make your own.

While I was working on my project on software licensing, I noticed that there are several packages in PyPI which do not have a requirements.txt file (the file mentioning the dependencies). License files, though not a rare species but surely an endangered one in the Python World. These made my life so difficult you lazy programmers, grrr).

“One has to solve her problem.” So whenever I used to meet any developer I used to tell them the best practices for developers. Some of them used to give me not always a good vibe. (oh you irritating lady :(). Most of them were saddened by the long list of best practices. (me (grrr, grrr, grr))

I wanted to upload my new project, gitcen in PyPI. I shared my this thought with my eternal rival. He gave me quite smile.
He thinking : "Lady now you will understand". I smiled back.
Me thinking : "ohh man I hate you".

The code was all set and working

I used both PyCharm and jupyter notebook to write code.

The main job is done, now I will only have to upload my project at PyPI, the Pyhton Package Index.

requirements.txt

I created a requirements.txt file. One has to add the external dependencies. It helps to create the the similar development environment. Instead of manually typing the dependencies one can use pip freeze which will show the all modules installed in that particular virtual environment.

$ pip freeze
$ pip freeze > requirements.txt

This is easy, (Lazy coders).

setup.py

Now is the time for setup.py. It is a Python file which helps us to get the package. In the file, we call the setup function. The parameter inside the function are the different metadata of the project. The setup.py file of gitcen looks like this:

from setuptools import setup

setup(  
    name="gitcen",
    version='0.1.0',
    description="A project to find git information.",
    long_description="A project to find git information about authors' commits,most active days and time.",
    author="Anwesha Das.",
    author_email="anwesha@das.community",
    url="https://github.com/anweshadas/gitcen",
    license="GPLv3+",
    py_modules=['gitcen'],
    install_requires=[
        'Click',
        'pygit2==0.24'
    ],
    entry_points='''
        [console_scripts]
        gitcen=gitcen:main
    ''',
    classifiers=(
        'Development Status :: 3 - Alpha',
        'Intended Audience :: Developers',
        'License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)',
        'Programming Language :: Python :: 3.6'
    )
)

A little bite of history, previously there was package a named distutils which provided the setup function. But now we use the package setuptools.

It includes different information such as name, license, entry point etc. There is a parameter classifiers . Read more about it here.

This was lenghty but ok. May these coders are not that lazy (while coding, otherwise is. As per my experience with my man goes.)

Creating Source Distribution

The next was to create a compressed tarball for source distribution.

$python3 setup.py sdist

Output of the command is a tar.gz file.

Creating PyPI account

PyPI, the Python Package Index, the third party official software repository for Python ecosystem. One has to create an account in it.

test.pypi.org

If one wants to test the upload, test.pypi.org is the address to go to. Know more about the steps here.

When will this thing end?

Use twine

twine helps to upload packages in PyPI. It assembles the utilities to interact with PyPI. The first step was to install twine. I installed it from dnf package manager.

$ sudo dnf install python3-twine

I had to register my project. Using the following command

$ twine register dist/gitcen-0.1.0.tar.gz

Finally the moment had come! I uploaded gitcen in PyPI with the following command

 $ twine upload dist/gitcen-0.1.0.tar.gz

I had already put my authentication information in the ~/.pypirc

Now anyone can install gitcen from PyPI

$ pip install gitcen

Therefore going through this now I understand that it is very frustrating. You have a working code and you are not been able to share this with the community (easily, fast). If we give it a thought then we can realize that these are steps one has to memorize. As I was a first timer there were many new things to which I came across, so it made my life difficult. And yes,

Sorry coders, for not understanding your pain!

"What are plugins?" and other proceedings of the inaugural PyCon Comparative Plugin Systems BoF.

Within the programming world, and the Python ecosystem in particular, there are a lot of presumptions around plugins. Specifically, we take them for granted. "It's just a plugin.""Oh, another plugin library?"

So for PyCon 2017, I resolved to dismiss the dismissals by revisiting plugins, and it may have been the best programming decision I've made all year.

Why plugins?

For all types of software, open-source or otherwise, the scalability of development poses a problem long before scalability of performance and other technical challenges. Engaging more developers creates code contention and bugs. Too many cooks is all it takes to spoil the broth.

All growing projects need an API for code integration.

Call them plugins, modules, or extensions, from your browser to your kernel, they are the widely successful solution. Tellingly, the only thing wider than the success of plugin-based architecture is the variety of implementations.

Python's dynamic nature in particular seems to encourage inventiveness. The more the merrier, usually, but at some point we cloud a tricky space. How different could these plugin systems be? How wide is the range of functionalities, really? How does a developer choose the right plugin system for a given project? For that matter, what is a plugin system anyway? No one I talked to had clear answers.

So when PyCon 2017 rolled around, I knew exactly what I wanted to do: call together a team of developers to get to the bottom of the above, or at the very least, answer the question,

"What happens when you ask a dozen veteran Python programmers to spill their guts about plugins?"

Setting examples

Our group leapt into action by listing off plugin systems as fast as we could:

• stevedore
• twisted.plugin
• Mercurial extensions
• pytest plugins
• gather
• venusian
• pluginbase
• straight.plugin
• pylint plugins
• flake8 plugins
• raw setuptools entrypoints
• zope.component
• Django command extensions
• SQLAlchemy dialects/DBAPIs
• Sphinx extensions
• Buildout extensions
• Pike
• Others that came and went a little too fast to jot down

With our plate heaping with examples like these, we all felt ready to dig into our big questions.

Taxonomizing

For our first bit of analysis, we asked: What practical and fundamental attributes differentiate these approaches? If we had to create a taxonomy, what characteristics would we look for?

Generalizability

You'll notice our list of example plugin systems included several very specialized examples, from pylint to SQLAlchemy. Many projects even use totally internal plugin systems to achieve better factoring.

Bespoke plugin systems like pylint's are a valuable reference for anyone looking to account for patterns in their own system, especially generic systems like pike and stevedore.

Discovery

A plugin system's first job is locating the plugins to load. The split here is whether plugins are individually specified, or automatically discovered based on paths and patterns.

In either case, we need paths. Some systems provide search functionality, exchanging explicitness for convenience. This can be a good trade, especially when plugins number in the double digits, or whenever less technical users are concerned.

Install location

Closely related to discovery, our next differentiator was the degree to which the plugin system leveraged Python's own package management facilities. Some systems, like venusian, were designed to encourage pip install-ing plugins, searching for them in site-packages, alongside the application itself.

Other systems have their own search paths, locating plugins in the user directory and elsewhere on the filesystem. Still other systems are designed for plugins inside the application tree, as is the case with Django apps.

Plugin independence

One of the most challenging parts of plugin development is finding ways of independently reusing and testing code, while keeping in mind the code's role as an optional component of another application.

In some systems, like Django's, the tailoring is so tightly coupled that reusability doesn't make sense. But other approaches, like gather's, keeps plugin code independently usable.

Dependency registration

Almost all plugins work by providing some set of hooks which are findable and callable by the core. We found another differentiator in whether and how plugins could gain access to resources from the core, and even other plugins.

Not all systems support this, preferring to keep plugins as leaf participants in the application. Those simplistic setups hit limits fast. The next best, and most common, solution is to simply pass the whole core state at the time of hook invocation, providing plugins with the same access as the core. It works, but the API becomes the whole system state.

More advanced systems allow plugins to publish an inventory of dependencies, which the core then injects. Higher granularity enables lazier evaluation for a performance boost, and more explicit structure helps create a more maintainable application overall.

Drawing a line

With our group feeling like we were approaching the nature of things, we reversed direction, asking instead: What isn't a plugin system?

Establishing explicit boundaries and specific counterexamples proved instrumental to producing a final definition.

Is eval() a plugin system? We thought maybe, at first. But the more we thought about it, no, because the code itself was not sufficiently abstracted through a loading or namespacing system.

Is DNS a plugin system? It has names and namespaces galore. But no, because code is not being loaded in. Remote services in general are beyond the boundary of what a plugin can be. They exist out there, and we call out to them. They're plugins, not callouts.

A definition

So with our boundaries established, we were ready to offer a definition:

A plugin system is a software facility used by a running program to discover and load code, often containing hooks called by the host application

But, by this definition, isn't Python's built-in import functionality a plugin system? Mostly, yes! Python's import system is a plugin system.

• For discovery it uses sys.path, various "site" directories and ".pth" files, and much more.
• For installation, it uses site-packages, user .local directories, and more.
• As far as independent reusability, virtually every module can be made its own entrypoint.
• As for dependency registration, every module is tossed into sys.modules with the others, but also has access to import and sys, making roughly every module an equal partner in application state.

Python's import system is a powerful one, with a plugin system of its own. But finders, loaders, and import hooks aren't Python's plugin system. For that, you need to look to the site module.

Motivation

With our hour nearly up, all these proximate details still needed to be distilled into an ultimate motivation behind plugins. To this end, we returned to one of software engineering's fundamental principles: Separation of concerns.

We want to reason about our software. We want to know what state it is in. What we all want is the ability to say, "the core is ready, proceeding to load modules/extensions/plugins." We want to defer loading some code so that we can add extra instrumentation, checks, resiliency, and error messages to that loading process. If something misbehaves, we can do better than a stack trace and an ImportError.

Python's import system is a plugin system of sorts, but because we use it all the time, we've already used up most of the concern separation potential of import. Hence, all the creativity around plugin systems, seeking a balance between feeling native to Python, while not still successfully separating concerns.

In conclusion

So now we have achieved a complete view of the Python plugin system ecosystem, from motivation to manifestation.

By numbers alone, it may seem on the face like there are more than enough Python plugin solutions. But looking at the motivation and taxonomy above, it's clear that there are still several gaps waiting to be filled.

By taking a holistic look at the implementations and motivations, the PyCon 2017 Plugins Open Session ended with the conclusion that even Python's wide selection could use expansion.

So, until next year, go forth and continue to build! The future of well-factored code depends on it.¹