Quantcast
Channel: Planet Python
Viewing all 23329 articles
Browse latest View live

Mahmoud Hashemi: Changing the Tires on a Moving Codebase

$
0
0

2020 was a year of reckonings. And for all that was beyond one’s control, as the year went on, I found myself pouring more and more into the one thing that felt within reach: futureproofing of the large enterprise web application I helped build, SimpleLegal.

Now complete, this replatforming easily ranks in my most complex projects, and right now, holds the top spot for the happiest ending. That happiness comes at a cost, but with some the right approach that cost may not be as high as you think.

The Bottom Line

We took SimpleLegal’s primary product, a 300,000 line Django-1.11-Python 2.7-Redis-Postgres-10 codebase, to a Django 2.2-Python 3.8-Postgres-12 stack, on-schedule and without major site incidents. And it feels amazing.

Speaking as tech lead on the project, what did it look like? For me, something like this:

But as Director of Engineering, what did it cost? 3.5 dev years and just about $2 per line of code.

And I'm especially proud of that result, because along the way, we also substantially improved the speed and reliability of both the site and development process itself. The product now has a bright future ahead, ready to shine in sales RFPs and compliance questionnaires. Most importantly, there’ll be no worrying about when to delicately break it to a candidate that they’ll be working with unsupported technology.

In short, a large, solid investment that’s already paying for itself. If you just came here for the estimate we wish we had, you've got it. This post is all about how your team can achieve the same result, if not better.

The Setup

The story begins in 2013, when a freshly YC-incubated SimpleLegal made all the right decisions for a new SaaS LegalTech company: Python, Django, Postgres, Redis. In classic startup fashion, features came first, unless technology was a blocker. Packages were only upgraded incidentally.

By 2019, the end of this technical runway had drawn near. While Python 2 may be getting extended support from various vendors, there were precious few volunteers in sight to do Django 1 CVE patches in 2021. A web framework’s a riskier attack surface, so we finally had our compliance forcing function, and it was time to pay off our tech debt.

The Outset

So began our Tech Refresh replatforming initiative, in Q4 2019. The goal: Upgrade the stack while still shipping features, like changing the tires of a moving car. We wanted to do it carefully, and that would take time. Here are some helpful ground rules for long-running projects:

  1. Any project that gets worked on 10+ hours per week deserves a 30-minute weekly sync.
  2. Every recurring meeting deserves a log. Put it in the invite. Use that Project Log to record progress, blockers, and decisions.
  3. It’s a marathon, not a sprint. Avoid relying on working nights, weekends, and holidays.

We started with a sketch of a plan that, generously interpreted, ended up being about halfway correct. Some early guesses that turned into successes:

  1. Move to pip-tools and unpin dependencies based on extensive changelog analysis. Identify packages without py23 compatible versions. (Though we’ve since moved to poetry.)
  2. Add line coverage reporting to CI
  3. Revamp internal testing framework to allow devs to quickly write tests

More on these below. Other plans weren’t so realistic:

  1. Take our CI from ~60% to 95% line coverage in 6 months
  2. Parallelized conversion of app packages over the course of 3 months
  3. Use low traffic times around USA holidays (Thanksgiving, Christmas, New Years) to gradually roll onto the new app before 2021.

We were young! As naïve as we were, at least we knew it would be a lot of work. To help shoulder the burden, we scouted, hired, and trained three dedicated off-shore developers.

The Traction Issues

Even with added developers, by mid-2020 it was becoming obvious we were dreaming about 95% coverage, let alone 100%. Total coverage may be best practice, but 3.5 developers couldn’t cover enough ground. We were getting valuable tests, and even finding old bugs, but if we stuck with the letter of the plan, Django 2 would end up being a 2022 project. At 70%, we decided it was time to pivot.

We realized that CI is more sensitive than most users for most of the site. So we focused in on testing the highest impact code. What’s high-impact? 1) the code that fails most visibly and 2) the code that’s hardest to retry. You can build an inventory of high-impact code in under a week by looking at traffic stats, batch job schedules, and asking your support staff.

Around 80% of the codebase falls outside that high-traffic/high-impact list. What to do about that 80%? Lean in on error detection and fast time-to-fix.

The Sentry Pivot

One nice thing about startup life is that it’s easy to try new tools. One practice we’ve embraced at SimpleLegal is to reserve every 5th week for developers to work on the development process itself, like a coordinated 20% time. Even the best chef can’t cook five-star food in a messy kitchen. This was our way of cleaning up the shop and ultimately speeding up the ship.

During one such period, someone had the genius idea to add dedicated error reporting to the system, using Sentry. Within a day or two, we had a site you could visit and get stack traces. It was pretty magical, and it wasn’t until Tech Refresh that we realized that while integration takes one dev-day, full adoption can take a team months.

You see, adding Sentry to a mature-but-fast-moving system means one thing: noise. Our live site was erroring all the time. Most errors weren’t visible or didn’t block users, who in some cases had quietly learned to work around longstanding site quirks. Pretty quickly, our developers learned to treat Sentry as a repository of debugging information. A Sentry event on its own wasn’t something to be taken seriously in 2019. That changed in 2020, with the team responsible for delivering a seamless replatform needing Sentry to be something else: a responsive site quality tool.

How did we get there? First step, enhance the data flowing into Sentry by following these best practices:

  1. Split up your products into separate Sentry projects. This includes your frontend and backend.
  2. Tag your releases. Don’t tag dev env deployments with the branch, it clutters up the Releases UI. Add a separate branch tag for searches.
  3. Split up your environments. This is critical for directing alerts. Our Sentry client environment is configured by domain conventions and Django’s sites framework. If it helps, here's a baseline, we use these environments:
    • Production: Current official release. DevOps monitored.
    • Sandbox: Current official release (some companies do next release). Used by customers to test changes. DevOps monitored.
    • Demo/Sales: Previous official release. Mostly internal traffic, but external visibility at prospect demo time. DevOps monitored.
    • Canary: Next official release. Otherwise known as staging. Internal traffic. Dev monitored.
    • ProdQA: Current official release. Used internally to reproduce support issues. Dev monitored.
    • QA: Dev branches, dev release, internal traffic. Unmonitored debugging data.
    • Local test/CI: Not published to Sentry by default.

With issues finally properly tagged and searchable, we used Sentry’s new Discover tool to export issues weekly, and prioritize legacy errors. To start, we focused on high-visibility production errors with non-internal human users. Our specific query: has:user !transaction:/api/* event.type:error !user.username:*@simplelegal.*

We triaged into 4 categories: Quick fix (minor bug), Quick error (turn an opaque 500 error into a actionable 400 of some form), Spike (larger bug, requires research), and Silence (using Sentry’s ignore feature). Over 6 weeks we went from over 2500 weekly events down to less than 500.

Further efforts have gotten us under 100 events per week, spread across a handful of issues, which is more than manageable for even a lean team. While "Sentry Zero" remains the ideal, we achieved and maintained the real goal of a responsive flow, in large part thanks to the Slack integration. Our team no longer hears about server errors from our Support team. In fact, these days, we let them know when a client is having trouble and we’ve got a ticket underway.

And it really is important to develop close ties with your support team. Embedded in our strategy above was that CI is much more sensitive than a real user. While perfection is tempting, it’s not unrealistic to ask a bit of patience from an enterprise user, provided your support team is prepared. Sync with them weekly so surprise is minimized. If they’re feeling ambitious, you can teach them some Sentry basics, too.

The New Road

With noise virtually eliminated, we were ready to move fast. While the lean-in on fast-fixing Sentry issues was necessary, a strong reactive game is only useful if there are proactive changes being pushed. Here are some highlights we learned when making those changes:

Committing to transactions

Used properly, rollbacks can make it like errors never happened, the perfect complement to a fast-fix strategy.

The truly atomic request

Get as much as possible into the transactions. Turn on ATOMIC_REQUESTS, if you haven’t already. Some requests do more than change the database, though, like sending notifications and enqueuing background tasks.

At SimpleLegal, we rearchitected to defer all side effects (except logging) until a successful response was being returned. Middleware can help, but mainly we achieved this by getting rid of our Redis queue, and switching to a PostgreSQL-backed task queue/broker. This arrangement ensures that if an error occurs, the transaction is rolled back, no tasks are enqueued, and the user gets a clean failure. We spot the breakage in Sentry, toggle over to the old site to unblock, and their next retry succeeds.

Transactional test setup

Transactionality also proved key to our testing strategy. SimpleLegal had long outgrown Django’s primitive fixture system. Most tests required complex Python to set up, making tests slow to write and slow to run. To speed up both writing and running, we wrapped the whole test session in a transaction, then, before any test cases run, we set up exemplary base states. Test cases used these base states as fixtures, and rolled back to the base state after every test case. See this conftest.py excerpt for details.

Better than best practices

Software scenarios vary so widely, there’s an art to knowing which advice isn’t for you. Here’s an assortment of cul de sacs we learned about firsthand.

The utility of namespaces

Given how code is divided into modules, packages, Django apps, etc., it may be tempting to treat those as units of work. Don’t start there. Code divisions can be pretty arbitrary, and it’s hard to know when you’ve pulled on a risky thread.

Assuming there are automated refactorings, as in a 2to3 conversion, start by porting by type of transformation. That way, one need only review a command and a list of paths affected. Plus, automated fixes necessarily follow a pattern, meaning more people can fix bugs arising from the refactor.

Coverage tools

Coverage was a mixed bag for us. Obviously our coverage-first strategy wasn’t tenable, but it was still useful for prioritization and status checks. On a per-change basis, we found coverage tools to be somewhat unreliable. We never got to the bottom of why coverage acted nondeterministically, and we left the conclusion at, “off-the-shelf tools like codecov are probably not targeted at monorepos of our scale.”

In running into coverage walls, we ended up exploring many other interpretations of coverage. For us, much higher-priority than line coverage were “route coverage” (i.e., every URL has at least one integration test) and “model repr coverage” (i.e., every model object had a useful text representation, useful for debugging in Sentry). With more time, we would have liked to build tools around those, and even around online-profiling based coverage statistics, to prioritize the highest traffic lines, not just the highest traffic routes. If you’ve heard of approaches to these ends, we’d love to discuss them with you.

Flattening database migrations

On the surface, reducing the number of files we needed to upgrade seems logical. Turns out, flattening migrations is a low-payoff strategy to get rid of files. Changing historical migration file structure complicated our rollout, while upgrading migrations we didn’t flatten was straightforward. Not to mention, if you just wanted the CI speedup, you can take the same page from the Open EdX Platform that we did: build a base DB cache that you check in every couple months.

Turns out, you can learn a lot from open-source applications.

Easing onto the stack

If you have more than one application, use the smaller, simpler application to pilot changes. We were lucky enough to have a separate app whose tests ran faster, making for a tighter development loop we coul learn from. Likewise, if you have more than one production environment, start rollouts with the one with the least impact.

Clone your CI jobs for the new stack, too. They’ll all fail, but resist the urge to mark them as optional. Instead, build a single-file inventory of all tests and their current testing state. We built a small extension for our test runner, pytest, which bulk skipped tests based on a status inventory file. Then, ratchet: unskip and fix a test, update the file, check that tests pass, and repeat. Much more convenient and scannable than pytest mark decorators spread throughout the codebase. See this conftest.py excerpt for details.

The Rollout

In Q4 2020, we doubled up on infrastructure to run the old and new sites in parallel, backed by the same database. We got into a loop of enabling traffic to the new stack, building a queue of Sentry issues to fix, and switching it back off, while tracking the time. After around 120 hours of new stack, strategically spread around the clock and week, enough organizational confidence had been built that we could leave the site on during our most critical hours: Mondays and Tuesdays at the beginning of the month.

The sole hiccup was an AWS outage Thanksgiving week. At this point we were ahead of schedule, and enough confidence had been built in our fast-fix workflow that we didn’t need our original holiday testing windows. And for that, mmany thanks were given.

We kept at the fast-fix crank until we were done. Done isn't when the new system has no errors, it's when traffic on the new system has fewer events than the old system. Then, fix forward, and start scheduling time to delete the scaffolding.

The Aftermath

So, once you’re on current LTS versions of Django, Python, Linux, and Postgres, job complete, right?

Thankfully, tech debt never quite hits 0. While updating and replacing core technologies on a schedule is no small feat, replacing a rusty part with a shiny one doesn’t change a design. Architectural tech debt -- mistakes in abstractions, including the lack thereof -- can present an even greater challenge. Solutions to those problems don’t generalize between projects as cleanly, but they do benefit from up-to-date and error-free foundations.

For all the projects looking to add tread to their technical tires, we hope this retrospective helps you confidently and pragmatically retrofit your stack for years to come.

Finally, big thanks to Uvik for the talent connection, and the talent: Yaroslav, Serhii, and Oleh. Shoutouts to Kurt, Justin, and Chris, my fellow leads. And the cheers to business leadership at SimpleLegal and everywhere, for seeing the value in maintainability.



Python Bytes: #224 Join us on a Python adventure back to 1977

$
0
0
<p>Special guest: <a href="https://twitter.com/calvinhp"><strong>Calvin Hendryx-Parker</strong></a></p> <p><strong>Live stream</strong></p> <a href='https://www.youtube.com/watch?v=-yJCDyAepGc' style='font-weight: bold;'>Watch on YouTube</a><br> <br> <p><strong>Michael #1:</strong> <a href="https://pypi.org/project/awsimple/"><strong>AWSimple</strong></a></p> <ul> <li>by James Abel </li> <li>AWSimple is a more object oriented interface on top of boto3 for some of the common “serverless” AWS services: S3, DynamoDB, SNS, and SQS. </li> <li><strong>Features:</strong> <ul> <li>Simple Object Oriented API on top of boto3</li> <li>One-line S3 file write, read, and delete</li> <li>Automatic S3 retries</li> <li>Locally cached S3 accesses</li> <li>True file hashing (SHA512) for S3 files (S3's etag is not a true file hash)</li> <li>DynamoDB full table scans (with local cache option)</li> <li>DynamoDB secondary indexes</li> <li>Built-in pagination (e.g. for DynamoDB table scans and queries). Always get everything you asked for.</li> <li>Can automatically set SQS timeouts based on runtime data (can also be user-specified)</li> </ul></li> <li>Caching: S3 objects and DynamoDB tables can be cached locally to reduce network traffic, minimize AWS costs, and potentially offer a speedup.</li> </ul> <p><strong>Brian #2:</strong> <a href="https://coverage.readthedocs.io/en/latest/source.html"><strong>coverage and installed packages</strong></a></p> <ul> <li>I’ve covered coverage.py a lot on Test &amp; Code, starting with <a href="https://testandcode.com/12">episode 12</a>, and even talked about it on <a href="https://testandcode.com/147">episode 147</a>, and many others.</li> <li>Except there’s something I missed, hidden in plain sight, all this time.</li> <li><code>coverage --source</code> , as well as <code>pytest --cov</code> if using <code>pytest-cov</code> plugin, is not just a path.</li> <li>“You can specify source to measure with the <code>--source</code> command-line switch, or the <code>[run] source</code> configuration value. The value is a comma- or newline-separated list of directories <strong><em>*</strong>or package names</em>*. If specified, only source inside these directories or packages will be measured.” - <a href="https://coverage.readthedocs.io/en/latest/source.html">coverage.py docs</a>, (emphasis mine)</li> <li>Up to now I was doing this trick I picked up from I don’t remember where I would run coverage from the top level project directory, specify the source as the project source, and set a <code>[paths]</code> setting in .coveragerc, the <code>source</code> setting to both the project source and the site-packages directory. </li> <li>Then the report would show the coverage of the source code, even though it was the site-packages code that was running.</li> <li>That trick is still nice to specify the output as your project directory, which is usually a shorter relative path.</li> <li>However, it’s not essential. You can just specify the source as the package name, without the above trick, and coverage will report the coverage of the installed package. That is usually good enough. </li> <li>Super cool</li> </ul> <p><strong>Calvin #3:</strong> <a href="https://avinayak.github.io/algorithms/programming/2021/02/19/finding-mona-lisa-in-the-game-of-life.html"><strong>Finding Mona Lisa in the Game of Life with JAX</strong></a></p> <ul> <li>by Atul Vinaya</li> <li>Lots of great code examples</li> <li>Showcases the speed increase you can get using JAX on a GPU vs CPU unvectorized</li> <li>Initial implementation took days of CPU time to get a rough result</li> <li>JAX compiles numpy to highly vectorized code to run on a GPU</li> <li>Requires some refactor of the code to optimize for a highly parallel run on GPUs</li> <li>Post includes link to notebook used for the project </li> <li>“Running ~1000 iterations for a 483px wide Mona Lisa on the google colab GPU runtime only takes around 40 seconds!”</li> </ul> <p><strong>Michael #4:</strong> <a href="https://www.theregister.com/2021/03/02/python_pypi_purges/"><strong>Python Package Index nukes 3,653 malicious libraries uploaded soon after security shortcoming highlighted</strong></a></p> <ul> <li>From Mark Little</li> <li>Recall Google’s Python goal was around PyPI security.</li> <li>Related (from <a href="https://twitter.com/AntonioAndrade">@tonny</a>) <a href="https://nakedsecurity.sophos.com/2021/03/07/poison-packages-supply-chain-risks-user-hits-python-community-with-4000-fake-modules/">Poison packages – “Supply Chain Risks” user hits Python community with 4000 fake modules</a></li> <li>PyPI has removed 3,653 malicious packages uploaded days after a security weakness in the use of private and public registries was highlighted.</li> <li>“Developers are often advised to review any code they import from an external library though that advice isn't always followed.” ← yeah</li> <li>Last month, security researcher Alex Birsan demonstrated how easy it is to take advantage of these systems through <a href="https://www.theregister.com/2021/02/10/library_dependencies_attack/">a form of typosquatting</a> that exploited the interplay between public and private package registries.</li> <li>Birsan set out to see whether he could identify the names of private packages used inside companies and create malicious packages using those library names to place in the public package registries – the indexes that keep track of available software modules.</li> <li>The names of private packages turned out to be rather easy to find, particularly in the Node.js/JavaScript ecosystem because private package.json files show up rather often in public software repositories.</li> <li>So Biran crafted identically named libraries that he designed to sneak system configuration data through corporate firewalls.</li> <li>The challenge then became getting applications that require private libraries to look for those file names in a polluted public source. As it turns out, it's common for corporate software developers to rely on a hybrid configuration for their applications, one that references private internal packages but also supports fetching dependencies from a public registry, in order to ensure packages are up-to-date.</li> <li>The companies that Birsan managed to attack with this technique include Apple, Microsoft, Netflix, PayPal,Shopify, Tesla, Uber, and Yelp. And for his efforts, he has been awarded at least $130,000 from bug bounty programs involving these firms.</li> <li>Birsan's success in carrying out such attacks should set off alarm bells. Software supply chain attacks present a higher degree of risk than many threat scenarios because they have the potential to affect so many downstream victims</li> <li>Makes me want to setup <a href="https://pypi.org/project/devpi-server/">devpi</a> + <a href="https://pypi.org/project/devpi-constrained/">devpi-constrained</a> just for internal projects.</li> <li><strong>What to do?</strong></li> <li><strong>Don’t do mass bogus uploads like this to prove your point.</strong> We appreciate the message you are trying to deliver, but it’s already been documented so you are just making distracting work for other people who could more usefully be doing something else for the project.</li> <li><strong>Don’t choose a PyPI package juat because the name looks right.</strong> Check that you really are downloading the right module from the right publisher. Even legitimate modules sometimes have names that clash, compete or confuse.</li> <li><strong>Don’t hook internal projects to external repositories by mistake.</strong> If you are using Python packages that you haven’t published externally, then the one thing you can be sure of is that all external copies of “your” package are imposter modules, probably malware.</li> <li><strong>Don’t blindly download package updates into your own development or build systems.</strong> Test and review everything you download before you approve it for use. Remember that packages typically include update-time scripts that run when you do the update, so malware infections could be delivered as part of the update process, not of the module source code that ultimately gets installed.</li> </ul> <p><strong>Brian #5:</strong> <a href="https://github.com/brandon-rhodes/python-adventure"><strong>python-adventure</strong></a></p> <ul> <li>Brandon Rhodes</li> <li>"This is a faithful port of the “Adventure” game to Python 3 from the original 1977 FORTRAN code by Crowther and Woods (it is driven by the same <code>advent.dat</code> file!) that lets you explore Colossal Cave, where others have found fortunes in treasure and gold, though it is rumored that some who enter are never seen again. “</li> <li>“For extra authenticity, the output of the Adventure game in this mode, <code>python3 -m adventure</code>, is typed to your screen at 1200 baud.”</li> <li>“<em>Colossal Cave Adventure</em> is the first known work of <a href="https://en.wikipedia.org/wiki/Interactive_fiction">interactive fiction</a> and, as the first text adventure game, is considered the precursor for the adventure game genre. “ - <a href="https://en.wikipedia.org/wiki/Colossal_Cave_Adventure">wikipedia</a></li> <li>related: <ul> <li><a href="https://en.wikipedia.org/wiki/Zork">Zork</a>, 77-79, also an adventure game, was inspired by <a href="https://en.wikipedia.org/wiki/Colossal_Cave_Adventure">Collossal Cave Adventure</a>, 75-77</li> <li><a href="https://chuck-nbc.fandom.com/wiki/Zork">Zork on Chuck</a> - Brian’s a Chuck fan</li> <li>Brandon, can we have Zork also?</li> </ul></li> <li>side note: <ul> <li>Closest I got was <a href="https://en.wikipedia.org/wiki/Dungeons_of_Daggorath">Dungeons of Daggorath</a> on TRS-80. Not text based. Early 80’s </li> </ul></li> </ul> <p><strong>Calvin #6:</strong> <a href="https://hakibenita.com/django-32-exciting-features"><strong>Exciting New Features in Django 3.2</strong></a></p> <ul> <li>From Haki Benita</li> <li>Upcoming LTS Release in the 3 series</li> <li>Expected in April</li> <li>Post highlights some interesting new features that you might not have noticed</li> <li><strong>New Features</strong> <ul> <li>Covering Indexes in Postgres support (performance plus!)</li> <li>Timezones are hard and TruncDate now helps keep you from pulling out the foot cannon</li> <li>JSONObject DB Functions, helping the unstructured data world keep using Postgres</li> <li>Signal.send_robust() now logs exceptions so you don’t have to!</li> <li>The new QuerySet.alias() method allows creating reusable aliases for expressions (Performance!)</li> <li>The new <code>display</code> decorator makes creating calculated admin fields cleaner</li> <li>Value Expressions Detects Type, more cleaning up to allow the ORM to figure it out</li> <li>Notable missing feature is Async ORM, but this will be awesome when it lands</li> <li>More are listed on the <a href="https://docs.djangoproject.com/en/dev/releases/3.2/">Django 3.2 Release Page</a></li> </ul></li> </ul> <p>Extras:</p> <p>Michael:</p> <ul> <li><a href="https://twitter.com/AntonioAndrade/status/1362725245942136837">Is Python on Mars</a>? </li> <li>FastAPI Website course is out: <a href="https://talkpython.fm/fastapi-web"><strong>talkpython.fm/fastapi-web</strong></a></li> <li>Are you thinking of going to PyCon 2021? Over at Talk Python, we're giving away 5 tickets to the event: <a href="https://talkpython.fm/pycon2021"><strong>talkpython.fm/pycon2021</strong></a></li> <li>Be sure to join us @ <a href="https://pythonbytes.fm/youtube"><strong>pythonbytes.fm/youtube</strong></a></li> <li>Got a chance to <a href="https://twitter.com/finding_genius/status/1369679439345319938"><strong>speak to the medical field about Python and programming superpowers</strong></a> on the Finding Genius podcast.</li> </ul> <p><strong>Calvin:</strong></p> <ul> <li>DjangoCon Europe 2021 CFP is Open until 4/1 https://2021.djangocon.eu/talks/cfp/</li> <li>Python Web Conf 2021 <ul> <li>4 Tracks this year</li> <li>60 Amazing Speakers (almost 20% women)</li> <li>Tickets</li> <li>Professional $199</li> <li>Student $99</li> <li>Grants Available!</li> </ul></li> </ul> <p>Joke:</p> <pre><code> /** Logger */ private Logger logger = Logger.getLogger(); </code></pre> <pre><code> // This is black magic // from // *Some stackoverlow link // Don’t play with magic, it can BITE. </code></pre> <pre><code> # For the sins I am about to commit, may Guido van Rossum forgive me </code></pre> <pre><code> // Remove this if you wanna be fired </code></pre> <pre><code> } catch(Exception ex) { // Houston, we have a problem } </code></pre> <pre><code> int getRandomNumber() { Return 4; // chosen by fair dice roll. // guaranteed to be random. } </code></pre> <p><a href="https://twitter.com/LinuxHandbook/status/1368974401979383810">https://twitter.com/LinuxHandbook/status/1368974401979383810</a></p>

Stack Abuse: How to Split a List Into Even Chunks in Python

$
0
0

Introduction

Splitting strings and lists are common programming activities in Python and other languages. Sometimes we have to split our data in peculiar ways, but more commonly - into even chunks.

The language does not have a built-in function to do this and in this tutorial, we'll take a look at how to split a list into even chunks in Python.

For most cases, you can get by using generators:

def chunk_using_generators(lst, n):
    for i in range(0, len(lst), n):
        yield lst[i:i + n]

Though, there are other interesting ways to do this, each with their own pros and cons!

Split a List Into Even Chunks of N Elements

A list can be split based on the size of the chunk defined. This means that we can define the size of the chunk. If the subset of the list doesn't fit in the size of the defined chunk, fillers need to be inserted in the place of the empty element holders. We will be using None in those cases.

Let's create a new file called chunk_based_on_size.py and add the following contents:

def chunk_based_on_size(lst, n):
    for x in range(0, len(lst), n):
        each_chunk = lst[x: n+x]

        if len(each_chunk) < n:
            each_chunk = each_chunk + [None for y in range(n-len(each_chunk))]
        yield each_chunk

print(list(chunk_based_on_size([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13], 7)))

The above chunk_based_on_size() function takes the arguments: lst for the list and chunk_size for a number to split it by. The function iterates through the list with an increment of the chunk size n. Each chunk is expected to have the size given as an argument. If there aren't enough elements to make a split of the same size, the remaining unused elements are filled with None.

Running this script returns the following list of lists:

$ python3 chunk_based_on_size.py
[[1, 2, 3, 4, 5, 6, 7], [8, 9, 10, 11, 12, 13, None]]

The list has been split into equal chunks of 7 elements each.

Python has utilities to simplify this process. We can use the zip_longest function from itertools to simplify the previous function. Let's create a new file chunk_using_itertools.py and add the following code:

from itertools import zip_longest

def chunk_using_itertools(lst):
    iter_ = iter(lst)
    return list(zip_longest(iter_, iter_, iter_, iter_, iter_, iter_, iter_))

print(chunk_using_itertools([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13]))

This code iterates elements and returns a chunk of the desired length - based on the arguments you provide. We've put 7 iter_ arguments here. The zip_longest() function aggregates and returns elements from each iterable. In this case, it would aggregate the elements from the list that's iterated 7 times in one go. This then creates numerous iterators that contain 7 sequential elements, which are then converted to a list and returned.

When you execute this snippet, it'll result in:

$ python3 chunk_using_itertools.py
[[1, 2, 3, 4, 5, 6, 7], [8, 9, 10, 11, 12, 13, None]]

This shorter function produces the same input. However, it's much more limited as we have to manually write how many elements we want in the code, and it's a bit awkward to just put a bunch of iter_s in the zip_longest() call.

The best solution would be using generators. Let's create a new file, chunk_using_generators.py:

def chunk_using_generators(lst, n):
    for i in range(0, len(lst), n):
        yield lst[i:i + n]

print(list(chunk_using_generators([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13], 7)))

This generator yields a sublist containing n elements. At the end, it would have yielded a sublist for every chunk. Running this code produces this output:

$ python3 chunk_using_generators.py
[[1, 2, 3, 4, 5, 6, 7], [8, 9, 10, 11, 12, 13]]

This solution works best if you don't need padding with None or otherwise.

Split a List Into a N Even Chunks

In the previous section, we split the list based on the size of individual chunks so that each chunk has the same amount of elements. There's another way to interpret this problem. What do we do when we want to split a list not based on the number of elements in each chunk, but on the number of chunks we want to be created?

For example, instead of splitting a list into chunks where every chunk has 7 elements, we want to split a list into 7 even chunks. In this case, we may not know the size of each chunk.

The logic is similar to the previous solutions, however, the size of the chunk is the ceiling value of the length of the list divided by the number of chunks required. Similar to the previous code samples, if a chunk happens to have vacant spots, those will be filled by the filler value None:

import math

def chunk_based_on_number(lst, chunk_numbers):
    n = math.ceil(len(lst)/chunk_numbers)

    for x in range(0, len(lst), n):
        each_chunk = lst[x: n+x]

        if len(each_chunk) < n:
            each_chunk = each_chunk + [None for y in range(n-len(each_chunk))]
        yield each_chunk

print(list(chunk_based_on_number([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13], chunk_numbers=7)))

We determine how many lists we need to create and store that value in n. We then create a sublist for the two elements at a time, padding the output in case our chunk size is smaller than the desired length.

When we execute that file we'll see:

$ python3 chunk_based_on_number.py
[[1, 2], [3, 4], [5, 6], [7, 8], [9, 10], [11, 12], [13, None]]

As seen in the output above, the list has been split into 7 individual lists of equal sizes, based on the argument chunk_numbers provided.

Conclusion

In this article, we have seen some of the ways by which a list can be split into even-sized chunks and lists based on custom methods and by using the built-in modules.

The solutions mentioned in this tutorial, are not limited to the ones defined here, but there are multiple other creative ways by which you can split your list into even-chunks too.

Python Anywhere: System updates on 10 February and 11 March

$
0
0

It’s been a year since PythonAnywhere went all-remote, but it has not slowed us down, and today was the time to deploy an exciting set of changes to our system.

PyCon: T-shirt Ordering is Back Online

$
0
0

Thank you for the quick notification from our community questioning the shipping prices. We quickly contacted the supplier to question the pricing and they did identify a bug in the ordering site that affected the calculations of shipping.  This error was corrected and the ordering site is back online.


All those who placed orders have been contacted and informed of the error and the process to correct the shipping rate errors.  We appreciate your understanding as we worked through this.


We wish to be seeing you all in person so your t-shirts can be personally handed to you, but unfortunately this year we are unable to. Providing a conference t-shirt was always an important take away and memory from the conference and we did not want this year to be different. This required us to work with a vendor instead of handling shirts ourselves like we have in the past.  Shipping internationally is quite expensive and we have made sure the  lowest rates for package shipping are being utilized.  This does result in a longer transit time but makes the shipping cost more reasonable.


Due to generous offers from our wonderful community members we are looking into local distribution shops for Europe and Asia. Please stand-by for details as we continue research on this option.

John Cook: Pareto and Pandas

$
0
0

This post muses about what it means to learn a software library. I’ll use Pandas as an example, but the post isn’t just about Pandas.

Suppose you say “I want to learn Pandas.” That implicitly assumes Pandas one thing, and in a sense it is. In another sense Pandas is hundreds of things.

At the top level, the pandas module (version 1.2.0) has 142 things inside.

>>> import pandas as pd
    >>> len(dir(pd))
    142

The two most important things inside are the Series and DataFrame objects. They each in turn contain hundreds of things.

>>> len(dir(pd.Series))
    434
    >>> len(dir(pd.DataFrame))
    441

That’s evidence Pandas’ diversity. But here’s evidence of it’s unity: most of the things inside these two objects have the same names.

>>> s = set(dir(pd.Series))
    >>> d = set(dir(pd.DataFrame))
    >>> len(s.union(d))
    491
    >>> len(s - d)
    50
    >>> len(d - s)
    57

Pandas kinda has a fractal dimension, having both complexity and unity. The best way to think about it is not as one monolithic thing, or as hundreds of isolated things. It’s a coherent, but not perfectly coherent, collection of related things. This is true of all software libraries. Pandas is more coherent than most libraries because it was initially the product of one mind, that of Wes McKinney.

This has a couple implications for what it means to “learn Pandas.” Because Pandas is big, you have to explore it strategically, not exhaustively. And because Pandas is coherent, part of what it means to learn Pandas is to develop a feel for the way Pandas does things.

No one is going to learn Pandas by studying every object, every method on every object, and every argument to every method on every object. It’s too big. That’s also unnecessary.

There’s probably something like a Pareto distribution on the usefulness of features. The most commonly used features are used far, far more often than the most obscure features.

It would be interesting to do some kind of survey to see which features are actually used and how often. But I don’t think that’s practical. The easiest thing to do would be to find some large code base that heavily uses Pandas. But that’s not typical of how Pandas is used. Probably most lines of code using Pandas are scattered over millions of small scripts, much of it not in production code.

A well-designed library makes it possible to make good guesses about functionality you haven’t used. You learn the gestalt of the library. You can always look up API documentation as needed, but you can’t develop an intuition for a library just-in-time.

“Learn Pandas” is a daunting goal, and maybe an impossible goal if by “learn” you mean explore exhaustively. But “learn how to do my common tasks quickly in Pandas” and “develop a feel for how to do things in Pandas” are much smaller tasks.

Related posts

The post Pareto and Pandas first appeared on John D. Cook.

Python Pool: 5 Ways to Remove the Last Character From String in Python

$
0
0

Introduction

String Manipulation is the most important feature in python. You can manipulate the string by many methods such as String slicing techniques, looping over the elements, and string methods. Sometimes we come to the situation where we require to remove the last character from the string in Python as it is added by mistake. So here, we will focus on removing the last character from the string.

Let us discuss certain methods through which we can remove or delete the last character from a string:

Different Ways to Remove Last Character From String in Python

Remove the Last Character From String in Python

1. Using Positive index by slicing

We can remove or delete the last character from the string by accessing the given string’s positive index.

Let us look at the example for the better understanding of the concept:

#python slicing using positive index
Str = "Latracal Solutionss"
l = len(Str)

Remove_last = Str[:l-1]

print("String : ",Remove_last)

Output:

String :  Latracal Solutions

Explanation:

Here, we have taken a sample string. Then we have calculated the length of the string to access the last character of the string. Finally, we have used slicing for removing the last character of the string and then printed the output. Hence, we have seen that the unwanted character has been removed.

2. Using Negative Index by Slicing

We can also remove or delete the last character from the string by accessing the given string’s negative index.

Let us look at the example for the better understanding of the concept :

#python slicing using negative index
Str = "Latracal Solutionss"

#last character is always at -1 position
Remove_last = Str[:-1]

print("String : ",Remove_last)

Output:

String :  Latracal Solutions

Explanation:

Here, we have taken a sample string str = “Latracal Solutionss.” Then for removing the last character of the string, the indexing always starts from -1. By slicing it from the -1 index, we have removed the last character of the string. At last, we have printed the output. Hence, we have seen that the unwanted character has been removed.

3. Using the rstrip function to Remove Last Character From String in Python

The string method rstrip is used to remove the characters from the right side of the string that is given to it. So we will be using it to remove or delete the last character of the string. It can be done only in a single line of code and straightforward method to remove the last character from the string.

Let us look at the example for the better understanding of the concept:

#taking input from the user
str = input("Enter the string : ")

remaining_string = str.rstrip(str[-1])

print("String : ",remaining_string)

Output:

Enter the string : Latracal solutionss
String :  Latracal solution

Explanation:

Here firstly, we have taken input from the user in the form of a string. Secondly, we have used the rstrip function in the given string and operated it to remove the last character from the string. At last, we have printed the output. Hence, we have seen that the unwanted character has been removed.

4. Using for loop to Remove Last Character From String in Python

We can also use the for loop to remove or delete the last character from the string. We will take out the length of the string from the len() function. Then we will take an empty string. After that, we will run the loop from 0 to l-2 and append the string into the empty string. At last, we will print the output as the remaining string.

Let us look at the example for the better understanding of the concept:

#sample string
str = "Latracal Solutionss"

#length of the string
l = len(str)

#empty string
Final_string = ""

#using loop
#add the string character by character
for i in range(0,l-2):
    Final_string = Final_string + str[i]

print("Final String : ",Final_string)

Output:

Final String :  Latracal Solution

Explanation:

Here firstly, we have taken the sample str = “Latracal Solutionss.” Secondly, we have calculated the length of the string through the len() function. Thirdly, we have taken an empty string to append the string to form the new string. Fourthly, we have used the for loop from the 0th index to l-2, and by each iteration, we have appended the character in the empty string. At last, we have printed the final string formed. Hence, we have seen that the unwanted character has been removed.

5. Using regex function

We can use regex() function in python, to match 2 groups in a string i.e.

  • Group 1: Every character in string except the last characters.
  • Group 2: Last character of a string.

Let us look at the example for the better understanding of the concept :

import re

def second_group(m):
    return m.group(1)

#sample string
str = "Latracal Solutionss"

# Remove last character from string
Final_String = re.sub("(.*)(.{1}$)", second_group, str)
print("Final String : ",Final_String)

Output:

Final String :  Latracal Solutions

Explanation:

Here firstly, we have imported the re module from python for regex. Secondly, we have made the Second_group function which will only return only group 1 from the match object delete other groups. Thirdly, we have taken a sample string str = “Latracal Solutionss.” Fourthly, we have applied the function with the proper syntax and tried to remove the last character only by putting value = 1 in the given syntax. Finally, we have printed the output and checked that the last character from the string was removed or deleted.

Also, Read>>7 Ways to Remove Characters from String in Python

Conclusion

In this tutorial, we have learned about all the methods through which we can remove or delete the string’s last character. All the methods are explained in deep with the help of examples. The basic and the easiest method was to use positive and negative indexes. You can use any of the methods you like to use according to your program or project requirement.

However, if you have any doubts or questions, do let me know in the comment section below. I will try to help you as soon as possible.

Happy Pythoning!

The post 5 Ways to Remove the Last Character From String in Python appeared first on Python Pool.

Talk Python to Me: #307 Python from 1994 to 2021, my how you've grown!

$
0
0
Python has changed a lot since its inception 30 years ago. On this episode, you'll meet Paul Everitt and Barry Warsaw. They have both been involved with Python since the very first Python conference (called SPAM1 even). We discuss how it's changed but also how so many of the pressures and ideas from the very early days are still playing out in 2021. I'm sure you'll enjoy all the stories and reminiscing.<br/> <br/> <strong>Links from the show</strong><br/> <br/> <div><b>Paul on Twitter</b>: <a href="https://twitter.com/paulweveritt" target="_blank" rel="noopener">@paulweveritt</a><br/> <b>Barry on Twitter</b>: <a href="https://twitter.com/pumpichank" target="_blank" rel="noopener">@pumpichank</a><br/> <br/> <b>Episode live stream</b>: <a href="https://www.youtube.com/watch?v=-ff1rSf4Yjs" target="_blank" rel="noopener">youtube.com</a><br/> <br/> <b>A Python Developer Explores Apple's M1 video</b>: <a href="https://www.youtube.com/watch?v=PnxlHfGdihI" target="_blank" rel="noopener">youtube.com</a><br/> <b>Let's Build a Fast, Modern Python API with FastAPI webcast</b>: <a href="https://www.youtube.com/watch?v=sBVb4IB3O_U" target="_blank" rel="noopener">youtube.com</a><br/> <br/> <b>Python.org 1997</b>: <a href="https://web.archive.org/web/19970501011626/http://www.python.org/" target="_blank" rel="noopener">python.org</a><br/> <b>Python is eating the world: How one developer's side project became the hottest programming language on the planet</b>: <a href="https://www.techrepublic.com/article/python-is-eating-the-world-how-one-developers-side-project-became-the-hottest-programming-language-on-the-planet/" target="_blank" rel="noopener">techrepublic.com</a><br/> <b>Some of Barry's music</b>: <a href="https://soundcloud.com/pumpichank/popular-tracks" target="_blank" rel="noopener">soundcloud.com</a><br/> <b>Barry’s early history of Python slides from BayPiggies</b>: <a href="https://slides.com/pumpichank/python-early-history" target="_blank" rel="noopener">slides.com</a><br/> <b>Backstory and liner notes for the Zen of Python song</b>: <a href="https://wefearchange.org/2020/05/zenofpython.rst.html" target="_blank" rel="noopener">wefearchange.org</a><br/> <b>Zen of Python song</b>: <a href="https://www.youtube.com/watch?v=i6G6dmVJy74" target="_blank" rel="noopener">youtube.com</a><br/> <br/> <b>PyCon Ticket Giveaway</b>: <a href="https://talkpython.fm/pycon2021" target="_blank" rel="noopener">talkpython.fm/pycon2021</a><br/></div><br/> <strong>Sponsors</strong><br/> <br/> <a href='https://talkpython.fm/square'>Square</a><br> <a href='https://talkpython.fm/linode'>Linode</a><br> <a href='https://talkpython.fm/training'>Talk Python Training</a>

Moshe Zadka: So you want to create a universe

$
0
0

A story about looking for a universe, and finding a pi(e)

This is fine. You need not feel shame. Many want to create a universe. But it is good you are being careful. A universe with sentient beings is a big moral responsibility.

It is good to start with something small. The smallest. Move up from there. So there is one point, that can move in one dimension. Just interesting enough to be exciting, but little chances of messing anything serious up.

It is good if the way the point moves can be described by a function. At each point in time, \(t\), the point is at a place, \(x\). This mapping is a function.

$$ f: \mathbb{R} \to \mathbb{R} $$

Right now there are no sentient beings in the universe. But inevitably, you will want to create a universe with such beings. Beings that want the universe to be predictable. Better start practicing now!

This means you want \(f\) to have a simple mathematical description. The concept of death is morally complicated, but you want to allow for the potential of beings with a limited lifespan. This means that figuring out how \(f\) changes should not rely on \(t\).

One way to have a predictable function that does not depend on \(t\) is to define \(f\) with a position-independent differential equation (PIDE): an equation that involves \(f\) and its derivatives.

You are just starting out, so why not have the simplest PIDE?

$$ f' = f $$

Any simpler and your universe will be constant! Solving differential equations is hard. A solution probably exists, right? Hopefully, one that is more interesting than the constant zero function.

$$ f = 0 $$

Yes, it definitely solves it, but that sounds like a really boring universe. If a solution that is not \(0\) at \(0\) exists, \(f\), then

$$ f/f(0) $$

is also a solution since derivatives are linear. The function \(f/f(0)\) is an interesting solution. Call it \(e\).

For a constant \(c\), \(e(x + c)/e(c)\) solves the equation and is \(1\) at \(0\). Differential equations have a unique solution with the same starting condition, so

$$ e(x + c)/e(c) = e(x) $$

or, equivalently

$$ e(x + c) = e(x)e(c) $$

As a result, with a little induction,

$$ e(n/m) ^ m = e(1)^n $$

or

$$ e(n/m) = \sqrt[m]{e(1) ^ n} $$

If you use the convention that

$$ \sqrt[m](a) = a^{1/m} $$

, you get

$$ e(n/m) = e(1)^{n/m} $$

Your universe is one dimensional, but you cannot help thinking about two dimensions. Is there a way to get a second dimension for "free"? You decide to leave the details for later, and for now, just see if \(e\) can be extended to the complex numbers. This is sure to come handy later.

Since \(e' = e\), \(e'' = e\) nd so on. In particular

$$ 1 = e(0) = e'(0) = e''(0) ... $$

This means that the Taylor series for \(e\) looks like

$$ e(x) = \Sigma_{n=0}^{\infty} x^n / n! $$

This converges for every \(x\). Because it converges absolutely everywhere, it can be extended to complex number with the same formula, and \(e' = e\) over the complex numbers as well.

If \(t\) is a real number,

$$ e(-it) = \overline {e (it) } $$

and so

$$ 1 = e(0) = e(it + (-it)) = e(it)e(-it)=e(it)\overline{(e(it))} = || e(it) || $$

Nifty, for real \(t\), you get that \(e(it)\) is on the unit circle. But where on the unit circle?

$$ \operatorname{Re} e(2i) = 1 - 2^2/2! + 2^4/4! - 2^6 / 6! + d $$
$$ = 1 - 2 + 16/24 - 64/720 + ... = -1 + 2/3 - 4/45 + d = -0.4\bar{2} + d $$

Where \(d\) represents the rest of the series.

We can estimate \(|d|\) as follows:

$$ |d| \leq \Sigma_{n=0}^{\infty} 2^{8 + n}/(8 + n)! \leq \Sigma_{n=0}^{\infty} ((2/315) 2^n / 4^n)) $$
$$ \leq 2/315 < 1/10 = 0.1 $$

and

$$ \operatorname{Re} e(2i) < -0.4\bar{2} + d < -0.4\bar{2} + 0.1 < -0.3 < 0 $$

Now

$$ \operatorname{Re} e(0i) = \operatorname{Re} e(0) = \operatorname{Re} 1 = 1 > 0 $$

and \(t \to \operatorname{Re} e(ti)\) is a continuous function. This means there must be a minimal \(t\) such that

$$ \operatorname{Re} e(ti) = 0 $$

This is fun! It is not obvious how this ties into the original goal of building universes, but it definitely cannot hurt! This is an interesting number. You decide to give it a name.

The minimal \(t\) such that \(\operatorname{Re} e(ti) = 0\) will be \(\rho\). Since \(||e(\rho i)|| = 1\), this means that

$$ e(\rho i) = \pm i $$

and so

$$ e(4 \rho i) = (\pm 1)^4 i^4 = 1 $$

Looks like \(4 \rho\) is even more interesting than \(\rho\), maybe it should have its own name. How about \(\tau\)? With this new symbol, we get

$$ e (\tau i + x) = e(\tau i)e(x) = 1e(x) = e(x) $$

So \(e\), the tentative universe-evolution function, has a period of \(\tau i\). You did not expect it. It is good you started with a simple universe, there are many things to learn before creating an interesting one.

You had a special name for \(\rho\) and for \(4 \rho\), it seems almost rude not to have a name for their geometric mean. All this universe creation is hungry work, though. It would be so much easier to think if you had a piece of...

Back to the topic at hand, you decide to call the geometric mean of \(\rho\) and \(\tau\), \(\tau / 2\), \(\pi\):

$$ \pi = \tau / 2 $$

Time to relax and eat a nice piece of pie. You definitely deserved it. Whether it is savory or sweet, a pie is delicious. Enjoy it. Savor it. The universe will be waiting for you, right here.

Satisfied and with a full tummy, you get back to the universe. You gave \(\rho\), \(\pi\), and \(\tau\) names.

Any idea what their approximate value is?

You know that \(0 < \rho < 2\), but this is a pretty wide gap of ignorance. Calculating \(e(it)\), for \(t\) in that range, seems to converge quickly.

Just a handful of terms gave you something accurate to within \(2/315\). Time to leave abstract thinking, and crank up your universe simulation machine. You want to have a sense of the values involved here.

importmathdefapproximate_re_e_i(t):returnsum((-1)**n*t**(2*n)/math.factorial(2*n)forninrange(0,10))

With a decent approximation of $ \operatorname{Re} e(it) $, you look for where the function is zero using binary search. It might not be the fastest way, but your universe simulator can handle it.

deffind_zero():low,high=0,2while(high-low)>0.001:midpoint=(high+low)/2value=approximate_re_e_i(midpoint)ifvalue<0:high=midpointelse:low=midpointreturn(high+low)/2

Now it is time to activate the simulator, and find the values.

rho=find_zero()tau=4*rhopi=tau/2

Wonderful progress on the universe for today. A great day. A wonderful day. A day you want to celebrate.

But when?

With \(\rho <2\), you know that \(\pi <8\). Sounds like the integer part of it could be the month, maybe?

month=int(pi)

All that is left is to choose a day. There are thirty days in a month, so hopefully two digits will do here.

rest=pi-monthday=int(rest*100)
importdatetimeyear=datetime.date.today().yearcelebration=datetime.date(year=year,month=month,day=day)
print("Celebrate on",celebration)
Celebrate on 2021-03-14

Enjoy your pie!

Test and Code: 148: Coverage.py and testing packages

$
0
0

How do you test installed packages using coverage.py?

Also, a couple followups from last week's episode on using coverage for single file applications.

Sponsored By:

Support Test & Code : Python Testing

Links:

<p>How do you test installed packages using coverage.py? </p> <p>Also, a couple followups from last week&#39;s episode on using coverage for single file applications. </p><p>Sponsored By:</p><ul><li><a href="https://linode.com/testandcode" rel="nofollow">Linode</a>: <a href="https://linode.com/testandcode" rel="nofollow">If it runs on Linux, it runs on Linode. Get started on Linode today with $100 in free credit for listeners of Test & Code.</a></li></ul><p><a href="https://www.patreon.com/testpodcast" rel="payment">Support Test & Code : Python Testing</a></p><p>Links:</p><ul><li><a href="https://testandcode.com/147" title="episode 147: Testing Single File Python Applications/Scripts with pytest and coverage" rel="nofollow">episode 147: Testing Single File Python Applications/Scripts with pytest and coverage</a></li><li><a href="https://coverage.readthedocs.io/en/coverage-5.5/source.html" title="Specifying source files — Coverage.py documentation" rel="nofollow">Specifying source files — Coverage.py documentation</a></li><li><a href="https://hynek.me/articles/testing-packaging/" title="Testing & Packaging - Hynek" rel="nofollow">Testing & Packaging - Hynek</a></li><li><a href="https://beyondgrep.com/" title="ack" rel="nofollow">ack</a></li></ul>

Real Python: The Real Python Podcast – Episode #51: Navigating Options for Deploying Your Python Application

$
0
0

What goes into the decision of how to host your Python code or application in the cloud? Which technology stack is the right size for your project? This week on the show, we have Calvin Hendryx-Parker. Calvin talks about cloud hosting options, infrastructure choices, and deployment tools.


[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

PyBites: Naughts and Crosses Gets a Little Help

$
0
0

Header image: Naughts and Crosses, Os and Xs, Tic Tac Toe

Index

Introduction

Would you like to play a game?(Wargames (1983))

If you happen to have recently hacked into NORAD (North American Aerospace Defense Command) then it is quite possible that you are stuck in the 1980s and are looking to play Global Thermonuclear War(Wargames (1983)), take it from me: it really won't turn out hope you hope.

On the other hand if you are looking for what is arguably the simplest of all strategy games, then you've come to the right place. This article was inspired by the PyBites Coding Challenge 12Build a Tic-tac-toe Game.

Game concept

Image of naughts and crosses grid

Traditionally, naughts and crosses (Wikipedia article on Tic-tac-toe) is a pencil and paper game for two players: one playing the symbol 'O', pronounced 'naught' or 'oh', and the other playing 'X', pronounced 'cross' or 'ex'. The game begins with a 3x3 grid of nine cells, and the players take turns at drawing their symbols into one of the empty cells. The aim is to achieve the occupation of three cells in a row, horizontally, vertically or diagonally, with their own symbol, whilst at the same time interfering with their opponents attempts to do likewise. There are eight possible winning lines are shown in the figure below.

Image of winning positions

Basic game implementation

The first thing that is necessary in implementing any game is deciding how it should be represented both to the player and when it is held in the computer memory. We are working with fixed a three by three grid in this game, so it is easy to see that a series of nine consecutive elements of an array can be used. Like this:

Image of array of nine memory cells numbered zero to nine

These memory cells, in turn, map to the three rows of three cells that will be displayed to the player. To help the player to associate the positions of the cells, however, it is convenient to give a different external mapping that is based upon something familiar. A familiar three by three grid is found on the keyboard numeric keypad, but this is 'upside down' compared to how we have suggested the internal representation will be achieved.

This difference in representation can be handled in a few different ways:

  • change the internal representation to match the external representation;
  • change the external representation to match the internal representation; or
  • find a way to translate between the two representations as and when needed.

There are implications for each of these options. Firstly, if the internal representation is changed, it will be necessary to extract the details for the grid when it is being displayed in a non-sequential order. This is not an impossible situation, but adds a layer of complexity that is probably undesirable. Secondly, if the external representation is changed, then it inconveniences the player who will have to think more about the value of the cell rather than the position. Thirdly, and finally, translating between the two systems allows the positives of both representations to be exercised with a minor inconvenience of ensuring that the interface between the two sides is patrolled appropriately.

Assuming that the third option is taken we have the following representations:

Image of internal cell numberingImage of external cell numbering

Starting the code

It is always wise to start off a Python file with a descriptive comment about the file, who wrote it, any copyright declarations, the place within a project that it belongs and what the weather is like when you write the code, possibly less likely on that last one. So, we begin our project and, as it is going to be a single file, just comment on what the whole project is about. If this was to be a one of a number of files then the description would focus upon the functionality of this particular file.

"""    Proper name: 'Naughts and Crosses'    Irish name: 'Xs and Os'    American name: 'Tic Tac Toe'    Reason: Who on Earth has a clue?  It's an historical thing."""

Lovely. That'll do nicely. Not much coding going on there though. There are going to be some imports of various modules to help us with the coding, straight off the bat we know that we are going to be dealing with some kind of memory cells, this brings to mind that we'll need coding hints for things like List and perhaps Union so we'll add those in for a starter. Later on we'll add more in at this place in the file.

fromtypingimportList,Union

There are a number of elements of the game that can be usefully given names to help to document the code as we go along. The extra advantage to defining things by name at this point is that if we want to change things later on, then any changes made here automatically reflect through the whole code. (Hurrah!)

So, what elements need to be defined? We have already identified that there are internal and external representations involved, therefore those should be defined along with the mapping between the two. Externally we know that we're going to have 'O' and 'X' to show the symbols—that much is obvious—but there is a third symbol too: the blank. The blank could be just a space character, but that isn't very visible if you want to see the grid, particular when it is empty, but also when working out which row and column a single symbol might be on. For this reason we'll use the underscore ('_') to represent a blank space.

# External representations of the playing symbols BLANK_SYM:str='_'O_SYM:str='O'X_SYM:str='X'

That will make any grids clear:

 ___     _X_     ___
 ___     ___     _O_
 ___     ___     ___
Blank / One X / One O

What about internal representations? Does it matter? We could use the same symbols, however we do know that string comparisons are slower than integer comparisons, and so it would probably be better to use the quicker of the two given that there are a lot of comparisons that will be made. Let us keep it simple and just use the values zero to two.

# Internal representations of the playing symbols BLANK_VALUE:int=0O_VALUE:int=1X_VALUE:int=2

Now that we have those defined, we can create a couple of dictionaries that map between the two systems:

# Translation between systemsVAL_TO_SYM:dict={BLANK_VALUE:BLANK_SYM,O_VALUE:O_SYM,X_VALUE:X_SYM,}SYM_TO_VAL:dict={BLANK_SYM:BLANK_VALUE,O_SYM:O_VALUE,X_SYM:X_VALUE,}

The chances are that whilst we are going through checking for winning lines and such like, we might want a quick way to refer to the opponent of the current player; we might not need it, but it could prove useful:

OPPONENT:dict={O_VALUE:X_VALUE,X_VALUE:O_VALUE,}

Just a couple of other things that need to be taken care of: the translation of internal to external grid positions, and vise versa. Looking at the graphical images of the two representations, it is easy to write out a couple of dictionary constants mapping between the two:

Image of internal and external representations of grid

INTERNAL_TO_EXTERNAL:dict={0:7,1:8,2:9,3:4,4:5,5:6,6:1,7:2,8:3}EXTERNAL_TO_INTERNAL:dict={7:0,8:1,9:2,4:3,5:4,6:5,1:6,2:7,3:8}

In order to help validate the player input we'll create a list of valid positions for their pieces, remember that range() returns value that includes the first parameter, but excludes the second parameter:

VALID_POSITIONS:Set[int]=set(range(1,10))

I originally created this as a list but realised later that it was better (essentially more efficient) as a set so it is necessary to add Set to the from typing import line

Oh, and of course we need a list of winning combinations so that it is quick to check. It is convenient to specify a winning line as a tuple of three integers, so here we have a list of (tuples of (three integers)): ah, Tuple, that's another item to import from the typing module. The winning lines are describing the internal numbering as per this diagram:

Image of winning combination in same order as defined arrayImage of internal numbering

# A winning combination exists for three symbols in a row:WINNING_COMBINATIONS:List[Tuple[int,int,int]]=[(0,1,2),(3,4,5),(6,7,8),(0,3,6),(1,4,7),(2,5,8),(0,4,8),(2,4,6),]

The main class

Class. Yes, this game is going to be encompassed within an object! We can start getting into the nitty-gritty right off by declaring our class… in deference to our US cousins we'll call our class TicTacToe—it's both shorter than NaughtsAndCrosses and less likely to cause argument over 'Nau…' versus 'Nou…' (Naught vs. Nought).

At the start of the class we'll define a few variables that are going to be useful throughout the game code, and define the types that we're expecting them to hold:

classTicTacToe:# define the types of various internal variables_board:List[int]_turn:int_turn_cycle:Iterator[int]_move:int

Going through these:

  • _board: this will hold the current internal representation of the playing area;
  • _turn: keeping track of whose turn it is will be very necessary, this should only ever be O_VALUE or X_VALUE;
  • _turn_cycle: a useful iterator object that returns an endless sequence of O_VALUE, X_VALUE, O_VALUE, X_VALUE, and so on; and
  • _move: a count of the number of moves taken. There are nine spaces and so there is a maximum of nine moves.

As we're adding yet another type hint here, it's probably best to make sure that our import statement has caught up with all the various additions:

fromtypingimportList,Union,Set,Tuple,Iterator

Hopefully that's got all our typing needs covered now.

An important part of every class in the __init__() initializer method, or Constructor, that is used to set up the blank object:

def__init__(self):"""Constructor, allocate the blank board"""# Create an array of cells to hold the grid positions.self._board=[BLANK_VALUE]*len(VALID_POSITIONS)self._turn_cycle=cycle([O_VALUE,X_VALUE])self._turn=self._next_turn()self._move=0

A little explanation may be useful here, the _board is set by the curious construction above: multiplying a single element list by the length of a set?? In this case the * does not literally indicate mathematical multiplication, but rather duplication: it indicates that the list element is to be repeated. As an example here is a sort console log showing the use of the functionality:

Python 3.8.5 (default, Sep  4 2020, 07:30:14) 
[GCC 7.3.0] on linux
>>> f = [1]
>>> f
[1]
>>> f * 8
[1, 1, 1, 1, 1, 1, 1, 1]

This is a simple method of creating a list containing a repeated element, however, it should be noted that it is not good for very large lists due to the requirement of core memory in making the list. We know that in this case the length of the VALID_POSITIONS set is only nine so there is no worry about excessive memory use.

The _turn_cycle variable is created with a very useful element of the itertools module. cycle creates a repeating iterator from the provided iterable parameter. Here we are passing a two element list, consisting of the two player options, and in return we get an object that will respond to the next() method by returning each element in turn, when the list is exhausted it automatically starts at the beginning again (Python Docs, itertols.cycle(iterable)). Our imports do not currently include the itertools module, so we need to add the extra import line at the top of the file:

fromitertoolsimportcycle

Looking back at the __init__() method it can be seen that _turn is initialised by the return value of another method, _next_turn(). It is possibly easy to guess what this method does, but let's sketch it in now to be sure:

def_next_turn(self)->int:returnnext(self._turn_cycle)

Yes, we're using the _turn_cycle variable to pick off the next player; this is why it was necessary to assign _turn_cycle before _turn in the __init__() method. It might seem a little odd to have a 'private' method that just returns the value from a private iterator, but there are two reasons for doing this: 1. it ensures that the syntax for next(self._turn_cycle) is consistently followed; and, more importantly 2. it allows for additional functionality to be performed when the next turn is being selected without having to seek out every instance of the operation.

Arguably one of the most important elements of the object is to provide a visible version of the board layout, remember that our internal representation follows for directly printing out in three rows so there is no need to manipulate the order before generating the output:

def__str__(self):"""Print the board"""return'\n'.join(' '.join(VAL_TO_SYM[c]forcinself._board[s*3:(s+1)*3])forsinrange(3))

This looks a little complex because it is using multiply nested str.join() statements. It may become clearer if we unwind it into a longer version:

def__str__(self):"""Alternative view of same function"""output=[]forsinrange(3):row=[]fortinrange(3):c=self._board[s*3+t]row.append(VAL_TO_SYM[c])output.append(' '.join(row))return'\n'.join(output)

It is much easier to see now that there are two loops counting from zero to two (remember that the value in range() is exclusive on the upper limit); the outer loop s counts the rows, whilst the inner loop t counts the columns. A cell value c is extracted from the _board using an offset calculation, and then the VAL_TO_SYM dictionary translates the internal value to the external symbol and adds it to row. Once a full row has been compiled, it is added to output as a single string with spaces between each value using ' '.join(row). Finally, once the three rows have been added to output, they, in turn, are joined together with a newline character inserted between each row. The joined together rows are returned as a single string, completing the method. The 'single line' version of the method, above, has all the same functionality, but tightly parcelled up in a nested list comprehension.

Now we're going to introduce a couple of helper methods to the class. These two methods will not actually change any element of the class object, but rather compute the logical mappings between internal and external grid positions. Since they are not making any changes, they can be marked as 'static' methods. All they actually do is to use the dictionaries that we set up earlier to perform the translation. Using methods in this way, again allows modifications to structures without requiring wholesale rewrites.

@staticmethoddef_ndx_to_cell_(ndx:int)->int:returnEXTERNAL_TO_INTERNAL[ndx]@staticmethoddef_cell_to_ndx_(cell:int)->int:returnINTERNAL_TO_EXTERNAL[cell]

In order to save on typing, we have named these methods using the abbreviation 'ndx' for the external index, and 'cell' for the internal cell offset.

Right, we're now up to the point where we're going to actually put a players move onto the board. As this is an interaction between the player and the computer, it is always wise to attempt to anticipate things that the player could do wrong… such as entering the number of a cell that has already been played, or even trying to play in cells that do not exist! Python has the perfect mechanism for flagging up this kind of error: the Exception system. We can create a couple of custom exceptions for ourselves by putting the following code above the main class declaration:

classBlockedCell(Exception):passclassInvalidMove(Exception):pass

As easy as that! To create a custom exception all we need to do is to create a class inheriting from the systems Exception class. If we wanted our exception to do something beyond the normal exception system, such as automatic logging of the error, then it can be built onto this definition; otherwise, if we just want a new exception name, it is appropriate to state pass, meaning (in this case) don't add anything to the standard class.

So to the player_move() method. As the game object is stateful (it knows what the current state of play is and keeps track of which player is playing), we can just pass the location that the user wishes to play in. This will be an external position as it's coming from the user, so we'll be making use of the _ndx_to_cell_() helper method to do the appropriate translation once we've made sure that the target_position is on the board. Here's the code:

defplayer_move(self,target_position:int):"""        Attempt to place the player move        May raise exceptions: BlockCell, InvalidMove"""iftarget_positioninVALID_POSITIONS:cell=self._ndx_to_cell_(target_position)ifself._board[cell]isBLANK_VALUE:self._board[cell]=self._turnelse:raiseBlockedCell(f'Cannot play at {target_position}, it is already held by {VAL_TO_SYM[self._board[cell]]}')else:raiseInvalidMove(f'Invalid move, {target_position} not available')

Notice that the documentary comment (the docstring) at the start of the method notes that it can raise exceptions: this is a good habit to cultivate so that you know by requesting the documentation either by using an IDE that automatically does so, or by typing help(TicTacToe.player_move) in the interactive python console:

>>> from tictactoe import TicTacToe 
>>> help(TicTacToe.player_move)
Help on function player_move in module tictactoe:

player_move(self, target_position: int)
    Attempt to place the player move

    May raise exceptions: BlockCell, InvalidMove

The second thing to note is that it doesn't assume that the player is either 'O' or 'X', rather it uses the _turn variable to identify the current player. In this way, the same routine can play to the board for either player.

Once the player has made their move, the (private) method _next_turn() will advance to the next player. Why was it made as a 'private' method? The reason was because it is designed to work with internal representations, it is a wrapper around the _turn_cycle iterator, and we want to stop the internal values being inadvertently revealed outside the object. We need a public method that can be used to switch players, save the active player, and which returns the external representation of the player whose turn it is:

defnext_player(self)->str:self._turn=self._next_turn()returnVAL_TO_SYM[self._turn]

We use the private method to obtain the next scheduled player, save it as the current _turn and then use the VAL_TO_SYM dict to return either 'O' or 'X' as appropriate.

Who's the winner?

To summarise, our TicTacToe class so far:

  • sets up a blank board;
  • keeps track of which player has the turn;
  • allows the player to place a symbol on an empty space; and
  • returns a textural representation of the board.

There is something missing though: deciding a winner!

There are three distinct outcomes for a finished game: win, lose or draw. In order for the outcome to be revealed we can create properties that evaluate the state of the board with respect to the current player. As there is a common thread in all these tests, we make a helper method, find_winner(), to hold the common code.

deffind_winner(self)->Union[int,None]:"""Find a winner, 'O', 'X' or None"""forsin[O_VALUE,X_VALUE]:ifany(all(self._board[c]==sforcincombo)forcomboinWINNING_COMBINATIONS):returnsreturnNone

There is a strange looking construct in the if statement here any() and all() are builtin functions of Python, (Python Docs, all(iterable) and Python Docs, any(iterable)) that can evaluate a group of boolean expressions in a way that smartly returns a result as soon as it is confident. Taking any() as the example, it will work through all the elements of the iterable until it find one that evaluates to True, if it gets to the end of the iterable, or the iterable is empty, then it returns False. This technique of stopping as soon as a result is confident is known as short circuit evaluation. As a regular function it would be coded as:

defany(iterable):forelementiniterable:ifelement:returnTruereturnFalse

Similarly, all() works through the iterable until it finds one that is False:

defall(iterable):forelementiniterable:ifnotelement:returnFalsereturnTrue

(These two examples are taken from the Python Docs page on library functions.) Going back to the find_winner() method, we can see that there is an all() function nested inside an any() function: let us unpick what is going on. Just as with the __str__() method above, we have a nested comprehension, so we can unroll it and see it in an expanded view, taking into account the definitions of all() and any() above:

deffind_winner(self)->Union[int,None]:"""Find a winner, 'O', 'X' or None"""forsin[O_VALUE,X_VALUE]:# ANY of the followingany_ok=FalseforcomboinWINNING_COMBINATIONS:# ALL of the followingall_ok=Trueforcincombo:ifself._board[c]!=s:all_ok=Falsebreakifall_ok:any_ok=Truebreakifany_ok:returnsreturnNone

Starting from the bit that hasn't changed, we're going to loop through the two player values to make the comparisons. The variable s is being used to hold the currently being tested player, normally it would be recommended that a longer, descriptive, name would be used but to do so in the original short method would have resulted in an overlong line with negligible benefit: this kind of trade off should be considered carefully before being used too often. At the next level we process the any() function. As we're not using the builtin function, we need a flag to track when we can short circuit the test. We are going to loop through each of the possible WINNING_COMBINATIONS, passing each combination to the all() function level. Again a flag is needed to keep track…

We've come to the meat of the method now: combo holds a particular winning combination of cells, all_ok is set to True as the default 'return' value for the inner loop and any_ok is set to False as the default 'return' value for the outer loop. The inner loop looks through the cells indicated in combo to check if they all have the currently being tested player: as soon as a cell is found that doesn't have that value, all_ok is reset to False and a break is executed to 'escape' from the for c in combo: loop. At this point either the test found a cell that wasn't the right player, or it looped right through finding all the cells belonging to the currently being tested player. If the latter happened, then all_ok will still be set to True and we've found a winning line so we can set any_ok to True and break out of the for combo in… loop. Again we do a check, this time on any_ok to see if there's been a winning line found, and, if so, we actually return the winning player value. Finally if the for s in… loop passes out with no winning line found for either player then we return None. This indicates 'no winner' not 'draw'.

Now that we have that helper method we can create a set of properties that will provide the current state of the game:

@propertydefwin(self)->bool:"""Test if the game is won"""returnself.find_winner()==self._turn@propertydeflose(self)->bool:"""Test if the game is lost"""returnself.find_winner()==OPPONENT[self._turn]@propertydefdraw(self)->bool:"""Test if the game is a draw"""returnnot(any(c==BLANK_VALUEforcinself._board))@propertydefwin_draw_lose(self)->bool:"""Test if the game is still in play"""returnself.winorself.loseorself.draw

The properties win and lose purely check the returned value to see if it is the player or the opponents internal value. On the other hand, draw searches through the board for a BLANK_VALUE, if there are no blanks present then there is no where left to play, therefore, it's a draw. Lastly, the property win_draw_lose checks to see if there is any reason for the game to have ended without being specific.

One final property and we should be ready to play! What is that last property? The player symbol is needed to prompt the appropriate player for their turn:

@propertydefplayer(self)->str:returnVAL_TO_SYM[self._turn]

Let's play

Time to write some code to actually use the TicTacToe class. In the normal line of development we would have been developing this piece of code in tandem with the actual class, however, in order to concentrate on the class it has been left until this point. As it's going to be tacked to the end of the file we'll use a special test so that it can still be used as a module if so desired:

if__name__=="__main__":

When Python imports a file through an import instruction (or similar), it automatically sets the special variable __name__ to the name of the module. However, when Python loads a module as a script in order to 'run' it, then __name__ is set to the value "main" allowing us to easily detect when we're, ahem, the main file.

We are going to set up a 'forever' loop to keep playing… we won't ever want to stop will we??

whileTrue:

At the start of our game loop we create an instance of the TicTacToe object, and announce our intention:

game=TicTacToe()print("Let's play Naughts and Crosses!")

There is an inner loop here that keeps going until the game is won, drawn or lost… we have a useful property to check that. Each time around the loop we print out what the board look like, asking the object to print like this automatically invokes the __str__() method:

whilenotgame.win_draw_lose:print('\nCurrent state of game:')print(game)

Now we have some player input so we should anticipate problems, using a try…except… construct will make sure we have a chance of keeping control of any such problems. We ask for a game play, using game.player to display either 'O' or 'X' and then attempt to convert that typed in game play into an integer. Note that if the player does not enter a number then the attempted conversion will raise an exception, we'll handle this later but we need not worry about it in the rest of this piece of code.

try:mv=input(f'Where would you like to play your {game.player}? ')position=int(mv)

We have the players intended move now in position (external representation), so that can be passed to game.player_move() to place the appropriate symbol. Remember that player_move() can raise a couple of exceptions, however, as we are in the middle of a try…except… construct we need not worry about errors at this point and can continue knowing that the move was accepted.

game.player_move(position)

This is the point at which we check for the game having been won or drawn, as the current player cannot have lost by making a move, we need not check for the 'lose' condition. If either of the two conditions are met then we print out the game grid in its final position along with a message indicating the win or draw status. Then there is a break in each case: this will exit the while not game.win_draw_lose loop allowing a new game to begin.

ifgame.win:print(game)print(f'**WIN** We have a WINNER!!  Well done {game.player}.')breakifgame.draw:print(game)print('**DRAW** That was a little pointless in the end.')break

If the game is not won or drawn though, then we move the game object on to the next player before repeating the play position input at the top of the while not game.win_draw_lose loop.

game.next_player()

Remember those error possibilities with the game.player_move(position)? Well here at the end of the try… construct we have a series of except… clauses picking up on the specific errors that we are expecting. In each case the response is to print an error message as appropriate and then to return to the top of the while loop.

exceptInvalidMove:print(f'Poor choice, Grasshopper, "{mv}" is not and acceptable move: use the numeric keypad layout!')exceptBlockedCell:print(f'Sorry that spot is already taken.')

There was one other possible exception that we anticipated: when converting the players typed in move, we attempt to convert the string to an integer. This can raise a ValueError that we pick up here to give an appropriate error message before going on back to the top of the while loop.

exceptValueError:print(f'Please indicate position as though it were the numeric keypad.')

At this point, if everything works out as it should, you should have a working Noughts and Crosses game for two players; or yourself playing two sides!

I always lose playing against myself

It can get tiring playing just against yourself, perhaps we can spice things up a bit by getting the computer to play against us?

The Wikipedia article on Tic-tac-toe has a set of steps forming a strategy that can be followed when playing a game, these steps are an excellent starting point for programming an artificial opponent. Let us take a look at the steps from a programming point of view and work out if it is possible to craft an opponent:

1. Win

Go for the win! What we actually want to seek here is one of the winning rows which has a gap and two of the current players symbols in.

Examples of one move short of a winning line

It would be useful if we could check each line without actually needing to compare all the values individually. There are actually ten distinct combinations of 'O' and 'X' lines, ignoring ordering and reflections, as can be seen in the following graphic

Image of rows of distinct combinations of Os and Xs

We need a method to:

  • uniquely identify each of the possible lines;
  • particularly pick out two Os or two Xs and a blank, whatever order the symbols are; and
  • not interfere with the existing code logic.

In order to fulfil these requirements we should look at a method that combines the three internal values into a single value that is the same regardless of ordering: this leads us to a summation or product of the values. Let's see how this works out.

Currently, we have:

# Internal representations of the playing symbolsBLANK_VALUE:int=0O_VALUE:int=1X_VALUE:int=2

The combinations result in:

combinationvaluessumproduct
_ _ _0 0 000
O _ _1 0 010
O O _1 1 020
O O O1 1 131
O X _1 2 030
O O X1 1 242
X _ _2 0 020
X X _2 2 040
X X X2 2 268
X X O2 2 154

From these results it is easy to see that two Xs and a blank uniquely give a sum of four and a product of zero; unfortunately two Os and a blank give exactly the same values as one X and two blanks. The product value actually only really identifies when there is a space in the line, and it is necessary to calculate both the sum and the product to correctly identify the one line that we do want. This seems excessive. Part of the problem with separating the identity of 'O' and 'X' is that the values we currently have are factors of each other, perhaps using prime numbers would make a difference?

Suggested new values:

# Internal representations of the playing symbolsBLANK_VALUE:int=2O_VALUE:int=3X_VALUE:int=5

The combinations result in:

combinationvaluessumproduct
_ _ _2 2 268
O _ _3 2 2712
O O _3 3 2818
O O O3 3 3927
O X _3 5 21030
O O X3 3 51145
X _ _5 2 2920
X X _5 5 21250
X X X5 5 515125
X X O5 5 31375

Now we have unique values for both the two Os and the two Xs, in the sum and the product. However, in the summation there is a conflict of identification for the rows consisting of three Os and of a single X. We could experiment with other values, but the product values are fully unique across our requirements.

In order to seek a winning move we can scan through the WINNING_COMBINATIONS to search for a product of 18 for a 'O' win or 50 for an 'X' win. In our code we can do something like this:

# New importsfromcollectionsimportdefaultdictfromfunctoolsimportreducefromoperatorimportmul# Update the internal representations of the playing symbols, this will automatically change # the internal values throughout the program.BLANK_VALUE:int=2O_VALUE:int=3X_VALUE:int=5# Add a new function to the class.classTicTacToe:def_ai_move(self)->int:"""Identify a move for the computer to make"""winning_lines=defaultdict(set)forlineinWINNING_COMBINATIONS:prod=reduce(mul,[self._board[c]forcinline])winning_lines[prod].add(line)

We are creating a dictionary of lines that share the same 'product value', so if there are multiple lines capable of a winning move then they will all be listed in the set associated with the appropriate value. Speaking of winning moves, it would be useful to define the values that we are looking for up at the top of the file, we'll create a dictionary named WINNING_PROD:

# The product of values in an *almost* winning line…WINNING_PROD={O_VALUE:O_VALUE*O_VALUE*BLANK_VALUE,X_VALUE:X_VALUE*X_VALUE*BLANK_VALUE,}

Note that the references are for O_VALUE, X_VALUE and 'BLANK_VALUE' so that if we change those again, they will be automatically updated. Now that that has been done, it is a very simple test to check if a winning line is available.

# 1. Let's see if there is a winning line for the current playerifwinning_lines[WINNING_PROD[self._turn]]:# find the blank in the first winning line.line=winning_lines[WINNING_PROD[self._turn]].pop()return[nforninlineifself._board[n]==BLANK_VALUE][0]

This takes advantage of the defaultdict property of returning an empty entry if the requested index doesn't exist. If there is a set associated with the appropriate index then one item, it doesn't matter which as multiple entries will all be appropriate 'nearly winning lines.' A list comprehension searches out the BLANK_VALUE and finally returns the internal location of the gap by dereferencing the first, and only, element.

2. Block

If we cannot win with the next move, then the next priority is to make sure that our opponent doesn't win… we need to play a blocking move; or, to put it another way, put our symbol in the spot when the opponent could make a winning move. Conveniently we have already created a dictionary holing all the lines that could be played, and that includes 'nearly winning lines' of our opponents too. This means that a block move is just a case of looking for a winning position like above, but for the opponent.

# 2. Block: If the opponent has two in a row, the player#       must play the third themselves to block the opponent.ifwinning_lines[WINNING_PROD[OPPONENT[self._turn]]]:# find the blank to play a block.line=winning_lines[WINNING_PROD[OPPONENT[self._turn]]].pop()return[nforninlineifself._board[n]==BLANK_VALUE][0]

This is almost exactly the same as the first section above, but uses the OPPONENT dictionary to pick out our opponents symbol instead.

The similarities of these two pieces of code indicate a good place to examine for refactoring once we have completed the function.

3. Fork

Creating a fork means placing a piece that can contribute to two separate lines to give a win.

Image showing examples of forks

In this image the first two grids show 'O' having just played a fork move (indicated by the green 'O'); in the second two grids we see just why these positions are called 'forks,' the cyan '+' marks the alternative places that 'X' could play, in each case 'O' wins on the next turn by playing the alternative.

4. Blocking an opponent's fork

In a similar case to the above, blocking an opponent's fork means anticipating when your opponent is about to play a move that will lead to a fork and playing in that spot before they can. If we examine the moves leading up to position A1 above:

Image of set of boards showing each move leading to OXO on the diagonal

'O' starts off playing in the bottom corner, 'X' replies by playing in the middle, then 'O' plays in the opposite corner… at this point 'X' spots that 'O' can play in either of the other two corners (marked with the cyan '+' symbols) and end up with a fork: he cannot possibly block both of these cases though, and so the actual solution is to go for the draw and play in a side.

For now, we'll leave coding of forks because we want some form of weakness in our algorithm that a player might have some small change of winning!

5. Centre

This play is a no-brainer: play in the centre. If this is the very first move of a game, then this is not such a good move though, as playing in a corner is the move that grants your opponent the most opportunities to make a mistake.

# 5. Center: A player marks the center.ifself._board[4]==BLANK_VALUE:return4

6. Opposite corner

If we have progressed this far through our algorithm, then the chances are that there haven't been many moves: let's see if we can find an empty space opposite a space occupied by our opponent:

# 6. Opposite corner: If the opponent is in a corner, the#       player plays the opposite corner.forx,yin[(0,8),(2,6),(6,2),(8,0)]:ifself._board[x]==OPPONENT[self._turn]andself._board[y]==BLANK_VALUE:returny

7. Empty corner

Failing to find an opposite corner, we just select an empty corner.

# 7. Empty corner: The player plays in a corner square.forxin[0,2,6,8]:ifself._board[x]==BLANK_VALUE:returnx

8. Empty side

Finally, all other spaces have been checked, so there can only be side(s) remaining. Select one of those.

# 8. Empty side: The player plays in a middle square on any#       of the 4 sides.forxin[1,3,5,7]:ifself._board[x]==BLANK_VALUE:returnx

By the time we have worked through all of these options a valid move will have been returned—in internal representation. However, what if something that we were not expecting happened? For example, if the _ai_move() method was called when the board has no moves left? It is worth including an exception at the end of a routine that is intended to return a value, rather than leave it to default to returning None:

raiseInvalidMove(f"Couldn't identify a move to make for AI controlled {self.player}")

It's your move machine!

Our _ai_move() method is now as complete as we need it to be, so all that is required is to build a mechanism for it to be called.

First off, as we will want to let the player know what is going on we need to have a method that calls this internal method and translates it to external representation ready to be shown to the user. The easiest, and quickest, way to achieve this is to add a method like this:

defai_move(self)->str:returnstr(self._cell_to_ndx_(self._ai_move()))

We wrap up the internal method in a call to the helper method _cell_to_ndx() before casting it to a string ready for return.

The driver code at the bottom of our file can now be modified to make use of this at any time to request an automatic move selection: notice that it doesn't look for an 'O' move or an 'X' move, it looks purely for a next move. This being the case, we can replace the move request input() with a call to game.ai_move() and receive an appropriate selection regardless of which player is next. So, we can replace:

mv=input(f'Where would you like to play your {game.player}? ')position=int(mv)

with

ifgame.player==O_SYM:mv=input(f'Where would you like to play your {game.player}? ')else:mv=game.ai_move()print(f'\nComputer chooses to play {game.player} at {mv}.')position=int(mv)

and the computer will seamlessly take over playing the 'X' moves.

And finally Esther…

There you have it: a playable naughts and crosses game to keep your little darlings entertained for hours… however, there is always room for improvement. For example:

  • In the _ai_move() code there is a great similarity in the code between options 1 (win) and 2 (block), maybe this could be pulled out into a helper method?
  • Once you start playing, there is no easy way to stop: would it be possible to give an easy escape by watching for the player typing 'quit' or just 'q'?
  • The printing of the grid doesn't give any help as to the numbers to press: you just have an error message telling you that the numeric keypad is the pattern to follow. What is a player is using a laptop without a numeric keypad?

Gists of code:

For your convenience, the completed versions of the naughts and crosses code are stored here:

-- Geoff Riley

Image of slogan: May the code be with you

References

Learn PyQt: PyQt6 vs PySide6: What's the difference between the two Python Qt libraries? — ...and what's exactly the same (most of it)

$
0
0

There is a new version of Qt (version 6) and with it new versions of PyQt and PySide -- now named PyQt6 & PySide6 respectively. In preparation for the Qt6 editions of my PyQt5 & PySide2 books I've been looking at the latest versions of the libraries to identify the differences between them and find solutions for writing portable code.

Interested in the PyQt6 or PySide6 book? If you buy the Qt5 edition you'll get the Qt6 (PyQt6 & PySide6) editions as soon as they're available!

In this short guide I'll run through why there are two libraries, whether you need to care (spoiler: you don't), what the differences are and how to work around them. By the end you should be comfortable re-using code examples from both PyQt6 and PySide6 tutorials to build your apps, regardless of which package you're using yourself.

Background

Why are there two libraries?

PyQt is developed by Phil Thompson of Riverbank Computing Ltd. and has existed for a very long time — supporting versions of Qt going back to 2.x. In 2009 Nokia, who owned Qt toolkit at the time, wanted to make the Python bindings for Qt available in a more permissive LGPL license.

It's called PySide because "side" is Finnish for "binder".

The two interfaces were basically equivalent, but over time development of PySide lagged behind PyQt. This was particularly noticeable following the release of Qt 5 — the Qt5 version of PyQt (PyQt6) has been available since mid-2016, while the first stable release of PySide6 was 2 years later. However, the Qt project has recently adopted PySide as the official Qt for Python release which should ensure its viability going forward. When Qt6 was released, both Python bindings were available shortly after.

PyQt6PySide6
First stable releaseJan 2021Dec 2020
Developed byRiverbank Computing Ltd.Qt
LicenseGPL or commercialLGPL
PlatformsPython 3Python 3

Which should you use? Well, honestly, it doesn't really matter.

Both packages are wrapping the same library — Qt6 — and so have 99.9% identical APIs (see below for the few differences). Anything you learn with one library will be easily applied to a project using the other. Also, no matter with one you choose to use, it's worth familiarizing yourself with the other so you can make the best use of all available online resources — using PyQt6 tutorials to build your PySide6 applications for example, and vice versa.

In this short chapter I'll run through the few notable differences between the two packages and explain how to write code which works seamlessly with both. After reading this you should be able to take any PyQt6 example online and convert it to work with PySide6.

Licensing

The main notable difference between the two versions is licensing — with PyQt6 being available under a GPL or commercial license, and PySide6 under a LGPL license.

If you are planning to release your software itself under the GPL, or you are developing software which will not be distributed, the GPL requirement of PyQt6 is unlikely to be an issue. However, if you want to distribute your software but not share your source code you will need to purchase a commercial license from Riverbank for PyQt6 or use PySide6.

Qt itself is available under a Qt Commercial License, GPL 2.0, GPL 3.0 and LGPL 3.0 licenses.

Namespaces & Enums

One of the major changes introduced for PyQt6 is the need to use fully qualified names for enums and flags. Previously, in both PyQt5 and PySide2 you could make use of shortcuts -- for example Qt.DecorationRole, Qt.AlignLeft. In PyQt6 these are now Qt.ItemDataRole.DisplayRole and Qt.Alignment.AlignLeft respectively. This change affects all enums and flag groups in Qt. In PySide6 both long and short names remain supported.

The situation is complicated somewhat by the fact that PyQt6 and PySide6 use subtly different naming conventions for flags. In PySide6 (and v5) flags are grouped under flag objects with the "Flag" suffix, for example Qt.AlignmentFlag -- the align left flag is Qt.AlignmentFlag.AlignLeft. The same flag group in PyQt6 is named just "Qt.Alignment". This means that you can't simply choose long or short form and retain compatibility between PyQt6 & PySide6.

UI files

Another major difference between the two libraries is in their handling of loading .ui files exported from Qt Creator/Designer. PyQt6 provides the uic submodule which can be used to load UI files directly, to produce an object. This feels pretty Pythonic (if you ignore the camelCase).

python
import sys
from PyQt6 import QtWidgets, uic

app = QtWidgets.QApplication(sys.argv)

window = uic.loadUi("mainwindow.ui")
window.show()
app.exec()

The equivalent with PySide6 is one line longer, since you need to create a QUILoader object first. Unfortunately the API of these two interfaces is different too (.load vs .loadUI).

python
import sys
from PySide6 import QtCore, QtGui, QtWidgets
from PySide6.QtUiTools import QUiLoader

loader = QUiLoader()

app = QtWidgets.QApplication(sys.argv)
window = loader.load("mainwindow.ui", None)
window.show()
app.exec_()

To load a UI onto an existing object in PyQt6, for example in your QMainWindow.__init__ you can call uic.loadUI passing in self(the existing widget) as the second parameter.

python
import sys
from PyQt6 import QtCore, QtGui, QtWidgets
from PyQt6 import uic


class MainWindow(QtWidgets.QMainWindow):

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        uic.loadUi("mainwindow.ui", self)


app = QtWidgets.QApplication(sys.argv)
window = MainWindow()
window.show()
app.exec_()

The PySide6 loader does not support this — the second parameter to .load is the parent widget of the widget you're creating. This prevents you adding custom code to the __init__ block of the widget, but you can work around this with a separate function.

python
import sys
from PySide6 import QtWidgets
from PySide6.QtUiTools import QUiLoader

loader = QUiLoader()

def mainwindow_setup(w):
    w.setWindowTitle("MainWindow Title")

app = QtWidgets.QApplication(sys.argv)

window = loader.load("mainwindow.ui", None)
mainwindow_setup(window)
window.show()
app.exec()

Converting UI files to Python

Both libraries provide identical scripts to generate Python importable modules from Qt Designer .ui files. For PyQt6 the script is named pyuic5

bash
pyuic6 mainwindow.ui -o MainWindow.py

You can then import the UI_MainWindow object, subclass using multiple inheritance from the base class you're using (e.g. QMainWIndow) and then call self.setupUI(self) to set the UI up.

python
import sys
from PyQt6 import QtWidgets
from MainWindow import Ui_MainWindow

class MainWindow(QtWidgets.QMainWindow, Ui_MainWindow):

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.setupUi(self)


app = QtWidgets.QApplication(sys.argv)
window = MainWindow()
window.show()
app.exec()

For PySide6 it is named pyside6-uic

bash
pyside6-uic mainwindow.ui -o MainWindow.py

The subsequent setup is identical.

python
import sys
from PySide6 import QtWidgets
from MainWindow import Ui_MainWindow

class MainWindow(QtWidgets.QMainWindow, Ui_MainWindow):

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.setupUi(self)


app = QtWidgets.QApplication(sys.argv)
window = MainWindow()
window.show()
app.exec_()

exec() or exec_()

The .exec() method is used in Qt to start the event loop of your QApplication or dialog boxes. In Python 2.7 exec was a keyword, meaning it could not be used for variable, function or method names. The solution used in both PyQt4 and PySide was to rename uses of .exec to .exec_() to avoid this conflict.

Python 3 removed the exec keyword, freeing the name up to be used. As a result from PyQt6 .exec() calls are named just as in Qt. However, PySide6 still uses .exec_().

Slots and Signals

Defining custom slots and signals uses slightly different syntax between the two libraries. PySide6 provides this interface under the names Signal and Slot while PyQt6 provides these as pyqtSignal and pyqtSlot respectively. The behavior of them both is identical for defining and slots and signals.

The following PyQt6 and PySide6 examples are identical —

python
my_custom_signal = pyqtSignal()  # PyQt6
my_custom_signal = Signal()  # PySide6

my_other_signal = pyqtSignal(int)  # PyQt6
my_other_signal = Signal(int)  # PySide6

Or for a slot —

python
@pyqtslot
def my_custom_slot():
    pass

@Slot
def my_custom_slot():
    pass

If you want to ensure consistency across PyQt6 and PySide6 you can use the following import pattern for PyQt6 to use the Signal and @Slot style there too.

python
from PyQt6.QtCore import pyqtSignal as Signal, pyqtSlot as Slot

You could of course do the reverse from PySide6.QtCore import Signal as pyqtSignal, Slot as pyqtSlot although that's a bit confusing.

QMouseEvent

In PyQt6 QMouseEvent objects no longer have the .pos(), .x() or .y() shorthand property methods for accessing the position of the event. You must use the .position() property to get a QPoint object and access the .x() or .y() methods on that. The .position() method is also available in PySide6.

Features in PySide6 but not in PyQt6

As of Qt 6 PySide supports two Python __feature__ flags to help make code more Pythonic with snake_case variable names and the ability to assign and access properties directly, rather than using getter/setter functions. The example below shows the impact of these changes on code --

python
table = QTableWidget()
table.setColumnCount(2)

button = QPushButton("Add")
button.setEnabled(False)

layout = QVBoxLayout()
layout.addWidget(table)
layout.addWidget(button)

The same code, but with snake_case and true_property enabled.

python
from __feature__ import snake_case, true_property

table = QTableWidget()
table.column_count = 2

button = QPushButton("Add")
button.enabled = False

layout = QVBoxLayout()
layout.add_widget(table)
layout.add_widget(button)

These feature flags are a nice improvement for code readability, however as they are not supported in PyQt6 it makes writign portable code more difficult.

Supporting both in libraries

You don't need to worry about this if you're writing a standalone app, just use whichever API you prefer.

If you're writing a library, widget or other tool you want to be compatible with both PyQt6 and PySide6 you can do so easily by adding both sets of imports.

python
import sys

if 'PyQt6' in sys.modules:
    # PyQt6
    from PyQt6 import QtGui, QtWidgets, QtCore
    from PyQt6.QtCore import pyqtSignal as Signal, pyqtSlot as Slot

else:
    # PySide6
    from PySide6 import QtGui, QtWidgets, QtCore
    from PySide6.QtCore import Signal, Slot

This is the approach used in our custom widgets library, where we support for PyQt6 and PySide6 with a single library import. The only caveat is that you must ensure PyQt6 is imported before (as in on the line above or earlier) when importing this library, to ensure it is in sys.modules.

To account for the lack of shorthand enum and flags in PyQt6 you can generate these yourself. For example, the following code will copy references for each of the enum objects elements up to their parent object, making them accessible as in PyQt5, PySide2 & PySide6. The code would only need to be run under PyQt6.

python
enums = [
    (QtCore.Qt, 'Alignment'),
    (QtCore.Qt, 'ApplicationAttribute'),
    (QtCore.Qt, 'CheckState'),
    (QtCore.Qt, 'CursorShape'),
    (QtWidgets.QSizePolicy, 'Policy'),
]

# Look up using the long name (e.g. QtCore.Qt.CheckState.Checked, used
# in PyQt6) and store under the short name (e.g. QtCore.Checked, used
# in PyQt5, PySide2 & accepted by PySide6).
for module, enum_name in enums:
    for entry in getattr(module, enum_name):
        setattr(module, entry.name, entry)

Alternatively, you can define a custom function to handle the namespace lookup.

python
def _enum(obj, name):
    parent, child = name.split('.')
    result = getattr(obj, child, False)
    if result:  # Found using short name only.
        return result

    obj = getattr(obj, parent)  # Get parent, then child.
    return getattr(obj, child)

When passed an object and a PyQt6 compatible long-form name, this function will return the correct enum or flag on both PyQt6 and PySide6.

python
>>> _enum(PySide6.QtCore.Qt, 'Alignment.AlignLeft')
PySide6.QtCore.Qt.AlignmentFlag.AlignLeft
>>> _enum(PyQt6.QtCore.Qt, 'Alignment.AlignLeft')
<Alignment.AlignLeft: 1>

The final complication is the mismatch in the exec_() and exec() method calls. You can work around this by implementing a function to check the presence of each method and call whichever exists.

python
def _exec(obj):
    if hasattr(obj, 'exec'):
        return obj.exec()
    else:
        return obj.exec_()

You can then use this function to exec objects like QApplication and QDialog in a portable way on both PyQt6 and PySide6.

python
app = QApplication(sys.argv)
_exec(app)

If you're doing this in multiple files it can get a bit cumbersome. A nice solution to this is to move the import logic and custom shim methods to their own file, e.g. named qt.py in your project root. This module imports the Qt modules (QtCore, QtGui, QtWidgets, etc.) from one of the two libraries, and then you import into your application from there.

The contents of the qt.py are the same as we used earlier —

python
import sys

if 'PyQt6' in sys.modules:
    # PyQt6
    from PyQt6 import QtGui, QtWidgets, QtCore
    from PyQt6.QtCore import pyqtSignal as Signal, pyqtSlot as Slot


else:
    # PySide6
    from PySide6 import QtGui, QtWidgets, QtCore
    from PySide6.QtCore import Signal, Slot


def _enum(obj, name):
    parent, child = name.split('.')
    result = getattr(obj, child, False)
    if result:  # Found using short name only.
        return result

    obj = getattr(obj, parent)  # Get parent, then child.
    return getattr(obj, child)


def _exec(obj):
    if hasattr(obj, 'exec'):
        return obj.exec()
    else:
        return obj.exec_()

You must remember to add any other PyQt6 modules you use (browser, multimedia, etc.) in both branches of the if block. You can then import Qt6 into your own application as follows —

python
from .qt import QtGui, QtWidgets, QtCore, _enum, _exec

...and it will work seamlessly across either library.

That's really it

There's not much more to say — the two libraries really are that similar. However, if you do stumble across any other PyQt6/PySide6 examples or features which you can't easily convert, drop me a note.

See the complete PyQt5 tutorial, from first steps to complete applications with Python & Qt5.

PyCharm: PyCharm 2021.1 EAP 4: WSL 2 Support and Faster Indexing

$
0
0

This EAP build brings several nice additions to WSL 2 support, makes indexing faster, and introduces a GitHub pull request template.

As usual, please read the blog post, try out the new features, and let us know if you encounter any problems.

pc-eap-2021-1

We encourage you to join the program to try out the new and improved features. By testing these updates and giving us feedback, you can help us make PyCharm better for you. As always, you can download the new EAP from our website, get it from the free Toolbox App, or update using snap if you’re an Ubuntu user.

Important! PyCharm EAP builds are not fully tested and might be unstable.

DOWNLOAD PYCHARM 2021.1 EAP

WSL 2 support

The best news is that from now on, you can work in PyCharm with your project stored on the WSL filesystem, without copying the project to your Windows file system.

pc-wsl-open-folder

Also, PyCharm now detects the WSL interpreter. If no WSL interpreter is configured for your project, PyCharm will look for system interpreters and set them as the default interpreter automatically.

What’s more, you can open any directory in \\wsl$ with PyCharm. If PyCharm detects any Python file in the directory, it will suggest creating a WSL-based interpreter. See dedicated PyCharm Help page.

Once the project is open, PyCharm Terminal will already be configured to run on WSL. As for version control, you can use Git that is installed on your WSL instance.

As we mentioned in the previous EAP blog post, PyCharm now allows you to use custom Linux distributions run on WSL. PyCharm auto-detects such a distribution and adds it to the Linux Distribution list. You don’t need to add extra pre-configuring – starting with 2021.1, PyCharm will do everything for you.

Although debugging is fully supported for WSL 2, users still need to do a few manual steps to configure the debugger to work with WSL 2. Thanks to the feedback from the PyCharm community (PY-44230), we’ve created a set of simple instructions on how to enable debugging for WSL configurations.

Faster indexing

Our new prebuilt indexes for the most popular Python interpreters have made the standard library indexing much faster. The new approach to prebuilt indexes also helped us make the PyCharm installation file smaller.

VCS

Support for a Git commit template

PyCharm now has a Git commit template, which you can use to create custom commit messages depending on your needs. If you define a commit template, the IDE will display the text from it as an initial commit message.

UX and UI updates in HTTP Client

When you run a request from the HTTP client, the IDE opens the Services tool window. It is now possible to collapse the returned HTML, JSON, or XML by clicking on the minus icon next to it.

The right vertical pane has been given a new look. Response view settings are now grouped under the ‘eye’ icon. When you click on it, you can choose to show line numbers and the format for viewing the response. What’s more, it is easy to scroll to the top and to the bottom of the response by clicking the respective icons.

The last new icon, ‘Copy Response Body To Clipboard’, allows you to copy the response body without all the additional info that the full response contains.

pc-http-ui

You can find the rest of the changes for this EAP build in the release notes.

Ready to join the EAP?

Some ground rules

  • EAP builds are free to use and expire 30 days after the build date.
  • You can install an EAP build side by side with your stable PyCharm version.
  • These builds are not fully tested and can be unstable.
  • Your feedback is always welcome. Please use our issue tracker and make sure to mention your build version

How to download

Download this EAP from our website. Alternatively, you can use the JetBrains Toolbox App to stay up to date throughout the entire EAP. If you’re on Ubuntu 16.04 or later, you can use snap to get PyCharm EAP and stay up to date.

The PyCharm team

Mirek Długosz: Automatic offline Twitter card images (og:image) generator for static sites

$
0
0

I felt a sudden desire to have visually appealing images in social media posts with links to my website, and after reviewing existing solutions, I ended up creating my own. In this post, I skim over solutions I have found and point out what mine does different.

You know how sometimes when you share a link on social media, it is displayed with big, eye-catching image? Something that may look a little like this:

A quick overview of existing solutions

Possibly the most common way of generating these images is creating them manually. If you are adept in tools like Adobe Photoshop, you can obtain great results in no time. If you aren’t, there are websites that can help - banners.beyondco.de, kapwing.com and motif.imgix.com are some that I have found.

There are two main problems with manual approach. First, it requires a lot of time to generate images for all of the content on your website. Second, it requires you to be extra careful if you want to maintain visual consistency of all the images.

Image-generating API services were created almost as if in a response to these limitations. They require you to register an account, but later you can just send carefully-crafted API request and they will send you generated image back in response. This allows for unattended generation of large number of images. Two popular websites providing services like that are cloudinary.com and og-image.vercel.app.

My main problem with these services is somewhat philosophical. I don’t fancy the idea of having my website build / deployment process dependent on some third-party, online API. For me, one of the benefits of static site generators is that they can be used in air-gapped environment. This is important component of ensuring that website is fully reproducible.

Possibly coming from similar position, some members of JavaScript community found another approach. They use their existing infrastructure to generate simple pages that look exactly how card image should look like. Then they open this page in browser automation tool, like Selenium, grab the snapshot of visible area and save it as an image. That solution was described by Leigh Halliday and Florian Kapfenberger.

I can see the benefits of this approach, as CSS is great for reproducibly creating complex visual layouts, browser has all the text layout options you may ever need, and if you are creating the static site, chances are that you are already familiar with HTML and CSS.

But I can’t shake off the feeling that this is completely overengineered. I shouldn’t need a browser to create image. Then, performance of this approach is not great - on my machine, opening up page in Selenium and saving screenshot on disk takes well over a second. Surely there must be a faster way of doing that.

Enter pelican-social-cards

I don’t think my requirements are that special. I want a tool that could create social media images in fully automated fashion, that can run unattended (after initial configuration), that works locally (offline) and that is not terribly slow. But, apparently, nobody created something that would check all these boxes.

So I wrote it myself. Let me introduce pelican-social-cards, a new plugin for Pelican.

Initial configuration is rather straightforward - all you really have to do is specifying template image through SOCIAL_CARDS_TEMPLATE variable. You probably also want to include social-cards directory in your STATIC_PATHS variable, and adjust one of many configuration settings. Plugin comes with extensive documentation that describes all supported options.

It creates images that look something like this:

If you like what you see, head on to PyPI and grab your copy while it’s hot!

Moving forward

One obvious drawback of my solution is tight coupling to Pelican. It will not work with any other site generator, which seriously limits its applicability. Extracting image generating code from Pelican binding code should be relatively straightforward, so hopefully this plugin may still be useful for others. I am open to discussion about creating some shared library and set of plugins for popular static site generators, if there are people out there willing to work on that.

At the same time, I feel that using Python for image-generating tool will ultimately limit the audience and number of contexts in which it will be used. Ideally, such tool could be distributed as statically-linked binary, so anyone could just drop it on their computer and use it directly. Plugins should be simple wrappers around this binary, which should enable creation of plugin for any static site generator out there. I think there is real opportunity for someone who is looking for ideas for side project in Go, Rust, or maybe C++.


John Ludhi/nbshare.io: Introduction To Python Tkinter GUI Programming

$
0
0

Introduction To Python Tkinter GUI Programming

Commonly there are 2 types of interfaces for desktops -  command-line interface (CLI) where users interact with the system by writing commands, and a Graphical User Interface (GUI) where users interact with the system through visual widgets such as buttons, menus, windows etc. Python provides a lot of libraries to create cross-platform GUIs such as Tkinter, Pymsgbox, Pyqt, and much more. In this series of articles, we will study the Tkinter package.

In this article, we will learn

● Basics of Tkinter

● Introduction to Tkinter Widgets

● Tkinter Hello World App

● Tkinter Button

● Tkinter Entry Widget

Basics Of Tkinter

Tkinter is an Inbuilt library commonly used for GUI development. The underlying implementation of Tkinter is in C language.

The basic structure of the Tkinter program is like this...

  1. Import tkinter
  2. Create root i.e. main window of App
  3. Attach widgets such as Buttons, Menu, Inputs etc.
  4. Create Infinite event loop

Introduction to Tkinter Widgets 

Widget is an entity such as GUI button, sidebar, menu that provides user interaction with operating system. 

Label Widget

Let’s create our first Tkinter app.

Tkinter hello world App

import tkinter as tk

root= tk.Tk()
# root is main window of GUI, it is the parent node.
root.geometry("400x300")
'''
syntax:
window.geometry("widthxheight")
'''
root.title("First GUI application")
'''
syntax:
window.title("Title you want")
'''
label = tk.Label(root, text="Hello World", font=("Times New Roman",23))
label.pack(side='bottom') # Packing is must.
root.mainloop()# the infinite event loop. This will run in infinite loop.

You must Pack the widget you created otherwise it will not be visible on the screen. 

Here is the official documentation about pack() method...

Output:

Next we will look in to... 

  1. How to create buttons
  2. How to take input from user

Tikinter Button

Tkinter Create Button

Check out the below Syntax ...

btn = tk.Button(root,text="Click Me",command=Funtion_which_will_be_triggerd)

Code

import tkinter as tk
def fun():
"""
This function will be triggered when we click button
"""
tk.Label(root,text="clicked").pack()
# this is another method for packing a widget ...
# Don't create instance of Label but the disadvantage is that we can't reuse it
if __name__ == "__main__":

# Creating main window
root= tk.Tk()
root.geometry("450x300")
root.title("Creating Buttons")
btn = tk.Button(root,text="Click Me",command=fun)
"""
syntax:
object= tk.Button(parent window, text="This will appear on button", command= pass reference of fun)
"""
btn.pack()
# pack button
root.mainloop()


Tkinter Entry Widget

Entry widget is used to capture the input from user. 

Basic Syntax:

Code

import tkinter as tk
def fun():
global hold_entry
tk.Label(root,text=str(hold_entry.get()),font=("Times New Roman",20,'italic')).pack(side='bottom')
hold_entry.set("") # This would clear the input value.
if __name__ == "__main__":
root = tk.Tk()
root.title("Taking Input's")
root.geometry("650x400")
# our app will take one input
hold_entry = tk.StringVar() # to hold input value of entry
etry= tk.Entry(root,textvariable=hold_entry,font=("Source code Pro",12,'bold'),show="*")
btn=tk.Button(root,text="Submit",command=fun)
etry.pack()
tk.Label(root,text="\n").pack() # for additional space
btn.pack()
root.mainloop()

We can disable the input option by passing DISABLED value to argument `state` of the entry widget

Demo code

import tkinter as tk
def fun():
pass #function to do nothing
if __name__ == "__main__":
root = tk.Tk()
root.title("Disabled Input's")
root.geometry("650x400")
hold_entry= tk.StringVar()
# create variable to hold value to be shown
hold_entry.set("Disbled Text");
etry= tk.Entry(root,textvariable=hold_entry,font=("Source code Pro",12,'bold'),state='disabled')
etry.pack()
tk.Button(root,text="submit",command=fun).pack()
root.mainloop()

Wrap Up

In this post, we have covered the basics of Tkinter. Stay tuned for more posts on GUI programming using Python Tkinter! 

Python Pool: 6 Ways to Use Numpy flatten() Method in Python

$
0
0

Introduction

In python, there are many ways to re-structure the array according to the need of the person. But, there are some cases when we need a one-dimensional array rather than two- dimensional array. This type of problem numpy library provides a function by which we can convert a two-dimensional array into a one-dimensional array, i.e., numpy.ndarray.flatten().

flatten numpy array

What is Numpy Flatten() method?

Numpy.ndarray.flatten() is used when we need to return the copy of the array in a 1-d array rather than a 2-d or multi-dimensional array.

Syntax

ndarray.flatten(order='C')  

Parameter

order: {‘C,’‘F,’ ‘A,’ ‘K’} – It is optional input in the function. Firstly, in this parameter, if order = ‘C,’ the array will get flattened in row-major order. Secondly, If order = ‘F’ – It means that the array will get flattened in column-major order.

Thirdly, If order = ‘A’ – It means to flatten in column-major order if Fortran is contiguous in memory. Otherwise, it will be row-major order. At last, If order = ‘K’ – It means that the array will get flattened in the same as it is there in the memory. By default, always the parameter is set to ‘C.’

Return value

y: ndarray– This function returns a copy of the input array, which gets flattened into one-dimensional.

Examples

Let us understand the numpy flatten() function of the numpy module in details with the help of examples:

1. using a 2-d array

In this example, we will be importing the numpy module as np. Then, we will take the input array as a 2-d array. Finally, we will apply the numpy flatten() function to convert a 2-d array into a 1-d array.

#python program using a 2-d array
#Import numpy library 

import numpy as np 
 
arr = np.array([[1,4,7], [2,5,8]]) 
 
output=arr.flatten()  

print("Input Array : ",arr)
print("Output Array : ",output)

Output:

Input Array :  [[1 4 7]
 [2 5 8]]
Output Array :  [1 4 7 2 5 8]

Explanation:

Here firstly, we w=have imported the numpy module with the alias name as np. Secondly, we have created a 2- d array using the array function. Thirdly, we have created the variable as output and assigned it the flatten() function. At last, we have printed the value of the output array.

2. using multi-dimensional array

In this example, we will be importing the numpy module as np. Then, we will take the input array as a multi-dimensional array. Finally, we will apply the numpy flatten() function to convert a multi-dimensional array into a 1-d array.

#python program using a multi-dimensional array
#Import numpy library 

import numpy as np 
 
arr = np.array([[1,4,7], [2,5,8], [12, 58, 36]]) 
 
output=arr.flatten()  

print("Input Array : ",arr)
print("Output Array : ",output)

Output:

Input Array :  [[ 1  4  7]
 [ 2  5  8]
 [12 58 36]]
Output Array :  [ 1  4  7  2  5  8 12 58 36]

Explanation:

Here firstly, we w=have imported the numpy module with the alias name as np. Secondly, we have created a multi-dimensional array using an array function. Thirdly, we have created the variable as output and assigned it the numpy flatten() function. At last, we have printed the value of the output array.

3. using order = ‘F’ as a parameter

In this example, we will be importing the numpy module as np. Then, we will take the input array as a 2-d array. Finally, we will apply the numpy flatten() function with the order = ‘F’ to convert a 2-d array into a 1-d array.

#python program using a 2-d array
#Import numpy library
#using order = 'F' 

import numpy as np 
 
arr = np.array([[1,4,7], [2,5,8]]) 
 
output=arr.flatten('F')  

print("Input Array : ",arr)
print("Output Array : ",output)

Output:

Input Array :  [[1 4 7]
 [2 5 8]]
Output Array :  [1 2 4 5 7 8]

Explanation:

Here firstly, we w=have imported the numpy module with the alias name as np. Secondly, we have created a 2- d array using the array function. Thirdly, we have created the variable as output and assigned it the flatten() function. Fourthly, we have used the order ‘F’ to print the output as column-major. At last, we have printed the value of the output array.

4. using order = ‘C’ as a parameter

In this example, we will be importing the numpy module as np. Then, we will take the input array as a 2-d array. Finally, we will apply the numpy flatten() function with the order = ‘C’ to convert a 2-d array into a 1-d array.

#python program using a 2-d array
#Import numpy library
#using order ='C' 

import numpy as np 
 
arr = np.array([[1,4,7], [2,5,8]]) 
 
output=arr.flatten('C')  

print("Input Array : ",arr)
print("Output Array : ",output)

Output:

Input Array :  [[1 4 7]
 [2 5 8]]
Output Array :  [1 4 7 2 5 8]

Explanation:

Here firstly, we w=have imported the numpy module with the alias name as np. Secondly, we have created a 2- d array using the array function. Thirdly, we have created the variable as output and assigned it the value of flatten() function. Fourthly, we have used the order ‘C’ to print the output as row-major. At last, we have printed the value of the output array.

5. using order = ‘A’ as a parameter

In this example, we will be importing the numpy module as np. Then, we will take the input array as a 2-d array. Finally, we will apply the numpy flatten() function with the order = ‘A’ to convert a 2-d array into a 1-d array.

#python program using a 2-d array
#Import numpy library
#using order ='A' 

import numpy as np 
 
arr = np.array([[1,4,7], [2,5,8]]) 
 
output=arr.flatten('A')  

print("Input Array : ",arr)
print("Output Array : ",output)

Output:

Input Array :  [[1 4 7]
 [2 5 8]]
Output Array :  [1 4 7 2 5 8]

Explanation:

Here firstly, we w=have imported the numpy module with the alias name as np. Secondly, we have created a 2- d array using the array function. Thirdly, we have created the variable as output and assigned it the flatten() function. Fourthly, we have used the order ‘A’ to print the output as row-major. At last, we have printed the value of the output array.

6. using order = ‘k’ as a parameter

In this example, we will be importing the numpy module as np. Then, we will take the input array as a 2-d array. Finally, we will apply the numpy flatten() function with the order = ‘K’ to convert a 2-d array into a 1-d array.

#python program using a 2-d array
#Import numpy library
#using order ='K' 

import numpy as np 
 
arr = np.array([[21,43,57], [12,85,78]]) 
 
output=arr.flatten('K')  

print("Input Array : ",arr)
print("Output Array : ",output)

Output:

Input Array :  [[21 43 57]
 [12 85 78]]
Output Array :  [21 43 57 12 85 78]

Explanation:

Here firstly, we w=have imported the numpy module with the alias name as np. Secondly, we have created a 2- d array using the array function. Thirdly, we have created the variable as output and assigned it the flatten() function. Fourthly, we have used the order ‘K’ to print the output as row-major. At last, we have printed the value of the output array.

Difference between flatten() and ravel()

These both the function are used to convert a multi-dimensional array into a one-dimensional array. But, there are some differences through which they differ in themselves.

Flatten()

  • It always returns a copy of the original array.
  • It is a function of ndarray object.
  • It is slower than ravel() as it takes more memory than ravel().
  • If the value is modified, the original array will not get affected.

Ravel()

  • It always returns the reference of the original array.
  • It is a library-level function.
  • It is faster than flatten().
  • If the value is modified, the original array will also get affected.
#flatten and ravel function in python example

import numpy as np

arr = np.array([[1,2,3], [4,5,6]])
print("Input Array : ",arr)
print("Dimension of array : ",(arr.ndim))
print("\n")

arr1 = arr.ravel()
print("Ravel output : ",arr1)
arr1[0] = 50
print("updated ravel array : ",arr1)
print("Input array : ",arr)
print("\n")

arr2 = arr.flatten()
print("Flatten output : ",arr2)
arr2[0] = 100
print("Updated flatten array : ",arr2)
print("Input array : ",arr)

Output:

Input Array :  [[1 2 3]
 [4 5 6]]
Dimension of array :  2


Ravel output :  [1 2 3 4 5 6]
updated ravel array :  [50  2  3  4  5  6]
Input array :  [[50  2  3]
 [ 4  5  6]]


Flatten output :  [50  2  3  4  5  6]
Updated flatten array :  [100   2   3   4   5   6]
Input array :  [[50  2  3]
 [ 4  5  6]]

Explanation:

Here firstly, we have imported the numpy module as np. Secondly, we have taken the input array and printed the input array and the array’s dimension. Thirdly, we have applied the ravel() function and printed the ravel() function’s output. We have then updated the value of arr2[0] = 50 and again printed the ravel output and printed the input array. So we can easily see the changes. The same done with the flatten() function there also updated the value, but the input array doesn’t change.

Conclusion

In this tutorial, we have discussed the flatten() function of the numpy module. All the parameters and their values are explained with the help of examples in detail. Examples will help you to understand the concept more accurately. You can use the order as per your requirement of the output one-dimensional array.

The post 6 Ways to Use Numpy flatten() Method in Python appeared first on Python Pool.

Python Pool: 6 Ways to Calculate Percentile of Numpy Array

$
0
0

What is Numpy Percentile?

The percentile method in the numpy module is used to calculate the nth percentile of the given data (array elements) along the specified axis. We basically use percentile in statistics which gives you a number that describes the value that a given percent of the values are lower than.

Syntax

numpy.percentile(a, q, axis=None, out=None, overwrite_input=False, keepdims=False)

Parameters

  1. a: array_like– This is the Input array.
  2. q: array_like of float– This is the percentile or sequence of percentile we need to compute. It should be between 0 to 100, both inclusive.
  3. axis : {int, tuple of int, None} – It is optional input. This is the axis along which we calculate the percentile. By default, we compute the percentile along with the flattened version of the array.
  4. out: ndarray – It is also an optional input. It is the alternative output array which we make to place the result. It should have the same shape and buffer length as the expected output, but the type can be cast by itself if it is required.
  5. overwrite_input: bool – It is also optional input. If the boolean value is True, we can modify the input array through intermediate calculations to save the memory.
  6. keepdims: bool– It is also optional input. If the value is set to be True, the reduced axes are left in the result as dimensions with one size. With this, the result will be correctly against the original array.

Return Value

If q is a single percentile and the axis is set to None, then the output is always a scalar or array with percentile values along the specified axis. 

Examples to Find Numpy Percentile

Let us understand the percentile function of the numpy module in details with the help of examples:

1. Numpy Percentile using 1-d Array

We will be using a 1-D array for calculating the percentile of the array by taking the input array.

#using 1-D array
#numpy.percentile() method

import numpy as np

arr = [5,6,9,87,2,3,5,7,2,6,5,2,3,4,69,4]
print("Array : ",arr)

x = np.percentile(arr, 50)
print("50 percentile : ",x)

Output:

Array :  [5, 6, 9, 87, 2, 3, 5, 7, 2, 6, 5, 2, 3, 4, 69, 4]
50 percentile :  5.0

Explanation:

Here firstly, we have imported the numpy module in python as np. Secondly, we have taken a 1-d array. Thirdly, we have printed the input array. Fourthly, we have used the percentile method as np.percentile() in which we have given arr and 50 percentile as the parameter and stored that value in the x variable. At last, we have printed the value of x. Hence, the output is printed on the screen.

2. using 2-D array

We will be using a 2-D array for calculating the percentile of the array by taking the input array.

#using 2-D array
#numpy.percentile() method

import numpy as np

arr = [[5,6,8],[6,9,2]]
print("Array : ",arr)

x = np.percentile(arr, 50)
print("50 percentile : ",x)

Output:

Array :  [[5, 6, 8], [6, 9, 2]]
50 percentile :  6.0

Explanation:

Here firstly, we have imported the numpy module in python as np. Secondly, we have taken a 2-d array. Thirdly, we have printed the input array. Fourthly, we have used the percentile method as np.percentile() in which we have given arr and 50 percentile as the parameter and stored that value in the x variable. At last, we have printed the value of x. Hence, the output is printed on the screen.

3. Numpy Percentile using axis = 0 in 2-D array

We will be using axis = 0 in a 2-D array for calculating the percentile of the array by taking the input array.

#using 2-D array axis = 0
#numpy.percentile() method

import numpy as np

arr = [[5,6,8],[6,9,2]]
print("Array : ",arr)

x = np.percentile(arr, 25, axis = 0)
print("50 percentile : ",x)

Output:

Array :  [[5, 6, 8], [6, 9, 2]]
50 percentile :  [5.25 6.75 3.5 ]

Explanation:

Here firstly, we have imported the numpy module in python as np. Secondly, we have taken a 2-d array. Thirdly, we have printed the input array. Fourthly, we have used the percentile method as np.percentile() in which we have given arr, 25 percentile, and axis = 0 as the parameter and stored that value in the x variable. At last, we have printed the value of x. Hence, the output is printed on the screen.

4. using axis = 1 in 2-d array

We will be using axis = 1 in a 2-D array for calculating the percentile of the array by taking the input array.

#using 2-D array axis = 1
#numpy.percentile() method

import numpy as np

arr = [[5,6,8],[6,9,2]]
print("Array : ",arr)

x = np.percentile(arr, 25, axis = 1)
print("50 percentile : ",x)

Output:

Array :  [[5, 6, 3], [6, 7, 2]]
50 percentile :  [4. 4. ]

Explanation:

Here firstly, we have imported the numpy module in python as np. Secondly, we have taken a 2-d array. Thirdly, we have printed the input array. Fourthly, we have used the percentile method as np.percentile() in which we have given arr, 25 percentile, and axis = 1 as the parameter and stored that value in the x variable. At last, we have printed the value of x. Hence, the output is printed on the screen.

5. Numpy Percentile using axis=1 and keepdims = true in a 2-D array

We will be using axis = 1 and Keepdims = True in a 2-D array for calculating the percentile of the array by taking the input array.

#using 2-D array axis = 1 and Keepdims = True
#numpy.percentile() method

import numpy as np

arr = [[10,9,4],[3,2,1]]
print("Array : ",arr)

x = np.percentile(arr, 50, axis = 1, keepdims = True)
print("50 percentile : ",x)

Output:

Array :  [[10, 9, 4], [3, 2, 1]]
50 percentile :  [[9.]
 [2.]]

Explanation:

Here firstly, we have imported the numpy module in python as np. Secondly, we have taken a 2-d array. Thirdly, we have printed the input array. Fourthly, we have used the percentile method as np.percentile() in which we have given arr, 50 percentile, axis = 1, and keepdims= True as the parameter. The value has stored that value in the x variable. At last, we have printed the value of x. Hence, the output is printed on the screen.

6. using Out parameter in 2-d array

We will be using axis = 0 and out in a 2-D array for calculating the percentile of the array by taking the input array.

import numpy as np

arr = [[10,9,4],[3,2,1]]
print("Array : ",arr)
m = np.percentile(arr, 50, axis=0)
out = np.zeros_like(m)
x = np.percentile(arr, 50, axis=0, out=out)
print("50 percentile : ",x)

Output:

Array :  [[10, 9, 4], [3, 2, 1]]
50 percentile :  [6.5 5.5 2.5]

Explanation:

Here firstly, we have imported the numpy module in python as np. Secondly, we have taken a 2-d array. Thirdly, we have printed the input array. Fourthly, we have used the percentile method as np.percentile() in which we have given arr, 50 percentile, axis = 0, and out(output array of the same shape and buffer length) as the parameter. The value has stored that value in the x variable. At last, we have printed the value of x. Hence, the output is printed on the screen.

Must Read

Numpy Percentile vs Quantile

Percentile– Percentile method in the numpy module through which we can calculate the nth percentile of the given data (array elements) along the specified axis.

Numpy Quantile– Quantile method in the numpy module through which we can calculate the qth quantile of the given data(array elements) along the specified axis.

Let us understand with the help of example:

#numpy percentile vs numpy quantile

import numpy as np

arr = [10,20,30,40,50]

print("Array : ",arr)
print("\n")

print("25 percentile : ",np.percentile(arr, 25))
print("50 percentile : ",np.percentile(arr, 50))
print("75 percentile : ",np.percentile(arr, 75))
print("\n")

print(".25 Quantile : ",np.Quantile(arr, .25))
print(".50 Quantile : ",np.Quantile(arr, .50))
print(".75 Quantile : ",np.Quantile(arr, .75))

Output:

Array :  [10, 20, 30, 40, 50]


25 percentile :  20.0
50 percentile :  30.0
75 percentile :  40.0


.25 Quantile :  20.0
.50 Quantile :  30.0
.75 Quantile :  40.0

Explanation:

Here firstly, we have imported the numpy module as np. Secondly, we have taken an input array in the arr variable. Thirdly, we have applied the percentile function in which we have calculated the 25th, 50th, and 75th percentile and printed the output. Fourthly, we have applied the quantile function in which we have calculated the .25th, .50th, and .75th percentile and printed the output. Hence, we can see the output and get to know that there is only a difference which says 1 percentile = .01 quantile.

Conclusion

In this tutorial, we have learned about the percentile calculation with the help of the percentile method in the numpy standard library. All the parameters are explained with examples in detail. You can use any parameter according to your need in your program or project.

The post 6 Ways to Calculate Percentile of Numpy Array appeared first on Python Pool.

Python Pool: 8 Examples to Use Python min() Function in the Best Way

$
0
0

Introduction

Python has several built-in functions. These all functions are the global functions of python, which can be called from python code. We do not have to import any library for it. In this tutorial, we will be studying the min() function in python. There are a lot of functions in python like max(), abs(), next(), pow(), etc.

What is min() in python?

The min() function in python is used to return the minimum values passed in its argument and lexicographically smallest value if the strings are passed as arguments.

Syntax of Python min()

min(iterables, key,default)

Parameter

  • Iterables– It is the type of objects like list, tuple, dictionary, etc.
  • Key– It is optional input in the min() function. This is the parameter where the iterables are passed, and the comparison is performed.
  • Default– It is also the optional input in the min() function. In this, we pass the default value if the given iterable is empty.

Return value of Python min() Function

It returns the minimum value from the iterables passed.

Exceptions

It also returns exceptions sometimes when we used to compare the conflicting types of iterables.

Examples

Let us understand the min() function of python in details with the help of examples:

1. using Python min() function in string

In this example, we will be taking the input as a string and finding the string’s minimum value.

#using input as a string

str = "LatrAcaLsSolutions"
Minimum = min(str)

print("Minimum value in the string: ", Minimum)

Output:

Minimum value in the string :  A

Explanation:

Here firstly, we will take the input as a string. Secondly, we will apply the min() function and store the output in the Minimum variable. At last, we will print the output. Hence, we will see the minimum value in the string.

2. using Python min() function in list of integers

In this example, we will be taking the input as a list containing integer values in it. Then, we will try to find the minimum value of the integer in the list.

#using a list of integers

lst = [5, 4, -5, 0, 4]
minimum = min(lst)
print("Minimum in the list is : ",minimum)

Output:

Minimum in the list is :  -5

Explanation:

Here firstly, we will take the input as a list of integers. Secondly, we will apply the min() and store the output in the minimum variable. At last, we will print the output. Hence, we will see the minimum value in the list of integers.

3. using min() function in list of strings

In this example, we will be taking the input as a list containing string values in it. Then, we will try to find the minimum value of the string in the list.

#using a list of string

lst = ["Latracal", "Solutions", "India"]
minimum = min(lst)
print("Minimum in the list is : ",minimum)

Output:

Minimum in the list is :  India

Explanation:

Here firstly, we will take the input as a list of strings. Secondly, we will apply the min() and store the output in the Minimum variable. At last, we will print the output. Hence, we will see the minimum value in the list of strings.

4. using Python min() function in tuple

In this example, we will be taking the input as a tuple. Then, we will try to find the minimum value present in the tuple.

#using tuple as an input

arr = (1, 5, 3, 9)
MIN = min(arr)

print("minimum value : ",MIN)

Output:

minimum value :  1

Explanation:

Here firstly, we will take the input as a tuple. Secondly, we will apply the min() and store the output in the Minimum variable. At last, we will print the output. Hence, we will see the minimum value in the tuple.

5. using min() function in dictionary

In this example, we will be taking the input as a dictionary. Then, we will try to find the minimum value present in the dictionary.

#using a dictionary as input

dict = {1: 4, 2: 9, 3: 1, 4: 4}
MIN = min(dict)

print("Minimum value : ",MIN)

Output:

Minimum value :  1

Explanation:

Here firstly, we will take the input as a dictionary. Secondly, we will apply the min() function and store the output in the MIN variable. At last, we will print the output. Hence, we will see the minimum value in the dictionary.

6. using Python min() function with a default value

In this example, we will be taking the input as an empty list. Then, we will try to find the minimum empty list with the default value.

#using default value

lst = []
x= min(lst, default =20)

print("Minimum value : ",x)

Output:

Minimum value :  20

Explanation:

Here firstly, we will take the input as an empty list as lst= []. Secondly, we will apply the min() function with the default value = 20 and store the x variable’s output. At last, we will print the output. Hence, we will see the minimum value in the empty string with the default value = 20.

7. using min() function with arguments and key function

In this example, we will be using min() with arguments and key function. we will define the function for calculating the string’s length.

#using the key function

def str_length(s):
    return len(s)
lst = ['LAtra', 'abc', 'Solu']
x = min(lst)

print(x, key=str_length))

Output:

abc

Explanation:

Here firstly, we will take the input as a list. Secondly, we will apply the min() function and store the output in the x variable. Thirdly, we will define the def function with str_length(s) as the name of it. Fourthly, we will try to print the min value with the key value = str_length. At last, we will print the output. Hence, we will see the minimum value in the string.

8. using Python min() function with multiple iterables

In this example, we will be using the multiple iterables as the input. Then we will print the minimum value of the iterables.

#using multiple iterables

lst1 = [1, 2, 3]
lst2 = [5, 15, 40, 25, 35, 45]
Minimum = min(lst1, lst2, key = len)
print("Minimum value : ", Minimum)

Output:

Minimum value :  [1, 2, 3]

Explanation:

Here firstly, we will take the input as multiple iterables with two lists as lst1 and lst2. Secondly, we will apply the min() function with both the list and the key-value = len. Thirdly, we will store the output in the Minimum variable. At last, we will print the output. Hence, we will see the minimum value in the tuple.

Conclusion

In this tutorial, we have learned how to calculate the minimum value using the min() python function. All the parameters and types are explained in detail with the help of examples. You can use any data such as list, tuple, dictionary, etc., as required. You can also use different types of parameters as required in the program.

The post 8 Examples to Use Python min() Function in the Best Way appeared first on Python Pool.

Python Pool: Numpy outer() Method Explained In-Depth With 5 Examples

$
0
0

Introduction to Numpy outer

Numpy outer() is the function in the numpy module in the python language. It is used to compute the outer level of products like vectors, arrays, etc. If we try to combine the two vectors of the array’s outer level, the numpy outer() function requires more than two levels of arguments that are passed into the function. It will be the array-like format, i.e., single or multi-parameter arguments. We can store the results in the out parameter. If we need to do the scientific calculations in the numpy library, those are calculated using dot operators.

What is Numpy outer() ?

The Numpy outer() function is used to compute the outer products of two vectors.

Syntax

numpy.outer(a, b, out = None)

Parameters of Numpy Outer

  1. a: array_like– It is the first input vector. The input is flattened if it is not 1- dimensional.
  2. b: array_like– It is the second input vector. The input is flattened if it is not 1- dimensional.
  3. Out: ndarray – It is optional in the function. It is the location where we store the result.

Return value

It returns the result as the outer products of two vectors. suppose i and j are the number of values in the array a and b. then, out[i, j] = a[i] * b[j].

Examples to Use Numpy outer() Function in the Best Way

Let us understand the outer function of the numpy module in details with the help of examples:

1. using linspace function to calculate numpy outer product

A linspace() function is used to return number spaces evenly concerning the interval. In this. we will import the numpy library as np. Then, we will take the input as np.ones() and np.linspace(). At last, we will apply the numpy outer() function for the result.

#Numpy outer() program
#using linspace()

import numpy as np

x = np.ones(6)
y = np.linspace(2, 6, 3)

Output = np.outer(x, y)

print("Result : ",Output)

Output:

Result :  [[2. 4. 6.]
 [2. 4. 6.]
 [2. 4. 6.]
 [2. 4. 6.]
 [2. 4. 6.]
 [2. 4. 6.]]

Explanation:

Here firstly, we have imported the numpy module as np. Secondly, we have used np.ones(6) which means the result will get printed 6 times. Thirdly, we have used the linspace function, which returns the even space numbers evenly w,r,t the interval. np.linspace(2, 6, 3) means that from range 2 to 6, it will be divided into 3 parts, i.e., 2, 4, and 6. At last, we have applied the numpy outer() function and printed the value of output. Hence, we can see that the output is printed on the screen.

2. using 1-d matrix To Calculate Numpy Outer Product

In this example, we will be importing the numpy module as np. Then, we will be taken the input in a 1-d matrix. Finally, we will apply the numpy outer() function to produce the outer products of two vectors or arrays.

#using a 1-d matrix
#importing numpy as np

import numpy as np

arr1 = np.array([2, 6])
arr2 = np.array([5, 8])

print("Array1 : ",arr1)
print("Array2 : ",arr2)

output = np.outer(arr1, arr2)
print("Result : ",output)

Output:

Array1 :  [2 6]
Array2 :  [5 8]
Result :  [[10 16]
 [30 48]]

Explanation:

Here firstly, we have imported the numpy module as np. Secondly, we have taken the input of two 1-d Array as np.array() in arr1 and arr2. Thirdly, we have printed the value of both the arrays as array1 and array2. At last, we have applied the numpy outer() function and printed the value of output. The output is printed as arr1[0] is multiplied with both the values of arr2, and arr1[1] is also multiplied with both values of arr2. Hence, we can see that the output is printed on the screen.

3. using 2-d matrix To Calculate Numpy Outer Product

In this example, we will be importing the numpy module as np. Then, we will be taken the input in a 2-d matrix. Finally, we will apply the numpy outer() function to produce the outer products of two vectors or arrays.

#using 2-d matrix
#importing numpy as np

import numpy as np

arr1 = np.array([[2, 5], [3, 5]])
arr2 = np.array([[1, 2], [2, 6]])

print("Array1 : ",arr1)
print("Array2 : ",arr2)

output = np.outer(arr1, arr2)
print("Result : ",output)

Output:

Array1 :  [[2 5]
 [3 5]]
Array2 :  [[1 2]
 [2 6]]
Result :  [[ 2  4  4 12]
 [ 5 10 10 30]
 [ 3  6  6 18]
 [ 5 10 10 30]]

Explanation:

Here firstly, we have imported the numpy module as np. Secondly, we have taken the input of two 2-d Array as np.array() in arr1 and arr2. Thirdly, we have printed the value of both the arrays as array1 and array2. At last, we have applied the numpy outer() function and printed the value of output. The out is printed as all the values of arr1 are multiplied with all the values of arr2. Hence, we can see that the output is printed on the screen.

4. using 3-d matrix To Calculate Numpy Outer Product

In this example, we will be importing the numpy module as np. Then, we will be taken the input in a 3-d matrix. At last, we will apply the numpy outer() function to produce the outer products of two vectors or arrays.

#using 3-d matrix
#importing numpy as np

import numpy as np

arr1 = np.array([[2, 1,3], [0, 2, 5], [0, 1, 2]])
arr2 = np.array([[1, 2, 3], [2, 5, 6], [2, 3, 4]])

print("Array1 : ",arr1)
print("Array2 : ",arr2)

output = np.outer(arr1, arr2)
print("Result : ",output)

Output:

Array1 :  [[2 1 3]
 [0 2 5]
 [0 1 2]]
Array2 :  [[1 2 3]
 [2 5 6]
 [2 3 4]]
Result :  [[ 2  4  6  4 10 12  4  6  8]
 [ 1  2  3  2  5  6  2  3  4]
 [ 3  6  9  6 15 18  6  9 12]
 [ 0  0  0  0  0  0  0  0  0]
 [ 2  4  6  4 10 12  4  6  8]
 [ 5 10 15 10 25 30 10 15 20]
 [ 0  0  0  0  0  0  0  0  0]
 [ 1  2  3  2  5  6  2  3  4]
 [ 2  4  6  4 10 12  4  6  8]]

Explanation:

Here firstly, we have imported the numpy module as np. Secondly, we have taken the input of two 3-d Array as np.array() in arr1 and arr2. Thirdly, we have printed the value of both the arrays as array1 and array2. At last, we have applied the numpy outer() function and printed the value of output. The out is printed as all the values of arr1 are multiplied with all the values of arr2. Hence, we can see that the output is printed on the screen.

5. using out parameter To Calculate Numpy Outer Product

In this example, we will be importing the numpy module as np. Then, we will be taken the input in a 1-d matrix. Then, we will be specifying the out parameter as a result of the same shape as the input. Finally, we will apply the numpy outer() function to produce the outer products of two vectors or arrays.

#using out parameter
#importing numpy as np

import numpy as np

arr1 = np.array([2, 6])
arr2 = np.array([5, 8])
result = np.zeros((2,2))

print("Array1 : ",arr1)
print("Array2 : ",arr2)

output = np.outer(arr1, arr2, out = result)
print("Result : ",output)

Output:

Array1 :  [2 6]
Array2 :  [5 8]
Result :  [[10. 16.]
 [30. 48.]]

Explanation:

Here firstly, we have imported the numpy module as np. Secondly, we have taken the input of two 1-d Array as np.array() in arr1 and arr2. Thirdly, we have printed the value of both the arrays as array1 and array2. Finally, we have applied the numpy outer() function without as the parameter and printed the value of output in the result variable. The output is printed as arr1[0] is multiplied with both the values of arr2, and arr1[1] is also multiplied with both values of arr2. Hence, we can see that the output is printed on the screen.

Numpy outer() vs Numpy Inner()

Numpy.outer(): The Numpy outer() function is used to compute two vectors’ outer products.

Numpy.inner(): The Numpy outer() function is used to compute two arrays’ inner products.

Let us understand this with the help of example:

taking 1-d array

import numpy as np

arr1 = np.array([1,2,3])
arr2 = np.array([4,5,6])
print("output : ",np.outer(arr1,arr2))
print("\n")
print("output : ",np.inner(arr1,arr2))

Output:

output :  [[ 4  5  6]
 [ 8 10 12]
 [12 15 18]]


output :  32

Explanation:

Here firstly, we have imported the numpy module as np. Secondly, Then we have to take two input arrays as arr1 and arr2. Thirdly, we have applied outer() and inner() and printed the output. So, in the outer(), we can see that all the arr1 and arr2 multiply themselves and formed a matrix, and an inner() we can see that each value multiplied with its same index and added itself then gives the output.

Numpy outer() vs Numpy dot()

Numpy.outer() : The Numpy outer() function is used to compute two vectors’ outer products.

Numpy.dot() : The Numpy dot() function is used to compute the dot product of two arrays.

Let us understand this with the help of example:

#outer() vs dot()

import numpy as np

arr1 = np.array([[1,2], [4,5]])
arr2 = np.array([[7,8], [10,11]])

print("Outer() : ",np.outer(arr1,arr2))
print("\n")
print("dot() : ",np.dot(arr1,arr2))

Output:

Outer() :  [[ 7  8 10 11]
 [14 16 20 22]
 [28 32 40 44]
 [35 40 50 55]]


dot() :  [[27 30]
 [78 87]]

NOTE:

REMEMBER: for dot product the array should always be square

dot function is calculated as 
dot() = [[1*7 + 2*10 , 1*8 + 2*11], [4*7 + 5*10, 4*8 + 5*11]

Explanation:

Here firstly, we have imported numpy as np. Secondly, we have taken two arrays as arr1 and arr2 as input. Thirdly, We have applied the outer() and dot() and printed the output. In the note, I have explained to you how the dot product is calculated. Hence, we can see the output and the difference between the outer() and dot().

Conclusion

In this tutorial, we have discussed the outer function of the numpy module. All the parameters are explained in detail. With the help of examples, you can understand the concept in detail. You can use the out parameter according to the need of the project and program.

The post Numpy outer() Method Explained In-Depth With 5 Examples appeared first on Python Pool.

Viewing all 23329 articles
Browse latest View live


<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>