<p><strong>Watch the live stream:</strong></p>
<a href='https://www.youtube.com/watch?v=_lAgN2Klnvs' style='font-weight: bold;'>Watch on YouTube</a><br>
<br>
<p><strong>About the show</strong></p>
<p>Sponsored by: <a href="https://pythonbytes.fm/foundershub"><strong>Microsoft for Startups Founders Hub</strong></a>.</p>
<p>Special guest: Thomas Gaigher, creator/maintainer <a href="https://github.com/pypyr/pypyr">pypyr taskrunner</a></p>
<p><strong>Michael #1:</strong> <a href="https://twitter.com/_ChrisMay/status/1506761070064680971"><strong>March Package Madness</strong></a></p>
<ul>
<li>via Chris May</li>
<li>Start with 16 packages</li>
<li>They battle it out 2-on-2 in elimination rounds</li>
<li>Voting is once a week</li>
<li>So go vote!</li>
</ul>
<p><strong>Brian #2:</strong> <a href="https://nbpreview.readthedocs.io/en/latest/"><strong>nbpreview</strong></a></p>
<ul>
<li>“A terminal viewer for Jupyter notebooks. It’s like cat for ipynb files.”</li>
<li>Some cool features
<ul>
<li>pretty colors by default
<ul>
<li>piping strips formatting, so you can pass it to grep or other post processing</li>
<li>automatic paging</li>
</ul></li>
<li>syntax highlighting
<ul>
<li>line numbers and wrapping work nicely</li>
</ul></li>
<li>markdown rendering</li>
<li>images converted to block, character, or dots (braille)</li>
<li>dataframe rendering</li>
<li>clickable links</li>
</ul></li>
</ul>
<p>Thomas <strong>#3:</strong> <strong>pyfakefs</strong></p>
<ul>
<li>A fake file system!</li>
<li>It intercepts all calls that involve the filesystem in Python - e.g <code>open()</code>, <code>shutil</code>, or <code>pathlib.Path</code>.</li>
<li>This is completely transparent - your functional code does not know or need to know that under the hood it's been disconnected from the actual filesystem.</li>
<li>The nice thing about this is that you don't have to go patching <code>open</code> using <code>mock_open</code> - which works fine, but gets annoying quickly for more complex test scenarios.
<ul>
<li>E.g Doing a mkdir -p before a file write to ensure parent dirs exist.</li>
</ul></li>
<li>What it looks like without a fake filesystem:</li>
</ul>
<pre><code>in_bytes = b"""[table]
foo = "bar" # String
"""
# read
with patch('pypyr.toml.open',
mock_open(read_data=in_bytes)) as mocked_open:
payload = toml.read_file('arb/path.in')
# write
with io.BytesIO() as out_bytes:
with patch('pypyr.toml.open', mock_open()) as mock_output:
mock_output.return_value.write.side_effect = out_bytes.write
toml.write_file('arb/out.toml', payload)
out_str = out_bytes.getvalue().decode()
mock_output.assert_called_once_with('arb/out.toml', 'wb')
assert out_str == """[table]
foo = "bar""""</code></pre>
<ul>
<li>If you've ever tried to patch/mock out <code>pathlib</code>, you'll know the pain!</li>
<li>Also, no more annoying test clean-up routines or <code>tempfile</code> - as soon as the fake filesystem goes out of scope, it's gone, no clean-up required.</li>
<li>Not a flash in the pan - long history: originally developed by Mike Bland at Google back in 2006. Open sourced in 2011 on Google Code. Moved to Github and nowadays maintained by <a href="https://github.com/jmcgeheeiv">John McGehee</a>.</li>
<li>This has been especially useful for <a href="https://pypyr.io">pypyr</a>, because as a task-runner or automation tool pypyr deals with wrangling config files on disk a LOT (reading, generating, editing, token replacing, globs, different encodings), so this makes testing so much easier.
<ul>
<li>Especially to keep on hitting the 100% test coverage bar!</li>
</ul></li>
<li>Works great with <a href="https://docs.pytest.org/">pytest</a> with the provided <code>fs</code> fixture.
<ul>
<li>Just add the <code>fs</code> fixture to a test, and all code under test will use the fake filesystem.</li>
</ul></li>
<li>Dynamically switch between Linux, MacOs & Windows filesystems.</li>
<li>Set up paths/files in your fake filesystem as part of test setup with some neat helper functions.</li>
<li>Very responsive maintainers - I had a PR merged in less than half a day. Shoutout to <a href="https://github.com/mrbean-bremen">mrbean-bremen</a>.</li>
<li>Docs here: <a href="http://jmcgeheeiv.github.io/pyfakefs/release/">http://jmcgeheeiv.github.io/pyfakefs/release/</a></li>
<li>Github here: <a href="https://github.com/jmcgeheeiv/pyfakefs">https://github.com/jmcgeheeiv/pyfakefs</a></li>
<li>Real world example:</li>
</ul>
<pre><code>@patch('pypyr.config.config.default_encoding', new='utf-16')
def test_json_pass_with_encoding(fs):
"""Relative path to json should succeed with encoding."""
# arrange
in_path = './tests/testfiles/test.json'
fs.create_file(in_path, contents="""{
"key1": "value1",
"key2": "value2",
"key3": "value3"
}
""", encoding='utf-16')
# act
context = pypyr.parser.jsonfile.get_parsed_context([in_path])
# assert
assert context == {
"key1": "value1",
"key2": "value2",
"key3": "value3"
}
def test_json_parse_not_mapping_at_root(fs):
"""Not mapping at root level raises."""
# arrange
in_path = './tests/testfiles/singleliteral.json'
fs.create_file(in_path, contents='123')
# act
with pytest.raises(TypeError) as err_info:
pypyr.parser.jsonfile.get_parsed_context([in_path])
# assert
assert str(err_info.value) == (
"json input should describe an object at the top ""level. You should have something like\n""{\n\"key1\":\"value1\",\n\"key2\":\"value2\"\n}\n""at the json top-level, not an [array] or literal.")
</code></pre>
<p><strong>Michael #4:</strong> <a href="https://github.com/irgeek/StrEnum">strenum</a></p>
<ul>
<li>A Python Enum that inherits from str.</li>
<li>To complement enum.IntEnum in the standard library. Supports python 3.6+.</li>
<li>Example usage:</li>
</ul>
<pre><code> class HttpMethod(StrEnum):
GET = auto()
POST = auto()
PUT = auto()
DELETE = auto()
assert HttpMethod.GET == "GET"</code></pre>
<p>Use wherever you can use strings, basically:</p>
<pre><code> ## You can use StrEnum values just like strings:
import urllib.request
req = urllib.request.Request('https://www.python.org/', method=HttpMethod.HEAD)
with urllib.request.urlopen(req) as response:
html = response.read()
</code></pre>
<p>Can auto-translate casing with <code>LowercaseStrEnum</code> and <code>UppercaseStrEnum</code>.</p>
<p><strong>Brian #5:</strong> <a href="https://tdhopper.com/blog/code-review-guidelines">Code Review Guidelines for Data Science Teams</a></p>
<ul>
<li>Tim Hopper</li>
<li>Great guidelines for any team</li>
<li>What is code review for?
<ul>
<li>correctness, familiarity, design feedback, mutual learning, regression protection</li>
<li>NOT opportunities for
<ul>
<li>reviewer to impose their idiosyncrasies</li>
<li>dev to push correctness responsibility to reviewers</li>
<li>demands for perfection</li>
</ul></li>
</ul></li>
<li>Opening a PR
<ul>
<li>informative commit messages</li>
<li>consider change in context of project</li>
<li>keep them short</li>
<li>write a description that helps reviewer</li>
<li>include tests with new code</li>
</ul></li>
<li>Reviewing
<ul>
<li>Wait for CI before starting
<ul>
<li>I would also add “wait at least 10 min or so, requester might be adding comments”</li>
</ul></li>
<li>Stay positive, constructive, helpful</li>
<li>Clarify when a comment is minor or not essential for merging, preface with “nit:” for example</li>
<li>If a PR is too large, ask for it to be broken into smaller ones</li>
<li>What to look for
<ul>
<li>does it look like it works</li>
<li>is new code in the right place</li>
<li>unnecessary complexity</li>
<li>tests</li>
</ul></li>
</ul></li>
</ul>
<p>Thomas <strong>#6:</strong> <strong>Shell Power is so over. Leave the turtles in the late 80ies.</strong></p>
<ul>
<li>Partly inspired by/continuation of last week’s episode’s mention of running subprocesses from Python.</li>
<li>Article by <a href="https://twitter.com/itamarst"><strong>Itamar Turner-Trauring</strong></a>
<ul>
<li>Please Stop Writing Shell Scripts <a href="https://pythonspeed.com/articles/shell-scripts/">https://pythonspeed.com/articles/shell-scripts/</a></li>
</ul></li>
<li>Aims mostly at bash, but I'll happily include bourne, zsh etc. under the same dictum</li>
<li>If nothing else, solid listing of common pitfalls/gotchas with bash and their remedies, which is educational enough in and of itself already.
<ul>
<li>TLDR; Error handling in shell is hard, but also surprising if you're not particularly steeped in the ways of the shell.</li>
<li>Error resumes next, unset vars don't raise errors, piping & sub shells errs thrown away</li>
</ul></li>
<li>If you really-eally HAVE to shell, you prob want this boilerplate on top (aka <a href="http://redsymbol.net/articles/unofficial-bash-strict-mode/">unofficial bash strict mode</a>:</li>
</ul>
<pre><code>#!/bin/bash
set -euo pipefail
IFS=$'\n\t'
</code></pre>
<ul>
<li>This will,
<ul>
<li>-e: fail immediately on error</li>
<li>-u: fail on Unset vars</li>
<li>-o pipefail: raise immediately when piping</li>
<li>IFS: set Internal Field Separator to <code>newline | tab</code>, rather than <code>space | newline | tab</code>.
<ul>
<li>Prevents surprises when iterating over strings with spaces in them</li>
</ul></li>
</ul></li>
<li>Itamar lists common counter-arguments from shell script die-hards:
<ul>
<li>It's always there!
<ul>
<li>But so is the runtime of whatever you're actually coding in, and in the case of a build CI server. . .almost by definition.</li>
</ul></li>
<li>Git gud! (I'm paraphrasing)</li>
<li>Shell-check (linting for bash, basically)</li>
</ul></li>
<li>The article is short & sweet - mercifully so in these days of padded content.</li>
<li>The rest is going to be me musing out loud, so don't blame the OG author. So expanding on this, I think there're a couple of things going on here:</li>
<li>If anything, the author is going a bit soft on your average shell script. If you’re just calling a couple of commands in a row, okay, fine. But the moment you start worrying about retrying on failure, parsing some values into or out of some json, conditional branching - which, if you are writing any sort of automation script that interacts with other systems, you WILL be doing - shell scripts are an unproductive malarial nightmare.
<ul>
<li>Much the same point applies to Makefile. It’s an <em>amazing</em> tool, but it’s also misused for things it was never really meant to do. You end up with Makefiles that call shell scripts that call Makefiles. . .</li>
</ul></li>
<li>Given that coding involves automating stuff, amazingly often the actual automation of the development process itself is deprioritized & unbudgeted.</li>
<li>Sort of like the shoemaker's kid not having shoes.
<ul>
<li>Partly because when management has to choose between shiny new features and automation, shiny new features win every time.</li>
<li>Partly because techies will just "quickly" do a thing in shell to solve the immediate problem… Which then becomes part of the firmament like a dead dinosaur that fossilises and more and more inscrutable layers accrete on top of the original "simple" script.</li>
<li>Partly because coders would rather get on with clever but marginal micro-optimisations and arguing over important stuff like spaces vs tabs, rather than do the drudge work of automating the development/deployment workflow.</li>
</ul></li>
<li>There's the glimmering of a point in there somewhere: when you have to choose between shiny new features & more backoffice automation, shiny new features probably win.
<ul>
<li>Your competitiveness in the marketplace might well depend on this.</li>
<li>BUT, we shouldn’t allow the false idea that shell scripts are "quicker" or "lighter touch" to sneak in there alongside the brutal commercial reality of trade-offs on available budget & time.</li>
<li>If you have to automate quickly, it's more sensible to use a task-runner or just your actual programming language. If you're in python already, you're in luck, python's GREAT for this.</li>
</ul></li>
<li>Don’t confuse excellent cli programs like <code>git</code> , <code>curl</code> , <code>awscli</code>, <code>sed</code> or <code>awk</code> with a shell script. These are executables, you don’t need the shell to invoke these.</li>
<li>Aside from these empirical factors, a couple of psychological factors also.
<ul>
<li>Dealing with hairy shell scripts is almost a Technocratic rite of passage - coupled with imposter syndrome, it's easy to be intimidated by the Shell Bros who're steeped in the arcana of bash.</li>
<li>It's the tech equivalent of "back in my day, we didn't even have <<>>", as if this is a justification for things being more difficult than they need to be ever thereafter.</li>
<li>This isn't Elden Ring, the extra difficulty doesn't make it more <em>fun</em>. You're trying to get business critical work done, reliably & quickly, so you can get on with those new shiny features that actually pay the bills.</li>
</ul></li>
</ul>
<p><strong>Extras</strong> </p>
<p>Michael:</p>
<ul>
<li>A changing of the guard
<ul>
<li>Firefox → <a href="https://vivaldi.com">Vivaldi</a>
<ul>
<li>(here’s <a href="https://arstechnica.com/information-technology/2020/08/firefox-maker-mozilla-lays-off-250-workers-says-covid-19-lowered-revenue/">a little more info</a> on the state of Firefox/Mozilla financially) </li>
<li>(<a href="https://arstechnica.com/information-technology/2020/08/firefox-maker-mozilla-lays-off-250-workers-says-covid-19-lowered-revenue/?comments=1&post=39142373">threat team</a> is particularly troubling)</li>
</ul></li>
<li>Google email/drive/etc → <a href="http://Zoho.com">Zoho</a></li>
<li>@gmail.com to @customdomain.com </li>
<li>Google search → <a href="https://duckduckgo.com">DuckDuckGo</a></li>
<li>BTW Calendar apps/integrations and email clients are trouble</li>
</ul></li>
</ul>
<p><strong>Joke:</strong> <a href="https://twitter.com/PR0GRAMMERHUM0R/status/1505106836608933892">A missed opportunity - and cybersecurity</a></p>
↧