Python Bytes: #260 It's brutally simple: made just from pickle and zip

<p><strong>Watch the live stream:</strong></p> <a href='https://www.youtube.com/watch?v=F7hmqiTXADQ' style='font-weight: bold;'>Watch on YouTube</a><br> <br> <p><strong>About the show</strong></p> <p>Sponsored by <strong>Shortcut - Get started at</strong> <a href="http://shortcut.com/pythonbytes"><strong>shortcut.com/pythonbytes</strong></a></p> <p>Special guest: Chris Patti</p> <p><strong>Brian #1:</strong> <a href="https://til.simonwillison.net/python/cog-to-update-help-in-readme"><strong>Using cog to update --help in a Markdown README file</strong></a></p> <ul> <li>Simon Willison</li> <li>I’ve wanted to have a use case for Ned Batchelder’s <a href="https://nedbatchelder.com/code/cog">cog</a> <ul> <li>Cog is a utility that looks for specially blocks</li> </ul></li> </ul> <pre><code> [[[cog some code ]]] </code></pre> <p>and </p> <pre><code> [[[end]]] </code></pre> <ul> <li>These block can be in comments, <code>[HTML_REMOVED]</code> for markdown.</li> <li>When you run cog on a file, it runs the “some code” and puts the output after the middle <code>]]]</code> and before the <code>[[[end]]]</code>.</li> <li>Simon has come up with an excellent use, running <code>--help</code> and capturing the output for a <code>README.md</code> file for a CLI project.</li> <li>He even wrote a test, pytest of course, to check if the README.md needs updated.</li> </ul> <p><strong>Michael #2:</strong> <a href="https://calpaterson.com/bank-python.html"><strong>An oral history of Bank Python</strong></a></p> <ul> <li>Bank Python implementations are effectively proprietary forks of the <em>entire</em> Python ecosystem which are in use at many (but not all) of the biggest investment banks.</li> <li>The first thing to know about Minerva is that it is built on a global database of Python objects. <ul> <li>Barbara is a simple key value store with a hierarchical key space. It's brutally simple: made just from <a href="https://docs.python.org/3/library/pickle.html">pickle</a> and <a href="https://docs.python.org/3/library/zlib.html">zip</a>.</li> </ul></li> <li>Applications also commonly store their <em>internal</em> state in Barbara - writing dataclasses straight in and out with only very simple locking and transactions (if any). </li> <li>There is no filesystem available to Minerva scripts and the little bits of data that scripts pick up has to be put into Barbara.</li> <li>Barbara also has some "overlay" features:</li> </ul> <pre><code> # connect to multiple rings: keys are 'overlaid' in order of # the provided ring names db = barbara.open("middleoffice;ficc;default") # get /Etc/Something from the 'middleoffice' ring if it exists there, # otherwise try 'ficc' and finally the default ring some_obj = db["/Etc/Something"] </code></pre> <ul> <li>Lots of info about modeling with classes (instruments, books, etc)</li> <li>If you understand excel you will be starting to recognize similarities. </li> <li>In Excel, spreadsheets cells are also updated based on their dependencies, also as a directed acyclic graph. Dagger allows people to put their Excel-style modelling calculations into Python, write tests for them, control their versioning without having to mess around with files like CDS-OF-CDS EURO DESK 20180103 Final (final) (2).xlsx. </li> <li>Dagger is a key technology to get financial models out of Excel, into a programming language and under tests and version control.</li> <li>Time to drop a bit of a bombshell: the source code is in Barbara too, not on disk. Remain composed. It's kept in a special Barbara ring called sourcecode.</li> <li>Interesting table structures, like Pandas, but closer to a DB (MnTable)</li> <li>Over time the divergence between Bank Python and Open Source Python grows. Technology churns on both sides, much faster outside than in of course, but they do not get closer.</li> <li>Minerva has its own IDE - no other IDEs work if you keep your source files in a giant global database. </li> <li>What I can't understand is why it contains its own web framework. Investment banks have a one-way approach to open source software: (some of) it can come in, but none of it can go out</li> <li>BTW, I “read” this with <a href="https://www.naturalreaders.com/"><strong>naturalreaders app</strong></a></li> </ul> <p>C<strong>hris #3:</strong> <a href="https://github.com/kitao/pyxel"><strong>Pyxel</strong></a></p> <ul> <li>Pyxel is a ‘retro gaming console’ written in Python!</li> <li>This might seem old and un-shiny, but the restrictions imposed by the environment gift <strong>simplicity</strong> <ul> <li>Vastly decreased learning time and effort compared to something like Unity or even Pygame</li> <li>Straight forward simple commands, just like it was for micro-computers in the 80s <ul> <li>cls(), line(), rect(), circ() etc.</li> </ul></li> </ul></li> <li>Pyxel is somewhat more Python and less console than others like PICO-8 or TIC-80 but this is a feature! Use your regular development environment to build.</li> </ul> <p><strong>Brian #4:</strong> <a href="https://hynek.me/til/ditch-codecov-python"><strong>How to Ditch Codecov for Python Projects</strong></a></p> <ul> <li>Hynek Schlawack</li> <li>Codecov is a third party service that checks your coverage output and fails a build if coverage dropped. </li> <li>It’s not without issues.</li> <li>Hynek is using <a href="https://coverage.readthedocs.io/">coverage.py</a> <code>--fail-under</code> flag in place of this in GitHub actions.</li> <li>It’s not as simple as just adding a flag if you are using <code>--parallel</code> to combine coverage for multiple test runs into one report.</li> <li>Hynek is utilizing the coverage output as an artifact for each test, then pulling them all together in a <code>coverage</code> stage combine and check coverage.</li> <li>He provides the snippet of GH Action, and even links to a <a href="https://github.com/hynek/structlog/blob/main/.github/workflows/main.yml">working workflow file</a> using this process. Nice!</li> </ul> <p><strong>Michael #5:</strong> <a href="https://github.com/nschloe/tiptop"><strong>tiptop (like glances)</strong></a></p> <ul> <li>via Zach Villers</li> <li>tiptop is a command-line system monitoring tool in the spirit of <a href="https://en.wikipedia.org/wiki/Top_(software)">top</a>. It displays various interesting system stats, graphs it, and works on all operating systems.</li> <li>Really nice visualization for your servers</li> <li>Good candidate for <strong>pipx</strong> install tiptop</li> </ul> <p><strong>Chris #6:</strong> <a href="https://github.com/irmen/pyc64"><strong>pyc64</strong></a></p> <ul> <li>A Commodore 64 emulator written in pure Python!</li> <li>Not 100% complete - screen drawing is PETSCII character mode only <ul> <li>This still allows for a lot of interesting apps &amp; exploration</li> </ul></li> <li>Actual machine emulation using <a href="https://github.com/mnaberez/py65">py65</a> - a pure Python 6502 chip emulator!</li> <li>You can pop to a Python REPL from inside the emulator and examine data structures like memory, registers, etc!</li> <li>An incredible example of what Python is capable of</li> <li>0.6 Mhz with CPython and over 2Mhz with pypy!</li> </ul> <p><strong>Extras</strong></p> <p>Michael:</p> <ul> <li>Michael’s <a href="https://twitter.com/FlaskCon/status/1461245426082799618"><strong>FlaskCon 2021 HTMX Talk</strong></a></li> </ul> <p><strong>Chris</strong>: </p> <ul> <li><a href="https://amazon.jobs/en/internal/search?offset=0&amp;result_limit=10&amp;sort=relevant&amp;distanceType=Mi&amp;radius=1024km&amp;hiring_manager[]=Sean%20Stewart%20(zjamste)&amp;latitude=&amp;longitude=&amp;loc_group_id=&amp;loc_query=&amp;base_query=&amp;city=&amp;country=&amp;region=&amp;county=&amp;query_options=&amp;">Amazon OpsTech IT is hiring!</a> (If deemed appropriate :) <a href="https://amazon.jobs/en/internal/search?offset=0&amp;result_limit=10&amp;sort=relevant&amp;distanceType=Mi&amp;radius=1024km&amp;hiring_manager[]=Sean%20Stewart%20(zjamste)&amp;latitude=&amp;longitude=&amp;loc_group_id=&amp;loc_query=&amp;base_query=&amp;city=&amp;country=&amp;region=&amp;county=&amp;query_options=&amp;"></a></li> </ul> <p><strong>Joke:</strong> <a href="https://www.newyorker.com/cartoons/daily-cartoon/wednesday-november-17th-daylight-saving-screens"><strong>I hate how the screens get bright so early this time of year</strong></a></p>