Codementor: Subtleties of Python
Red Hat Developers: What, No Python in RHEL 8 Beta?
TL;DR Of course we have Python! You just need to specify if you want Python 3 or 2 as we didn’t want to set a default. Give yum install python3 and/or yum install python2 a try. Or, if you want to see what we recommend you install yum install @python36 or yum install @python27. Read on for why:
For prior versions of Red Hat Enterprise Linux, and most Linux Distributions, users have been locked to the system version of Python unless they got away from the system’s package manager. While this can be true for a lot of tools (ruby, node, PERL, php) the Python use case is more complicated because so many Linux tools (like yum) rely on Python. In order to improve the experience for RHEL 8 users, we have moved the Python used by the system “off to the side” and we introduced the concept of Application Streams based on Modularity.
Through Application Streams, in combination with Python’s ability to be parallel installed, we can now make multiple versions of Python available and easily installable, from the standard repositories, into the standard locations. No extra things to learn or manage. Now, users can choose what version of Python they want to run in any given userspace and it simply works. For more info, see my article, Introducing Application Streams in RHEL 8.
To be honest, the system maintainers also get some advantages of not being locked to an aging version of Python for our system tools. With users not relying on a particular version of Python coming with the system installation, we have the freedom to take advantage of new language features, performance improvements, and all the other goodness a developer gets when tracking near the upstream version.
However, this has resulted in a dilemma. When a user sits down at a fresh installation of RHEL 8 they will naturally expect that /usr/bin/python will run some version of Python. If you follow the recommendation in Python Enhancement Proposal (PEP) 394, that will be Python 2. However, at some point, a new PEP will likely want to change that recommendation to Python 3, probably during, the typically *10* year, life of RHEL 8! To put this in perspective, consider that RHEL 7 was released in 2014, and will be supported until 2024!
So, what do we do? Well, if we follow the current recommendation, we make some present day users happy. However, when the Python Community shifts to recommending Python 3 as the default, we will make new users unhappy.
As a result, we came to the tough conclusion, don’t provide a default, unversioned Python at all. Ideally, people will get used to explicitly typing python3 or python2. However for those that want an unversioned command, let them chose from the beginning which version of Python they actually want. So, yum install python results in a 404.
However, we do try to make it as easy as possible to get Python 2 or 3 (or both) on to your system. We recommend using yum install @python36 or yum install @python27 to take advantage of the recommended set of packages to install. If all you really need is *just* the Python binaries, you can use yum install python3 or yum install python2.
We have also setup the alternatives infrastructure so that when you install either (or both) you can easily make /usr/bin/python point to the right place using alternatives --config python. However, as we explained above, and aligned with the Python PEP, we don’t recommend relying on /usr/bin/python being the correct python for your application.
Note: the same issue arises for Python wrapper scripts like pip. Installing Python 3 will put pip3 in your path, but not unversioned pip. With Python modules like pip, venv, and virtualenv, you can avoid confusion and get the right version by running those as a module: python3 -m pip and avoiding the wrapper scripts. Using Python virtual environments is a best practice that also avoids the issues with version ambiguity, see How to install Python 3 on Red Hat Enterprise Linux 7 for virtual environment details and advice.
To conclude, yes, Python is included in RHEL 8! And, it will be even better than in the past! If you want more details on anything in this post, please see the How To Guide on Red Hat Developers.
Oh and if you haven’t downloaded RHEL 8 yet—go to developers.redhat.com/rhel8 now.
Additional Information
- Red Hat Enterprise Linux 8 Beta for developers
- Introducing Application Streams in RHEL 8.
- Petr Viktorin’s Python on RHEL 8 article (see the discussion of Platform Python)
- Introducing CodeReady Linux Builder
- Containers without daemons: Podman and Buildah available in RHEL 7.6 and RHEL 8 Beta
The post What, No Python in RHEL 8 Beta? appeared first on RHD Blog.
Stack Abuse: Handling Unix Signals in Python
UNIX/Linux systems offer special mechanisms to communicate between each individual process. One of these mechanisms are signals, and belong to the different methods of communication between processes (Inter Process Communication, abbreviated with IPC).
In short, signals are software interrupts that are sent to the program (or the process) to notify the program of significant events or requests to the program in order to run a special code sequence. A program that receives a signal either stops or continues the execution of its instructions, terminates either with or without a memory dump, or even simply ignores the signal.
Although it is defined in the POSIX standard, the reaction actually depends on how the developer wrote the script and implemented the handling of signals.
In this article we explain what are signals, show you how to sent a signal to another process from the command line as well as processing the received signal. Among other modules, the program code is mainly based on the signal module. This module connects the according C headers of your operating system with the Python world.
An Introduction to Signals
On UNIX-based systems, there are three categories of signals:
System signals (hardware and system errors): SIGILL, SIGTRAP, SIGBUS, SIGFPE, SIGKILL, SIGSEGV, SIGXCPU, SIGXFSZ, SIGIO
Device signals: SIGHUP, SIGINT, SIGPIPE, SIGALRM, SIGCHLD, SIGCONT, SIGSTOP, SIGTTIN, SIGTTOU, SIGURG, SIGWINCH, SIGIO
User-defined signals: SIGQUIT, SIGABRT, SIGUSR1, SIGUSR2, SIGTERM
Each signal is represented by an integer value, and the list of signals that are available is comparably long and not consistent between the different UNIX/Linux variants. On a Debian GNU/Linux system, the command kill -l displays the list of signals as follows:
$ kill -l
1) SIGHUP 2) SIGINT 3) SIGQUIT 4) SIGILL 5) SIGTRAP
6) SIGABRT 7) SIGBUS 8) SIGFPE 9) SIGKILL 10) SIGUSR1
11) SIGSEGV 12) SIGUSR2 13) SIGPIPE 14) SIGALRM 15) SIGTERM
16) SIGSTKFLT 17) SIGCHLD 18) SIGCONT 19) SIGSTOP 20) SIGTSTP
21) SIGTTIN 22) SIGTTOU 23) SIGURG 24) SIGXCPU 25) SIGXFSZ
26) SIGVTALRM 27) SIGPROF 28) SIGWINCH 29) SIGIO 30) SIGPWR
31) SIGSYS 34) SIGRTMIN 35) SIGRTMIN+1 36) SIGRTMIN+2 37) SIGRTMIN+3
38) SIGRTMIN+4 39) SIGRTMIN+5 40) SIGRTMIN+6 41) SIGRTMIN+7 42) SIGRTMIN+8
43) SIGRTMIN+9 44) SIGRTMIN+10 45) SIGRTMIN+11 46) SIGRTMIN+12 47) SIGRTMIN+13
48) SIGRTMIN+14 49) SIGRTMIN+15 50) SIGRTMAX-14 51) SIGRTMAX-13 52) SIGRTMAX-12
53) SIGRTMAX-11 54) SIGRTMAX-10 55) SIGRTMAX-9 56) SIGRTMAX-8 57) SIGRTMAX-7
58) SIGRTMAX-6 59) SIGRTMAX-5 60) SIGRTMAX-4 61) SIGRTMAX-3 62) SIGRTMAX-2
63) SIGRTMAX-1 64) SIGRTMAX
The signals 1 to 15 are roughly standardized, and have the following meaning on most of the Linux systems:
- 1 (SIGHUP): terminate a connection, or reload the configuration for daemons
- 2 (SIGINT): interrupt the session from the dialogue station
- 3 (SIGQUIT): terminate the session from the dialogue station
- 4 (SIGILL): illegal instruction was executed
- 5 (SIGTRAP): do a single instruction (trap)
- 6 (SIGABRT): abnormal termination
- 7 (SIGBUS): error on the system bus
- 8 (SIGFPE): floating point error
- 9 (SIGKILL): immmediately terminate the process
- 10 (SIGUSR1): user-defined signal
- 11 (SIGSEGV): segmentation fault due to illegal access of a memory segment
- 12 (SIGUSR2): user-defined signal
- 13 (SIGPIPE): writing into a pipe, and nobody is reading from it
- 14 (SIGALRM): the timer terminated (alarm)
- 15 (SIGTERM): terminate the process in a soft way
In order to send a signal to a process in a Linux terminal you invoke the kill command with both the signal number (or signal name) from the list above and the id of the process (pid). The following example command sends the signal 15 (SIGTERM) to the process that has the pid 12345:
$ kill -15 12345
An equivalent way is to use the signal name instead of its number:
$ kill -SIGTERM 12345
Which way you choose depends on what is more convenient for you. Both ways have the same effect. As a result the process receives the signal SIGTERM, and terminates immediately.
Using the Python signal Library
Since Python 1.4, the signal library is a regular component of every Python release. In order to use the signal library, import the library into your Python program as follows, first:
import signal
Capturing and reacting properly on a received signal is done by a callback function - a so-called signal handler. A rather simple signal handler named receiveSignal() can be written as follows:
def receiveSignal(signalNumber, frame):
print('Received:', signalNumber)
return
This signal handler does nothing else than reporting the number of the received signal. The next step is registering the signals that are caught by the signal handler. For Python programs, all the signals (but 9, SIGKILL) can be caught in your script:
if __name__ == '__main__':
# register the signals to be caught
signal.signal(signal.SIGHUP, receiveSignal)
signal.signal(signal.SIGINT, receiveSignal)
signal.signal(signal.SIGQUIT, receiveSignal)
signal.signal(signal.SIGILL, receiveSignal)
signal.signal(signal.SIGTRAP, receiveSignal)
signal.signal(signal.SIGABRT, receiveSignal)
signal.signal(signal.SIGBUS, receiveSignal)
signal.signal(signal.SIGFPE, receiveSignal)
#signal.signal(signal.SIGKILL, receiveSignal)
signal.signal(signal.SIGUSR1, receiveSignal)
signal.signal(signal.SIGSEGV, receiveSignal)
signal.signal(signal.SIGUSR2, receiveSignal)
signal.signal(signal.SIGPIPE, receiveSignal)
signal.signal(signal.SIGALRM, receiveSignal)
signal.signal(signal.SIGTERM, receiveSignal)
Next, we add the process information for the current process, and detect the process id using the methode getpid() from the os module. In an endless while loop we wait for incoming signals. We implement this using two more Python modules - os and time. We import them at the beginning of our Python script, too:
import os
import time
In the while loop of our main program the print statement outputs "Waiting...". The time.sleep() function call makes the program wait for three seconds.
# output current process id
print('My PID is:', os.getpid())
# wait in an endless loop for signals
while True:
print('Waiting...')
time.sleep(3)
Finally, we have to test our script. Having saved the script as signal-handling.py we can invoke it in a terminal as follows:
$ python3 signal-handling.py
My PID is: 5746
Waiting...
...
In a second terminal window we send a signal to the process. We identify our first process - the Python script - by the process id as printed on screen, above.
$ kill -1 5746
The signal event handler in our Python program receives the signal we have sent to the process. It reacts accordingly, and simply confirms the received signal:
...
Received: 1
...
Ignoring Signals
The signal module defines ways to ignore received signals. In order to do that the signal has to be connected with the predefined function signal.SIG_IGN. The example below demonstrates that, and as a result the Python program cannot be interrupted by CTRL+C anymore. To stop the Python script an alternative way has been implemented in the example script - the signal SIGUSR1 terminates the Python script. Furthermore, instead of an endless loop we use the method signal.pause(). It just waits for a signal to be received.
import signal
import os
import time
def receiveSignal(signalNumber, frame):
print('Received:', signalNumber)
raise SystemExit('Exiting')
return
if __name__ == '__main__':
# register the signal to be caught
signal.signal(signal.SIGUSR1, receiveSignal)
# register the signal to be ignored
signal.signal(signal.SIGINT, signal.SIG_IGN)
# output current process id
print('My PID is:', os.getpid())
signal.pause()
Handling Signals Properly
The signal handler we have used up to now is rather simple, and just reports a received signal. This shows us that the interface of our Python script is working fine. Let's improve it.
Catching the signal is already a good basis but requires some improvement to comply with the rules of the POSIX standard. For a higher accuracy each signal needs a proper reaction (see list above). This means that the signal handler in our Python script needs to be extended by a specific routine per signal. This works best if we understand what a signal does, and what a common reaction is. A process that receives signal 1, 2, 9 or 15 terminates. In any other case it is expected to write a core dump, too.
Up to now we have implemented a single routine that covers all the signals, and handles them in the same way. The next step is to implement an individual routine per signal. The following example code demonstrates this for the signals 1 (SIGHUP) and 15 (SIGTERM).
def readConfiguration(signalNumber, frame):
print ('(SIGHUP) reading configuration')
return
def terminateProcess(signalNumber, frame):
print ('(SIGTERM) terminating the process')
sys.exit()
The two functions above are connected with the signals as follows:
signal.signal(signal.SIGHUP, readConfiguration)
signal.signal(signal.SIGTERM, terminateProcess)
Running the Python script, and sending the signal 1 (SIGHUP) followed by a signal 15 (SIGTERM) by the UNIX commands kill -1 16640 and kill -15 16640 results in the following output:
$ python3 daemon.py
My PID is: 16640
Waiting...
Waiting...
(SIGHUP) reading configuration
Waiting...
Waiting...
(SIGTERM) terminating the process
The script receives the signals, and handles them properly. For clarity, this is the entire script:
import signal
import os
import time
import sys
def readConfiguration(signalNumber, frame):
print ('(SIGHUP) reading configuration')
return
def terminateProcess(signalNumber, frame):
print ('(SIGTERM) terminating the process')
sys.exit()
def receiveSignal(signalNumber, frame):
print('Received:', signalNumber)
return
if __name__ == '__main__':
# register the signals to be caught
signal.signal(signal.SIGHUP, readConfiguration)
signal.signal(signal.SIGINT, receiveSignal)
signal.signal(signal.SIGQUIT, receiveSignal)
signal.signal(signal.SIGILL, receiveSignal)
signal.signal(signal.SIGTRAP, receiveSignal)
signal.signal(signal.SIGABRT, receiveSignal)
signal.signal(signal.SIGBUS, receiveSignal)
signal.signal(signal.SIGFPE, receiveSignal)
#signal.signal(signal.SIGKILL, receiveSignal)
signal.signal(signal.SIGUSR1, receiveSignal)
signal.signal(signal.SIGSEGV, receiveSignal)
signal.signal(signal.SIGUSR2, receiveSignal)
signal.signal(signal.SIGPIPE, receiveSignal)
signal.signal(signal.SIGALRM, receiveSignal)
signal.signal(signal.SIGTERM, terminateProcess)
# output current process id
print('My PID is:', os.getpid())
# wait in an endless loop for signals
while True:
print('Waiting...')
time.sleep(3)
Further Reading
Using the signal module and an according event handler it is relatively easy to catch signals. Knowing the meaning of the different signals, and to react properly as defined in the POSIX standard is the next step. It requires that the event handler distinguishes between the different signals, and has a separate routine for all of them.
Shannon -jj Behrens: PyCon Notes: PostgreSQL Proficiency for Python People
Here's the video. Here are the slides. Here are my notes:
Christophe Pettus was the speaker. He's from PostgreSQL Experts.
PostgreSQL is a rich environment.
It's fully ACID compliant.
It has the richest set of features of any modern, production RDMS. It has even more features than
Oracle.
PostgreSQL focuses on quality, security, and spec compliance.
It's capable of very high performance: tens of thousands of transactions per second, petabyte-sized data sets, etc.
To install it, just use your package management system (apt, yum, etc.). Those systems will usually take care of initialization.
There are many options for OS X. Heroku even built a Postgres.app that runs more like a foreground app.
A "cluster" is a single PostgreSQL server (which can manage multiple databases).
initdb creates the basic file structure. PostgreSQL has to be up and running to run initdb.
To create a database:
sudo su - postgres
psql
To drop a database:
drop database this_new_database;
Debian runs initdb for you. Red Hat does not.
Debian has a cluster management system. Use it. See, for instance, pg_createcluster.
Always create databases as UTF-8. Once you've created it, you can't change it.
Don't use SQLASCII. It's a nightmare. Don't use "C locale".
pg_ctl is a built-in command to start and stop PostgreSQL:
cd POSTGRES_DIRECTORY
pg_ctl -D . start
Usually, pg_ctl is wrapped by something provided by your platform.
On Ubuntu, start PostgreSQL via:
service postgresql start
Always use "-m fast" when stopping.
Postgres puts its own data in a top-level directory. Let's call it $PGDATA.
Don't monkey around with that data.
pg_clog and pg_xlog are important. Don't mess with them.
On most systems, configuration lives in $PGDATA.
postgresql.conf contains server configuration.
pg_hba.conf contains authentication settings.
postgresql.conf can feel very overwhelming.
Avoid making a lot of changes to postgresql.conf. Instead, add the following to it:
include "postgresql.conf.include"
Then, mess with "postgresql.conf.include".
The important parameters fall into these categories: logging, memory, checkpoints, and the planner.
Logging:
Be generous with logging. It has a very low impact on the system. It's your best source of info for diagnosing problems.
You can log to syslog or log CSV to files. He showed his typical logging configuration.
He showed his guidelines / heuristics for all the settings, including how to finetune things. They're really good! See his slides.
As of version 9.3, you don't need to tweak Linux kernel parameters anymore.
Do not mess with fsync or synchronous_commit.
Most settings require a server reload to take effect. Some things require a server restart. Some can be set on a per-session basis. Here's how to do that. This is also an example of how to use a transaction:
begin;
set local random_page_cost = 2.5;
show random_page_cost;
abort;
pg_hba.conf contains users and roles. Roles are like groups. They form a hierarchy.
A user is just a role with login privs.
Don't use the "postgres" superuser for anything application-related.
Sadly, you probably will have to grant schema-modification privs to your app user if you use migrations, but if you don't have to, don't.
By default, DB traffic is not encrypted. Turn on SSL if you are running in a cloud provider.
In pg_hba.conf, "trust" means if they can log into the server, they can access Postgres too. "peer" means they can have a Postgres user that matches their username. "md5" is an md5 hash password.
It's a good idea to restrict the IP addresses allowed to talk to the server fairly tightly.
The WAL
The Write-Ahead Log is key to many Postgres operations. It's the basis for replication, crash recovery, etc.When each transaction is committed, it is logged to the write-ahead log.
Changes in the transaction are flushed to disk.
If the system crashes, the WAL is "replayed" to bring the DB to a consistent state.
It's a continuous record of changes since the last checkpoint.
The WAL is stored in 16MB segments in the pg_xlog directory.
Never delete anything from pg_xlog.
archive_command is a way to move the WAL segments to someplace safe (like a
different system).
By default, synchronous_commit is on, which means that commits do not return until the WAL flush is done. If you turn it off, they'll return when the WAL flush is queued. You might lose transactions in the case of a crash, but there's no risk of database corruption.
Backup and Recovery
Experience has shown that 20% of the time, your EBS volumes will not reattach when you reboot in AWS.pg_dump is a built-in dump/restore tool.
It takes a logical snapshot of the database.
It doesn't lock the database or prevent writes to disk.
pg_restore restores the database. It's not fast.
It's great for simple backups but not suitable for fast recovery from major failures.
pg_bench is the built in benchmarking tool.
pg_dump -Fc --verbose example > example.dump
Without the -Fc, it dumps SQL commands instead of its custom format.
pg_restore --dbname=example_restored --verbose example.dump
pg_restore takes a long time because it has to recreate indexes.
pg_dumpall --globals-only
Back up each database with pg_dump using --format=custom.
To do a parallel restore, use --jobs=.
If you have a large database, pg_dump may not be appropriate.
A disk snapshot + every WAL segment is enough to recreate the database.
To start a PITR (point in time recovery) backup:
select pg_start_backup(...);
Copy the disk image and any WAL files that are created.
select pg_stop_backup();
Make sure you have all the WAL segments.
The disk image + all the WAL segments are enough to create the DB.
See also github.com/wal-e/wal-e. It's highly recommended.
It automates backups to S3.
He explained how to do a PITR.
With PITR, you can rollback to a particular point in time. You don't have to replay everything.
This is super handy for application failures.
RDS is something that scripts all this stuff for you.
Replication
Send the WAL to another server.Keep the server up to date with the primary server.
That's how PostgreSQL replication works.
The old way was called "WAL Archiving". Each 16MB segment was sent to the secondary when complete. Use rsync, WAL-E, etc., not scp.
The new way is Streaming Replication.
The secondary gets changes as they happen.
It's all setup via recovery.conf in your $PGDATA.
He showed a recovery.conf for a secondary machine, and showed how to let it become the master.
Always have a disaster recovery strategy.
pg_basebackup is a utility for doing a snapshot of a running server. It's the easiest way to take a snapshot to start a new secondary. It's also useful for archival backups. It's not the fastest thing, but it's pretty foolproof.
Replication:
The good:
Easy to setup.
Schema changes are replicated.
Secondaries can handle read-only queries for load balancing.
It either works or it complains loudly.
The bad:
You get the entire DB cluster or none of it.
No writes of any kind to the secondary, not even temporary tables.
Some things aren't replicated like temporary tables and unlogged tables.
His advice is to start with WAL-E. The README tells you everything. It fixes a ton of problems.
The biggest problem with WAL-E is that writing to S3 can be slow.
Another way to do funky things is trigger-based replication. There's a bunch of third-party packages to do this.
Bucardo is one that lets you do multi-master setups.
However, they're fiddly and complex to set up. They can also fail quietly.
Transactions, MVCC, and Vacuum
BEGIN;INSERT ...;
INSERT ...;
COMMIT;
By the way, no bank works this way ;)
Everything runs inside of a transaction.
If there is no explicit transaction, each statement is wrapped in one for you.
Everything that modifies the database is transactional, even schema changes.
\d shows you all your tables.
With a transaction, you can even rollback a table drop.
South (the Django migration tool) runs the whole migration in a single transaction.
Many resources are held until the end of a transaction. Keep your transactions brief and to the point.
Beware of "IDLE IN TRANSACTION" sessions. This is a problem for Django apps.
A tuple in Postgres is the same thing as a row.
Postgres uses Multi-Version Concurrency Control. Each transaction sees its own version of the database.
Writers only block writers to the same tuple. Nothing else causes blocking.
Postgres will not allow two snapshots to "fork" the database. If two people try to write to the same tuple, Postgres will block one of them.
There are higher isolation modes. His description of them was really interesting.
He suggested that new apps use SERIALIZABLE. This will help you find the concurrency errors in your app.
Deleted tuples are not usually immediately freed.
Vacuum's primary job is to scavenge tuples that are no longer visible to any transaction.
autovacuum generally handles this problem for you without intervention (since version 8).
Run analyze after a major database change to help the planner out.
If someone tells you "vacuum's not working", they're probably wrong.
The DB generally stabilizes at 20% to 50% bloat. That's acceptable.
The problem might be that there are long-running transactions or idle-in-transaction sessions. They'll block vacuuming. So will manual table locking.
He talked about vacuum issues for rare situations.
Schema Design
Normalization is important, but don't obsess about it.Pick "entities". Make sure that no entity-level info gets pushed into the subsidiary items.
Pick a naming scheme and stick with it.
Plural or singular? DB people tend to like plural. ORMs tend to like singular.
You probably want lower_case to avoid quoting.
Calculated denormalization can sometimes be useful; copied denormalization is almost never useful.
Joins are good.
PostgreSQL executes joins very efficiently. Don't be afraid of them.
Don't worry about large tables joined with small tables.
Use the typing system. It has a rich set of types.
Use domains to create custom types.
A domain is a core type + a constraint.
Don't use polymorphic fields (fields whose interpretation is dependent on another field).
Don't use strings to store multiple types.
Use constraints. They're cheap and fast.
You can create constraints across multiple columns.
Avoid Entity-Attribute-Value schemas. They cause great pain. They're very inefficient. They make reports very difficult.
Consider using UUIDs instead of serials as synthetic keys.
The problem with serials for keys is that merging tables can be hard.
Don't have "Thing" tables like "Object" tables.
If a table has a few frequently-updated fields and a few slowly-updated fields, consider splitting the table. Split the fast-moving stuff out into a separate 1-to-1 table.
Arrays are a first-class type in PostgreSQL. It's a good substitute for using a subsidiary table.
A list of tags is a good fit for arrays.
He talked about hstore. It's much better than Entity-Attribute-Value. It's great for optional, variable attributes. It's like a hash. It can be indexed, searched, etc. It lets you add attributes to tables for users. Don't use it as a way to avoid all table modifications.
json is now a built in type.
There's also jsonb.
Avoid indexes on big things, like 10k character strings.
NULL it a total pain in the neck.
Only use it to mean "missing value".
Never use it to represent a meaningful value.
Let's call anything 1MB or more a "very large object". Store them in files. Store the metadata in the database. The database API is just not a good fit for this.
Many-to-many tables can get extremely large. Consider replacing them with array fields (either one way or both directions). You can use a trigger to maintain integrity.
You don't want more than about 250k entries in an array.
Use UTF-8. Period.
Always use TIMESTAMPTZ (which Django uses by default). Don't use TIMESTAMP. TIMESTAMPTZ is a timestamp converted to UTC.
Index types:
B-Tree
Use a B-Tree on a column if you frequently query on that column,
use one of the comparison operators, only get back 10-15% of the rows,
and run that query frequently.
It won't use the index if you're going to get back more than 15% of
the rows because it's faster to scan a table then scan an index.
Use a partial index if you can ignore most of the rows.
The entire tuple has to be copied into the index.
GiST
It's a framework to create indexes.
KNN indexes are the K-nearest neighbors.
GIN
Generalized inverted index. Used for full-text search.
The others either are not good or very specific.
Why isn't it using my index?
Use explain analyze to look at the query.
If it thinks it's going to require most of the rows, it'll do a table scan.
If it's wrong, use analyze to update the planner stats.
Sometimes, it can't use the index.
Two ways to create an index:
create index
create index concurrently
reindex rebuilds an index from scratch.
pg_stat_user_indexes tells you about how your indexes are being used.
What do you do if a query is slow:
Use explain or explain analyze.
explain doesn't actually run the query.
"Cost" is measured in arbitrary units. Traditionally, they have been "disk fetches". Costs are inclusive of subnodes.
I think explain analyze actually runs the query.
Things that are bad:
Joins between 2 large tables.
Cross joins (cartesian products). These often happen by accident.
Sequential scans on large tables.
select count(*) is slow because it results in a full table scan since you
have to see if the tuples are alive or dead.
offset / limit. These actually run the query and then throw away that many
rows. Beware that GoogleBot is relentless. Use other keys.
If the database is slow:
Look at pg_stat_activity:
select * from pg_stat_activity;
tail -f the logs.
Too much I/O? iostat 5.
If the database isn't responding:
Try connecting with it using psql.
pg_stat_activity
pg_locks
Python Particulars
psycopg2 is the only real option in Python 2.The result set of a query is loaded into client memory when the query completes. If there are a ton of rows, you could run out of memory. If you want to scroll through the results, use a "named" cursor. Be sure to dispose of it properly.
The Python 3 situation is not so great. There's py-postgresql. It's pure Python.
If you are using Django 1.6+, use the @atomic decorator.
Cluster all your writes into small transactions. Leave read operations outside.
Do all your writes at the very end of the view function.
Multi-database works very nicely with hot standby.
Point the writes at the primary, and the reads at the secondary.
For Django 1.5, use the @xact decorator.
Sloppy transaction management can cause the dreaded Django idle-in-transaction problem.
Use South for database migration. South is getting merged into Django in version 1.7 of Django.
You can use manual migrations for stuff the Django ORM can't specify.
Special Situations
Upgrade to 9.3.4. Upgrade minor versions promptly.Major version upgrades require more planning. pg_upgrade has to be run when the database is not running.
A full pg_dump / pg_restore is always the safest, although not the most practical.
Always read the release notes.
All parts of a replication set must be upgraded at once (for major versions).
Use copy, not insert, for bulk loading data. psycopg2 has a nice interface. Do a vacuum afterwards.
AWS
Instances can disappear and come back up without instance storage.EBS can fail to reattach after reboot.
PIOPS are useful (but pricey) if you are using EBS.
Script everything, instance creation, PostgreSQL, etc. Use Salt. Use a VPC.
Scale up and down as required to meet load. If you're just using them to rent a server, it's really expensive.
PostgreSQL RDS is a managed database instance. Big plus: automatic failover! Big minus: you can't read from the secondary. It's expensive. It's a good place to start.
Sharding
Eventually, you'll run out of write capacity on your master.postgres-xc is an open source fork of PostgreSQL.
Bucardo provides multi-master write capability.
He talked about custom sharding.
Instagram wrote a nice article about it.
Pooling
Opening a connection is expensive. Use a pooler.pgbouncer is a pooler.
pgPool II can even do query analysis. However, it has higher overhead and is more complex to configure.
Tools
Monitor everything.check_postgres.pl is a plugin to monitor PostgreSQL.
pgAdmin III and Navicat are nice clients.
pgbadger is for log analysis. So is pg_stat_statements.
Closing
MVCC works by each tuple having a range of transaction IDs that can see thattuple.
Failover is annoying to do in the real world. People use HAProxy, some pooler, etc. with some scripting, or they have a human do the failover.
HandyRep is a server-based tool designed to allow you to manage a PostgreSQL "replication cluster", defined as a master and one or more replicas on the same network.
Shannon -jj Behrens: JJ's Mostly Adequate Summary of the Django Meetup: When *Not* To Use the ORM & Goodbye REST: Building GraphQL APIs with Django
The Django meetup was at Prezi. They have a great space. They are big Django users.
Goodbye REST: Building APIs with Django and GraphQL
Jaden Windle, @jaydenwindle, lead engineer at Jetpack.
https://github.com/jaydenwindle/goodbye-rest-talk
They moved from Django REST Framework to GraphQL.
It sounds like a small app.
They're using Django, React, and React Native.
I think he said they used Reason and moved away from it, but I could be wrong.
They had to do a rebuild anyway, so they used GraphQL.
He said not to switch to GraphQL just because it's cool, and in general, avoid hype-driven development.
GraphQL is a query language, spec, and collection of tools, designed to operate over a single endpoint via HTTP, optimzing for perf and flexibility.
Key features:
- Query for only the data you need.
- Easily query for multiple resources in a single request.
- Great front end tooling for handling caching, loading / error states, and updates.
(I wonder if he's talking about Apollo, not just GraphQL. I found out later, that they are using Apollo.)
GraphQL Schema:
- Types
- Queries
- Mutations
- *Subscriptions
Graphene is the goto framework for GraphQL in Django.
He showed an example app.
You create a type and connect it to a model. It's like a serializer in DRF.
(He's using VS Code.)
It knows how to understand relationships between types based on the relationships in the models.
He showed the query syntax.
He showed how Graphene connects to the Django model. You're returning raw Django model objects, and it takes care of serialization.
There's a really nice UI where you can type in a query, and it queries the server. It has autocomplete. I can't tell if this is from Apollo, Graphene, or some other GraphQL tool.
You only pass what you need across the wire.
When you do a mutation, you can also specify what data it should give back to you.
There is some Mutation class that you subclass.
The code looks a lot like DRF.
Subscriptions aren't fully implemented in Graphene. His company is working on implementing and open sourcing something. There are a bunch of other possible real-time options--http://graphql-python/graphql-ws is one.
There's a way to do file uploads.
Important: There's this thing called graphene-django-extras. There's even something to connect to DRF automatically.
Pros:
- Dramatically improves front end DX
- Flexible types allow for quick iteration
- Always up-to-date documentation
- Only send needed data over the wire
Cons:
- Graphene isn't as mature as some other GraphQL implementations (for instance, in JS and Ruby)
- Logging is different when using a single GraphQL endpoint
- REST is currently better at server-side caching (E-Tag, etc.)
Graphene 3 is coming.
In the Q&A, they said they do use Apollo.
They're not yet at a scale where they have to worry about performance.
He's not entirely sure whether it's prone to the N+1 queries problem, but there are GitHub issues related to that.
You can do raw ORM or SQL queries if you need to. Otherwise, he's not sure what it's doing behind the scenes.
You can add permissions to the models. There's also a way to tie into Django's auth model.
Their API isn't a public API. It's only for use by their own client.
The ORM, and When Not To Use It
Christophe Pettus from PostgreSQL Experts.
He thinks the ORM is great.
The first ORM he wrote was written before a lot of the audience was born.
Sometimes, writing code for the ORM is hard.
Database agnosticism isn't as important as you think it is. For instance, you don't make the paint on your house color-agnostic.
Libraries have to be DB agnostic. Your app probably doesn't need to be.
Reasons you might want to avoid the query language:
- Queries that generate sub-optimal SQL
- Queries more easily expressed in SQL
- SQL features not available via the ORM
Django's ORM's SQL is much better than it used to be.
Don't use __in with very large lists. 100 is about the longest list you should use.
Avoid very deep joins.
It's not hard to chain together a bunch of stuff that ends up generating SQL that's horrible.
The query generator doesn't do a separate optimization pass that makes the query better.
It's better to express .filter().exclude().filter() in SQL.
There are so many powerful functions and operators in PostgreSQL!
SomeModel.objects.raw() is your friend!
You can write raw SQL, and yet still have it integrate with Django models.
You can even get stuff back from the database that isn't in the model definition.
There's some WITH RECURSIVE thing in PostgreSQL that would be prohibitively hard to do with the Django ORM. It's not really recursive--it's iterative.
You can also do queries without using the model framework.
The model framework is very powerful, but it's not cheap.
Interesting: The data has to be converted into Python data and then again into model data. If you're just going to serialize it into JSON, why create the model objects? You can even create the JSON directly in the database and hand it back directly with PostgreSQL. But make sure the database driver doesn't convert the JSON back to Python ;) Return it as raw text.
There are also tables that Django can't treat as model tables. For instance, there are logging tables that lack a primary key. Sometimes, you have weird tables with non-Django-able primary keys.
The ORM is great, though. For instance, it's great for basic CRUD.
Interfaces that require building queries in steps are better done with SQL--for instance, an advanced search function.
Summary:
- Don't be afraid to step outside the ORM.
- SQL isn't a bug. It's a feature. It's code like everything else.
- Do use the ORM for operations that it makes easier.
- Don't hesitate to use the full power of SQL.
Q&A:
Whether you put your SQL in model methods or managers is a matter of taste. Having all the code for a particular model in one place (i.e. the model or manager) is useful.
Write tests. Use CI.
Use parameter substitution in order to avoid SQL injection attacks. Remember, all external input is hostile. You can't use parameter substitution if the table name is dynamic--just be careful with what data you allow.
If you're using PostGIS, write your own queries for sure. It's really easy to shoot yourself in the foot with the GIS package.
gamingdirectional: Create a game over scene for pygame project
In this article we are going to create a game over scene for the pygame project, we will use back the start scene class which we have created previously to render our game over scene by slightly modify it for the multi-scenes uses. Here is the modify version of the start scene class, as you can see we have passed in a state variable to the start scene class’s draw method which will be used...
Talk Python to Me: #188 Async for the Pythonic web with Sanic
James Bennett: On degrees
Lately there’s been a recurring discussion on various social-media outlets about the relevance of academic degrees to a career in programming. Specifically: is a degree in computer science (or some other field perceived as related) a necessity for a career as a programmer? If not, is it still a good idea to have one?
I’ve jumped into a few such threads on Twitter, but I have complex thoughts on the topic, and 140-character or (now) 280-character ...
Davy Wybiral: Getting Started with GPS Modules | Tracking Device | Tutorial
Davy Wybiral: Experimenting with Electric Paint
Anyway, here it is:
Yasoob Khalid: Email Security & Privacy
Hi everyone! I hope all of you are doing well. Things have been super exciting on my side. Just got done with a file system checker project in C for my Operating Systems class. It was mentally draining but equally rewarding. This blog post is not about Python but rather about Emails.
This week I also gave a department wide talk at Colgate University on Email security and privacy and some of the SMTP features which are available at your disposal. It went fairly successful. This was a result of my own independent research on the topic. I tried to cover as much base as possible while also trying to keep the talk engaging.
If you want to learn more about the topic, you can take a look at the slides I used and do some research of your own. If something seems wrong please let me know in the comments below so that we all can benefit. You can also access (I think?) the speaker notes for more context behind the content on the slides.
If you are pressed for time, skip to the last slide to learn about some interesting attacks. One of them makes use of Cyrillic script. If you haven’t heard of Cyrillic script before you would love that slide.
I would love to hear from you all. Please share your views in the comments below about the talk or anything in general.
Weekly Python StackOverflow Report: (cliv) stackoverflow python report
These are the ten most rated questions at Stack Overflow last week.
Between brackets: [question score / answers count]
Build date: 2018-12-01 21:40:29 GMT
- Longest word chain from a list of words - [28/9]
- Why is 2 * x * x faster than 2 * ( x * x ) in Python? - [19/5]
- Do Python lambda functions help in reducing the execution times? - [14/1]
- Python function that identifies if the numbers in a list or array are closer to 0 or 1 - [11/10]
- Finding three integers such that their sum of cosine values become max - [9/5]
- Finding singulars/sets of local maxima/minima in a 1D-NumPy array (once again) - [9/3]
- Improve accuracy of image processing to count fungus spores - [9/2]
- Advanced Python Regex: how to evaluate and extract nested lists and numbers from a multiline string? - [8/2]
- Count appearances of a value until it changes to another value - [7/5]
- What is happening when I assign a list with self references to a list copy with the slice syntax `mylist[:] = [mylist, mylist, ...]`? - [7/3]
Python Bytes: #106 Fluent query APIs on Python collections
gamingdirectional: Create a next level scene for pygame project
In this article we will create a next level scene which will be displayed whenever the player has gained 30 points. Just like the previous chapter we are going to use back the start scene class as well as modify the game manager class and the overlap class First of all we will create the next scene surface and draw it on the game scene whenever the state of the game has changed to the next level...
PyBites: PyBites Twitter Digest - Issue 37, 2018
Digital Ocean Developer Trends of the quarter
Submitted by @BryanKimani
Very interesting report by @digitalocean - https://t.co/V3XRozQfdr#OpenSource
— Max (@maksek_ua) November 27, 2018
A great article on learning things in general
Submitted by @dgjustice
Forget Motivation and Double Your Chances of Learning Success https://t.co/fCLNFWw2Eh
— Ilkka Peltola (@ilkkapeltola) November 27, 2018
A conversation with Guido!
Here's my conversation with Guido van Rossum, creator of Python, one of the most popular and impactful programming… https://t.co/ACmfI6YVbZ
— Lex Fridman (@lexfridman) November 22, 2018
music21 - a Python-based toolkit for computer-aided musicology
Submitted by @Dan Shorstein
Really excited to play with the music21 library! Just found out about it https://t.co/7MwZ0KPw7W
— Dan Shorstein (@dtizzlenizzle) November 29, 2018
Shameless plug because we can! CodeChalleng.es can now help your company run coding interviews with Interview Bites!
PyBites can help you acquire #Python top talent! Enterprise Customers can now use our https://t.co/UYcrpuWnxX plat… https://t.co/ryw2AbtrHq
— Pybites (@pybites) November 27, 2018
Thanks to everyone that participated in the TalkPython Black Friday sale!
Wow, the response to the @talkpython#blackfriday sale has been really amazing. Thank you to everyone to bought a c… https://t.co/bhxqIWxDPb
— Talk Python Podcast (@TalkPython) November 25, 2018
Multiple assignment and tuple unpacking improve Python code readability
Submitted by @tryexceptpass
RT @treyhunner:"When you see Python code that uses hard coded indexes there's often a way to use multiple assignment to make your code mor…
— Cristian Medina (@tryexceptpass) November 23, 2018
Nice, TinyDB!
RT @funkatron: My neat new discovery today is TinyDB, a great, small document db for Python. Perfect for the slackbot I’m writing https://t…
— ernest w. durbin iii (@EWDurbin) November 30, 2018
Too handy not to share. Print a list of reverse DNS records for a subnet
for i in 198.187.190.{0..255}; do a=$( dig +short -x $i ); printf "$i: %s\n""$a"; done # Print a list of reverse DNS records for a subnet.
— Command Line Magic (@climagic) November 25, 2018
Check out Jake VanderPlas's Python Data Science Handbook
RT @mf_viz: The "Python Data Science Handbook" is an *excellent* free + open-source resource for learning #Python for #DataScience and #Mac…
— Jake VanderPlas (@jakevdp) November 28, 2018
A great message: Document Everything!
"If it is not written down, it does not exist." - Philippe Kruchten
— Programming Wisdom (@CodeWisdom) November 29, 2018
Some AMAZING responses to the question: "What's the most fun thing you have ever done with data?"
What's the most fun thing you have ever done with data? A project you really enjoyed, or personal data you collect… https://t.co/KkVegyQMPb
— Data Science Renee (@BecomingDataSci) November 30, 2018
Tracking all of your SQL Queries in Git
RT @beeonaposy: New blog post! Git Your SQL Together, on the why and how of tracking all of your SQL queries in git 💫 https://t.co/Tgrq58i…
— David Robinson (@drob) November 28, 2018
Serverless APIs with Python, AWS Lambda & API Gateway
Serverless APIs with Python, AWS Lambda & API Gateway by Hamish Campbell. https://t.co/HhfW7PQXr6
— Python Software (@ThePSF) December 01, 2018
So old...
If you don't want to feel old, stop reading now. 1TB hard drives are now over a decade old. https://t.co/PJiKXBka9Y
— Command Line Magic (@climagic) November 29, 2018
>>>frompybitesimportBob,JulianKeepCalmandCodeinPython!Vasudev Ram: Tower of Hanoi program in Higher-Order Perl book, ported to Python
Hi, readers,
I was reading the book Higher Order Perl (Wikipedia entry). It is by Mark Jason Dominus, a well-known figure in the Perl community. I've only read a bit of it so far, but it already looks very good. There are also many reviews of it, all of which say that it is a good book, including one by Damian Conway, another well-known Perl figure.
Early in the book, in chapter 1 - Recursion and Callbacks, there is a nice example program showing how to solve the Tower of Hanoi problem. I had come across the problem earlier, but had not seen a Perl solution before. It's not really any different from the same kind of solution written in other languages, except for some points related to Perl's syntax and semantics, such as the use of local (Perl's my) vs. global variables, specifically with regard to a recursive function, which is what Mark uses for Hanoi.
Anyway, since I had not implemented Hanoi before, I thought of porting Mark's hanoi function from Perl to Python. It's an easy recursive algorithm. (The Wikipedia article has multiple algorithms for Hanoi, some non-recursive too.)
I wrote two versions: the first is hard-coded to solve it for a tower with n equal to 3, where n is the number of disks, and the second does it in a loop for n ranging from 1 to 4, with a prompt and a pause after each one.If you press q at any of the prompts, the program quits. If you press any other key, it goes on to the next iteration, until the last one. For reading a keypress without needing to press Enter, I used the getch function (get character) from the msvcrt module that comes with Python. MSVCRT stands for Microsoft Visual C Run Time.So this program is Windows-specific. There are equivalent ways of reading a character without needing to press Enter, for Linux too, but they usually involve some low-level fiddling with the tty / terminal settings, termios, etc. I think ActiveState Code has a recipe for a getch-like function that works on Linux.Here is the first version, hanoi_01.py:
from __future__ import print_functionHere is the output on running it:
"""
This is the code for the Perl version by Mark Jason Dominus:
sub hanoi {
my ($n, $start, $end, $extra) = @_;
if ($n == 1) {
print "Move disk #1 from $start to $end.\n"; # Step 1
} else {
hanoi($n-1, $start, $extra, $end); # Step 2
print "Move disk #$n from $start to $end.\n"; # Step 3
hanoi($n-1, $extra, $end, $start); # Step 4
}
}
"""
# This is my Python version ported from the Perl version above:
def hanoi(n, start, end, extra):
"""
Function hanoi(n, start, end, extra)
Solve Tower of Hanoi problem for a tower of n disks,
of which the largest is disk #n. Move the entire tower from
peg 'start' to peg 'end', using peg 'extra' as a work space.
"""
if n == 1:
print("Move disk #1 from {} to {}.\n".format(start, end))
else:
# Recursive call #1
hanoi(n - 1, start, extra, end)
print("Move disk #{} from {} to {}.\n".format(n, start, end))
# Recursive call #2
hanoi(n - 1, extra, end, start)
# Call to hanoi() with 3 disks, i.e. n = 3:
hanoi(3, 'A', 'C', 'B')
$ python hanoi_01.pyHere is the second version, hanoi_02.py:
Move disk #1 from A to C.
Move disk #2 from A to B.
Move disk #1 from C to B.
Move disk #3 from A to C.
Move disk #1 from B to A.
Move disk #2 from B to C.
Move disk #1 from A to C.
from __future__ import print_functionHere is the output on running it - I pressed q after 3 iterations:
from msvcrt import getch
"""
Tower of Hanoi program
---
By Vasudev Ram
Web site: https://vasudevram.github.io
Blog: https://jugad2.blogspot.com
Twitter: https://mobile.twitter.com/vasudevram
Product store: https://gumroad.com/vasudevram
---
Translated to Python from the Perl version in the book Higher-Order Perl
by Mark Jason Dominus: https://hop.perl.plover.com/book/ , with some
minor enhancements related to interactivity.
"""
def hanoi(n, start, end, extra):
"""
Function hanoi(N, start, end, extra)
Solve Tower of Hanoi problem for a tower of N disks,
of which the largest is disk #N. Move the entire tower from
peg 'start' to peg 'end', using peg 'extra' as a work space.
"""
assert n > 0
if n == 1:
print("Move disk #1 from {} to {}.".format(start, end))
else:
hanoi(n - 1, start, extra, end)
print("Move disk #{} from {} to {}.".format(n, start, end))
hanoi(n - 1, extra, end, start)
for n in range(1, 5):
print("\nTower of Hanoi with n = {}:".format(n))
hanoi(n, 'A', 'C', 'B')
if n < 4:
print("\nPress q to quit, any other key for next run: ")
c = getch()
if c.lower() == 'q':
break
$ python hanoi_02.pyNote that getch does not echo the key that is pressed to the screen. If you want that, use getche, from the same module msvcrt.
Tower of Hanoi with n = 1:
Move disk #1 from A to C.
Press q to quit, any other key for next run:
Tower of Hanoi with n = 2:
Move disk #1 from A to B.
Move disk #2 from A to C.
Move disk #1 from B to C.
Press q to quit, any other key for next run:
Tower of Hanoi with n = 3:
Move disk #1 from A to C.
Move disk #2 from A to B.
Move disk #1 from C to B.
Move disk #3 from A to C.
Move disk #1 from B to A.
Move disk #2 from B to C.
Move disk #1 from A to C.
Press q to quit, any other key for next run:
Viewing the output, I observed that for 1 disk, it takes 1 move, for 2 disks, it takes 3 moves, and for 3 disks, it takes 7 moves. From this, by informally applying mathematical induction, we can easily figure out that for n disks, it will take 2 ** n - 1 moves. The Wikipedia article about the Tower of Hanoi confirms this.That same Wikipedia article (linked above) also shows that the Tower of Hanoi has some interesting applications:Excerpts:[ The Tower of Hanoi is frequently used in psychological research on problem solving. There also exists a variant of this task called Tower of London for neuropsychological diagnosis and treatment of executive functions. ][ Zhang and Norman[10] used several isomorphic (equivalent) representations of the game to study the impact of representational effect in task design. ][ The Tower of Hanoi is also used as a Backup rotation scheme when performing computer data Backups where multiple tapes/media are involved. ][ The Tower of Hanoi is popular for teaching recursive algorithms to beginning programming students. ]And this application is possibly the most interesting of the lot - ants solving the Tower of Hanoi problem :)[ In 2010, researchers published the results of an experiment that found that the ant species Linepithema humile were successfully able to solve the 3-disk version of the Tower of Hanoi problem through non-linear dynamics and pheromone signals. ]Also check out the section in that Wikipedia article with the title:"General shortest paths and the number 466/885"Not directly related, but similar to the above ratio, did you know that there is an easy-to-remember and better approximation to the number pi, than the ratio 22/7 that many of us learned in school?Just take the 1st 3 odd numbers, i.e. 1, 3, 5, repeat each of them once, and join all those 6 digits, to get 113355, split that number in the middle, and divide the 2nd half by the 1st half, i.e 355/113, which gives 3.1415929203539823008849557522124. (See the section titled "Approximate value and digits" in the Wikipedia article about pi linked above, for this and other approximations for pi).Pressing the pi button in Windows CALC.EXE gives 3.1415926535897932384626433832795, and comparing that with the ratio 355/113 above, shows that the ratio is accurate to 6 decimal places. I've read somewhere that this ratio was used by mathematicians in ancient India.Here are some interesting excerpts from the Wikipedia article about pi:[ The number pi is a mathematical constant. Originally defined as the ratio of a circle's circumference to its diameter, it now has various equivalent definitions and appears in many formulas in all areas of mathematics and physics. It is approximately equal to 3.14159. It has been represented by the Greek letter "p" since the mid-18th century, though it is also sometimes spelled out as "pi". It is also called Archimedes' constant. ][ Being an irrational number, pi cannot be expressed as a common fraction (equivalently, its decimal representation never ends and never settles into a permanently repeating pattern). Still, fractions such as 22/7 and other rational numbers are commonly used to approximate pi. The digits appear to be randomly distributed. In particular, the digit sequence of pi is conjectured to satisfy a specific kind of statistical randomness, but to date, no proof of this has been discovered. Also, pi is a transcendental number; that is, it is not the root of any polynomial having rational coefficients. ][ Being an irrational number, pi cannot be expressed as a common fraction (equivalently, its decimal representation never ends and never settles into a permanently repeating pattern). Still, fractions such as 22/7 and other rational numbers are commonly used to approximate p. The digits appear to be randomly distributed. In particular, the digit sequence of pi is conjectured to satisfy a specific kind of statistical randomness, but to date, no proof of this has been discovered. Also, pi is a transcendental number; that is, it is not the root of any polynomial having rational coefficients. This transcendence of pi implies that it is impossible to solve the ancient challenge of squaring the circle with a compass and straightedge. ][ Ancient civilizations required fairly accurate computed values to approximate pi for practical reasons, including the Egyptians and Babylonians. Around 250 BC the Greek mathematician Archimedes created an algorithm for calculating it. In the 5th century AD Chinese mathematics approximated pi to seven digits, while Indian mathematics made a five-digit approximation, both using geometrical techniques. The historically first exact formula for p, based on infinite series, was not available until a millennium later, when in the 14th century the Madhava–Leibniz series was discovered in Indian mathematics. ](Speaking of Indian mathematics, check out this earlier post by me:Bhaskaracharya and the man who found zero.)[ Because its most elementary definition relates to the circle, pi is found in many formulae in trigonometry and geometry, especially those concerning circles, ellipses, and spheres. In more modern mathematical analysis, the number is instead defined using the spectral properties of the real number system, as an eigenvalue or a period, without any reference to geometry.It appears therefore in areas of mathematics and the sciences having little to do with the geometry of circles, such as number theory and statistics, as well as in almost all areas of physics.The ubiquity of pi makes it one of the most widely known mathematical constants both inside and outside the scientific community. Several books devoted to pi have been published, and record-setting calculations of the digits of pi often result in news headlines. Attempts to memorize the value of pi with increasing precision have led to records of over 70,000 digits. ]
- Enjoy.
- Vasudev Ram - Online Python training and consulting
I conduct online courses on Python programming, Unix / Linux commands and shell scripting and SQL programming and database design, with course material and personal coaching sessions.The course details and testimonials are here.Contact me for details of course content, terms and schedule.Try FreshBooks: Create and send professional looking invoices in less than 30 seconds.Getting a new web site or blog, and want to help preserve the environment at the same time? Check out GreenGeeks.com web hosting.Sell your digital products via DPD: Digital Publishing for Ebooks and Downloads.Learning Linux? Hit the ground running with my vi quickstart tutorial.I wrote it at the request of two Windows system administrator friends who were given additional charge of some Unix systems. They later told me that it helped them to quickly start using vi to edit text files on Unix. Of course, vi/vim is one of the most ubiquitous text editors around, and works on most other common operating systems and on some uncommon ones too, so the knowledge of how to use it will carry over to those systems too.Check out WP Engine, powerful WordPress hosting.Sell More Digital Products With SendOwl.Get a fast web site with A2 Hosting.Creating online products for sale? Check out ConvertKit, email marketing for online creators.Teachable: feature-packed course creation platform, with unlimited video, courses and students.Posts about: Python * DLang * xtopdfMy ActiveState Code recipesFollow me on:
Podcast.__init__: Keeping Up With The Python Community For Fun And Profit with Dan Bader
Summary
Keeping up with the work being done in the Python community can be a full time job, which is why Dan Bader has made it his! In this episode he discusses how he went from working as a software engineer, to offering training, to now managing both the Real Python and PyCoders properties. He also explains his strategies for tracking and curating the content that he produces and discovers, how he thinks about building products, and what he has learned in the process of running his businesses.
Preface
- Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great.
- When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so check out Linode. With 200 Gbit/s private networking, scalable shared block storage, node balancers, and a 40 Gbit/s public network, all controlled by a brand new API you’ve got everything you need to scale up. Go to podcastinit.com/linode to get a $20 credit and launch a new server in under a minute.
- Visit the site to subscribe to the show, sign up for the newsletter, and read the show notes. And if you have any questions, comments, or suggestions I would love to hear them. You can reach me on Twitter at @Podcast__init__ or email hosts@podcastinit.com)
- To help other people find the show please leave a review on iTunes, or Google Play Music, tell your friends and co-workers, and share it on social media.
- Join the community in the new Zulip chat workspace at podcastinit.com/chat
- Your host as usual is Tobias Macey and today I’m interviewing Dan Bader about finding, filtering, and creating resources for Python developers at Real Python, PyCoders, and his own trainings
Interview
- Introductions
- How did you get introduced to Python?
- Let’s start by discussing your primary job these days and how you got to where you are.
- In the past year you have also taken over management of the Real Python site. How did that come about and what are your responsibilities?
- You just recently took over management of the PyCoders newsletter and website. Can you describe the events that led to that outcome and the responsibilities that came along with it?
- What are the synergies that exist between your various roles and projects?
- What are the areas of conflict? (e.g. time constraints, conflicts of interest, etc.)
- Between PyCoders, Real Python, your training materials, your Python tips newsletter, and your coaching you have a lot of incentive to keep up to date with everything happening in the Python ecosystem. What are your strategies for content discovery?
- With the diversity in use cases, geography, and contributors to the landscape of Python how do you work to counteract any bias or blindspots in your work?
- There is a constant stream of information about any number of topics and subtopics that involve the Python language and community. What is your process for filtering and curating the resources that are ultimately included in the various media properties that you oversee?
- In my experience with the podcast one of the most difficult aspects of maintaining relevance as a content creator is obtaining feedback from your audience. What do you do to foster engagement and facilitate conversations around the work that you do?
- You have also built a few different product offerings. Can you discuss the process involved in identifying the relevant opportunities and the creation and marketing of them?
- Creating, collecting, and curating content takes a significant investment of time and energy. What are your avenues for ensuring the sustainability of your various projects?
- What are your plans for the future growth and development of your media empire?
- As someone who is so deeply involved in the conversations flowing through and around Python, what do you see as being the greatest threats and opportunities for the language and its community?
Keep In Touch
- @dbader_org on Twitter
- Website
- dbader on GitHub
Picks
- Tobias
- Dan
- Black code formatter
- Łukasz Langa
Links
- Dan Bader
- Nerd Lettering
- Real Python
- PyCoders
- Computer Science
- Vancouver, BC
- Django
- Raymond Hettinger
- Data Science
- Flask
- Pythonista Cafe
- Python Tricks
The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA
gamingdirectional: Create a win scene and the level manager class for pygame project
In this article we will create a pop up win scene when the player has won all the levels which it will then ask the player whether he wants to start the game all over again or not? We will also create a framework for the level manager class which we will further add in more features in the next chapter. First of all, we will modify the start scene class again to include in a win scene graphic.
Techiediaries - Django: CSS Grid Layout Tutorial—Styling a Django Template
Throughout this tutorial, we'll learn about CSS Grid Layout. We'll be using a Django template.
There are many popular techniques for creating responsive layouts. In this tutorial, we'll be building a simple Django web application with a modern UI styled with CSS Grid layout.
By building the UI using CSS Grid, you'll learn about many useful and easy techniques for achieving popular requirements such as centering and spanning items, switching layouts depending on the screen size and creating responsive UIs.
Before, we dive into practical steps, let's first introduce CSS Grid.
What is CSS Grid?
CSS Grid is a modern 2-dimentionnal system for creating HTML layouts. It's now supported by most web browsers. It makes creating professional and complex layouts more easier than ever!
CSS Grid Layout allows you to build advanced grid layouts in CSS instead of HTML like the case for tables for example.
Unlike CSS floats and HTML tables you can create a grid layout in a straightforward way. You simply need to use an HTML element with its display property set to grid or inline-grid. This way, any elements contained in the parent element or the container becomes grid items. If you don't specify any other CSS Grid property, the items will be automatically arranged in a grid structure thanks to a powerful grid algorithm.
If you need more control over the grid items, you can use different CSS Grid properties to specify the different options for the items including position.
Prerequisites
If you want to follow this tutorial, step by step to build a CSS Grid UI for your Django application. You will need a few requirements. You need to have;
- A development environment ready with Python 3 and PIP installed,
- Basic knowledge of Python,
- Working knowledge of Django.
Creating a Virtual Environment & Installing Django
Now let's start by creating a virtual environment that will allow you to isolate your project's dependencies from the other Python dependencies installed on your system.
Go to your terminal and run the following command to create a virtual environment using the venv module included in Python 3.
$ cd ~
$ python3 -m venv env
Next, you need to activate this environment using:
$ source env/bin/activate
Now, you can install Django using pip:
$ pip install django
Creating a Django Project
After installing Django, you now need to create a project. Head back to your terminal and run the following command:
$ django-admin start css
$ django-admin.py startproject cssgriddemo
This command will generate a project named cssgriddemo.
Creating a Django Application
Now that you have created a Django project, you need to create a Django application. Head over to your terminal and run the following command to create the app:
$ python manage.py startapp gridui
Next, add it to the list of installed apps in the settings.py file:
INSTALLED_APPS=[# [...]'gridui']Next, we need to add a template and a view function.
Open the gridui/views.py file and add the following import:
fromdjango.views.generic.baseimportTemplateView```Nextaddthefollowingviewclass:```pythonclassHome(TempateView):template_name='gridui/index.html'Next, add the gridui/index.html template with the following code:
<!DOCTYPE html><htmllang="en"><head><metacharset="utf-8"><title>Django UI with CSS Grid Layout</title></head><body><divclass="header"><h1>Django UI with CSS Grid Layout</h1></div><divclass="sidebar">
.sidebar
</div><divclass="main"><divclass="item"><span>1</span></div><divclass="item"><span>2</span></div><divclass="item"><span>...</span></div><divclass="item"><span>12</span></div></div><divclass="footer">
Copyright 2018
</div></body></html>We create an HTML template with the header, sidebar, main and footer sections. In the main section, we include a set of items.
Finally, let's add an URL to our urls.py file
fromdjango.urlsimportpathfromgriduiimportviewsurlpatterns=[# [...]path('',views.Home.as_view())]You can now migrate your database and run the Django server:
$ python manage.py migrate
$ python manage.py runserver
You application will be running from localhost:8000.
This is a screenshot of our interface at this point:

Next, let's add basic CSS styling. Add a <style> tag and add the following styles:
<style>body{background:#478dee;margin:5px;padding:0px;font-family:-apple-system,BlinkMacSystemFont,“SegoeUI”,“Roboto”,“Oxygen”,“Ubuntu”,“Cantarell”,“FiraSans”,“DroidSans”,“HelveticaNeue”,sans-serif;}div.header{text-transform:uppercase;}</style>This is a screenshot of the interface:

Adding CSS Grid
Now that we have created a simple HTML structure, we can proceed by styling the UI using CSS Grid.
First we need to make the <body> element a grid container and define the grid columns, rows and areas:
body{//[...]display:grid;grid-gap:0.2vw;height:100vh;grid-template-columns:150px1fr;grid-template-rows:120px1fr61px;grid-template-areas:"header header""sidebar content""footer footer";}We use display:grid pair to make <body> a grid container. We also set a grid gap of _0.2vw.
We make use of grid-template-columns to define two columns, the first column has a fixed width of 150px and the second column has the remaining width.
fr is a fractional unit and 1fr means the element should take 1 part of the available space.
Next, we use grid-template-rows to define three rows:
- The first row has a fixed height of 120px ,
- The third row has a fixed height of 61px,
- The second row has the remaining space (1fr).
We finally use grid-template-areas for assigning the virtual cells, resulted from the intersection of columns and rows, to areas.
Now we need to define those areas specified in the areas template using grid-area:
div.header{grid-area:header;}div.sidebar{grid-area:sidebar;background:#0769f3;}div.footer{grid-area:footer;background:#3581eb;}div.main{grid-area:content;}This is the screen shot of the result now:

Adding a Nested CSS Grid
Now that we have created the main layout of the page, let's use CSS Grid to create a nested grid in the content area.
Grid children can also be Grid containers.
Let's define the content area as a grid container. In the <style> tag, add the following CSS:
div.main{display:grid;grid-gap:0.2vw;grid-template-columns:repeat(auto-fill,minmax(150px,1fr));grid-template-rows:repeat(auto-fill,minmax(150px,1fr));}Here we use repeat, auto-fill and minmax to define columns and rows.
Also add the following style to use an image background for items:
div.item{background-image:url('https://source.unsplash.com/daily');}This is the screen shot of the result after adding more <div class="item"> elements: :

CSS Grid grid-column, grid-row and span
CSS Grid enables you to control the position of grid elements using grid-column and grid-row.
You can also use span to define how much columns or rows a specified element takes.
Now back to our project. Let's span the second item of the content area across five columns and two rows and position it from column line 2 and row line 1 (default location).
div.main>div.item:nth-child(2){grid-column:2/span5;grid-row:1/span2;}This is a screen shot of the result:

Conclusion
In this tutorial you've seen how to create a beautiful UI for a Django application using CSS Grid Layout.
Mike Driscoll: PyDev of the Week: Erika Fille Legara
This week we welcome Erika Fille Legara (@eflegara) as our PyDev of the Week. Erika is a professor and program director at the Asian Institute of Management. She has spoken at PyCon Philippines. You can check out her website to see what else she has been up to or watch her talk below:
Let’s take a few moments to get to know her better!
Can you tell us a little about yourself (hobbies, education, etc):
Hi, I’m Erika. I am a physicist by training. I am your typical grad school (assoc.) professor/administrator who’s always trying her best to strike the appropriate balance between teaching, research, and fulfilling certain administrative duties. At the moment, the research projects I am engage in are mostly industry-driven.
Outside work, I enjoy traveling and travel photography. With the recent career transition, however, leisure travels have been minimized. Nowadays, I spend most of my free time reading, listening to music, and yes, binge-watching. I also love highway driving, every now and then, on weekends; it helps the mind relax. I like the fact that in long drives I get to listen to awesome road trip playlists without interruption.
Why did you start using Python?
I started writing scripts in C++ for my undergrad research. My thesis was on the study of complex systems; network science, in particular. One of my colleagues then at the lab introduced me to Python when he saw me writing really, really long scripts in C++ to build complex network models. He showed me how I can reduce the 50 or so (or longer) lines of code I wrote in C++ to only a few lines, less than 10 actually, in Python, thanks to all python scientific libraries developers and contributors. Since then, I’ve never looked back.
What other programming languages do you know and which is your favorite?
Other languages are C/C++, R, and MATLAB. I am also familiar with PHP.
But my one and only favorite is Python, of course.
What projects are you working on now?
I am currently working on a couple of research projects involving urban systems modeling, transport systems modeling, analysis and modeling of the spread of (mis/dis)information in social systems, and building predictive models for location analysis.
Which Python libraries are your favorite (core or 3rd party)?
NumPy and SciPy, Matplotlib, PANDAS, scikit-learn, NetworkX, graph-tool, Keras, TensorFlow, H2Oai
Is there anything else you’d like to say?
Never stop learning! The world, especially with today’s technologies, has so much to offer.
Thanks for doing the interview, Erika!
