Quantcast
Channel: Planet Python
Viewing all 24370 articles
Browse latest View live

Codementor: Subtleties of Python

$
0
0
A good software engineer understands how crucial attention to detail is; minute details, if overlooked, can make a world of difference between a working unit and a disaster. That’s why writing...

Red Hat Developers: What, No Python in RHEL 8 Beta?

$
0
0

TL;DR Of course we have Python! You just need to specify if you want Python 3 or 2 as we didn’t want to set a default. Give yum install python3 and/or yum install python2 a try. Or, if you want to see what we recommend you install yum install @python36 or yum install @python27. Read on for why:

For prior versions of Red Hat Enterprise Linux, and most Linux Distributions, users have been locked to the system version of Python unless they got away from the system’s package manager. While this can be true for a lot of tools (ruby, node, PERL, php) the Python use case is more complicated because so many Linux tools (like yum) rely on Python. In order to improve the experience for RHEL 8 users, we have moved the Python used by the system “off to the side” and we introduced the concept of Application Streams based on Modularity.

Through Application Streams, in combination with Python’s ability to be parallel installed, we can now make multiple versions of Python available and easily installable, from the standard repositories, into the standard locations. No extra things to learn or manage. Now, users can choose what version of Python they want to run in any given userspace and it simply works. For more info, see my article, Introducing Application Streams in RHEL 8.

To be honest, the system maintainers also get some advantages of not being locked to an aging version of Python for our system tools. With users not relying on a particular version of Python coming with the system installation, we have the freedom to take advantage of new language features, performance improvements, and all the other goodness a developer gets when tracking near the upstream version.

However, this has resulted in a dilemma. When a user sits down at a fresh installation of RHEL 8 they will naturally expect that /usr/bin/python will run some version of Python. If you follow the recommendation in Python Enhancement Proposal (PEP) 394, that will be Python 2. However, at some point, a new PEP will likely want to change that recommendation to Python 3, probably during, the typically *10* year, life of RHEL 8!  To put this in perspective, consider that RHEL 7 was released in 2014, and will be supported until 2024!  

So, what do we do? Well, if we follow the current recommendation, we make some present day users happy. However, when the Python Community shifts to recommending Python 3 as the default, we will make new users unhappy.

As a result, we came to the tough conclusion, don’t provide a default, unversioned Python at all. Ideally, people will get used to explicitly typing python3 or python2. However for those that want an unversioned command,  let them chose from the beginning which version of Python they actually want. So, yum install python results in a 404.

However, we do try to make it as easy as possible to get Python 2 or 3 (or both) on to your system. We recommend using yum install @python36 or yum install @python27 to take advantage of the recommended set of packages to install. If all you really need is *just* the Python binaries, you can use yum install python3 or yum install python2.

We have also setup the alternatives infrastructure so that when you install either (or both) you can easily make /usr/bin/python point to the right place using alternatives --config python. However, as we explained above, and aligned with the Python PEP, we don’t recommend relying on /usr/bin/python being the correct python for your application.

Note: the same issue arises for Python wrapper scripts like pip. Installing Python 3 will put pip3 in your path, but not unversioned pip. With Python modules like pip, venv, and virtualenv, you can avoid confusion and get the right version by running those as a module: python3 -m pip and avoiding the wrapper scripts. Using Python virtual environments is a best practice that also avoids the issues with version ambiguity, see How to install Python 3 on Red Hat Enterprise Linux 7 for virtual environment details and advice.

To conclude, yes, Python is included in RHEL 8! And, it will be even better than in the past! If you want more details on anything in this post, please see the How To Guide on Red Hat Developers.

Oh and if you haven’t downloaded RHEL 8 yet—go to developers.redhat.com/rhel8 now.

Additional Information

 

Share

The post What, No Python in RHEL 8 Beta? appeared first on RHD Blog.

Stack Abuse: Handling Unix Signals in Python

$
0
0

UNIX/Linux systems offer special mechanisms to communicate between each individual process. One of these mechanisms are signals, and belong to the different methods of communication between processes (Inter Process Communication, abbreviated with IPC).

In short, signals are software interrupts that are sent to the program (or the process) to notify the program of significant events or requests to the program in order to run a special code sequence. A program that receives a signal either stops or continues the execution of its instructions, terminates either with or without a memory dump, or even simply ignores the signal.

Although it is defined in the POSIX standard, the reaction actually depends on how the developer wrote the script and implemented the handling of signals.

In this article we explain what are signals, show you how to sent a signal to another process from the command line as well as processing the received signal. Among other modules, the program code is mainly based on the signal module. This module connects the according C headers of your operating system with the Python world.

An Introduction to Signals

On UNIX-based systems, there are three categories of signals:

  • System signals (hardware and system errors): SIGILL, SIGTRAP, SIGBUS, SIGFPE, SIGKILL, SIGSEGV, SIGXCPU, SIGXFSZ, SIGIO

  • Device signals: SIGHUP, SIGINT, SIGPIPE, SIGALRM, SIGCHLD, SIGCONT, SIGSTOP, SIGTTIN, SIGTTOU, SIGURG, SIGWINCH, SIGIO

  • User-defined signals: SIGQUIT, SIGABRT, SIGUSR1, SIGUSR2, SIGTERM

Each signal is represented by an integer value, and the list of signals that are available is comparably long and not consistent between the different UNIX/Linux variants. On a Debian GNU/Linux system, the command kill -l displays the list of signals as follows:

$ kill -l
 1) SIGHUP   2) SIGINT   3) SIGQUIT  4) SIGILL   5) SIGTRAP
 6) SIGABRT  7) SIGBUS   8) SIGFPE   9) SIGKILL 10) SIGUSR1
11) SIGSEGV 12) SIGUSR2 13) SIGPIPE 14) SIGALRM 15) SIGTERM  
16) SIGSTKFLT   17) SIGCHLD 18) SIGCONT 19) SIGSTOP 20) SIGTSTP  
21) SIGTTIN 22) SIGTTOU 23) SIGURG  24) SIGXCPU 25) SIGXFSZ  
26) SIGVTALRM   27) SIGPROF 28) SIGWINCH    29) SIGIO   30) SIGPWR  
31) SIGSYS  34) SIGRTMIN    35) SIGRTMIN+1  36) SIGRTMIN+2  37) SIGRTMIN+3  
38) SIGRTMIN+4  39) SIGRTMIN+5  40) SIGRTMIN+6  41) SIGRTMIN+7  42) SIGRTMIN+8  
43) SIGRTMIN+9  44) SIGRTMIN+10 45) SIGRTMIN+11 46) SIGRTMIN+12 47) SIGRTMIN+13  
48) SIGRTMIN+14 49) SIGRTMIN+15 50) SIGRTMAX-14 51) SIGRTMAX-13 52) SIGRTMAX-12  
53) SIGRTMAX-11 54) SIGRTMAX-10 55) SIGRTMAX-9  56) SIGRTMAX-8  57) SIGRTMAX-7  
58) SIGRTMAX-6  59) SIGRTMAX-5  60) SIGRTMAX-4  61) SIGRTMAX-3  62) SIGRTMAX-2  
63) SIGRTMAX-1  64) SIGRTMAX  

The signals 1 to 15 are roughly standardized, and have the following meaning on most of the Linux systems:

  • 1 (SIGHUP): terminate a connection, or reload the configuration for daemons
  • 2 (SIGINT): interrupt the session from the dialogue station
  • 3 (SIGQUIT): terminate the session from the dialogue station
  • 4 (SIGILL): illegal instruction was executed
  • 5 (SIGTRAP): do a single instruction (trap)
  • 6 (SIGABRT): abnormal termination
  • 7 (SIGBUS): error on the system bus
  • 8 (SIGFPE): floating point error
  • 9 (SIGKILL): immmediately terminate the process
  • 10 (SIGUSR1): user-defined signal
  • 11 (SIGSEGV): segmentation fault due to illegal access of a memory segment
  • 12 (SIGUSR2): user-defined signal
  • 13 (SIGPIPE): writing into a pipe, and nobody is reading from it
  • 14 (SIGALRM): the timer terminated (alarm)
  • 15 (SIGTERM): terminate the process in a soft way

In order to send a signal to a process in a Linux terminal you invoke the kill command with both the signal number (or signal name) from the list above and the id of the process (pid). The following example command sends the signal 15 (SIGTERM) to the process that has the pid 12345:

$ kill -15 12345

An equivalent way is to use the signal name instead of its number:

$ kill -SIGTERM 12345

Which way you choose depends on what is more convenient for you. Both ways have the same effect. As a result the process receives the signal SIGTERM, and terminates immediately.

Using the Python signal Library

Since Python 1.4, the signal library is a regular component of every Python release. In order to use the signal library, import the library into your Python program as follows, first:

import signal  

Capturing and reacting properly on a received signal is done by a callback function - a so-called signal handler. A rather simple signal handler named receiveSignal() can be written as follows:

def receiveSignal(signalNumber, frame):  
    print('Received:', signalNumber)
    return

This signal handler does nothing else than reporting the number of the received signal. The next step is registering the signals that are caught by the signal handler. For Python programs, all the signals (but 9, SIGKILL) can be caught in your script:

if __name__ == '__main__':  
    # register the signals to be caught
    signal.signal(signal.SIGHUP, receiveSignal)
    signal.signal(signal.SIGINT, receiveSignal)
    signal.signal(signal.SIGQUIT, receiveSignal)
    signal.signal(signal.SIGILL, receiveSignal)
    signal.signal(signal.SIGTRAP, receiveSignal)
    signal.signal(signal.SIGABRT, receiveSignal)
    signal.signal(signal.SIGBUS, receiveSignal)
    signal.signal(signal.SIGFPE, receiveSignal)
    #signal.signal(signal.SIGKILL, receiveSignal)
    signal.signal(signal.SIGUSR1, receiveSignal)
    signal.signal(signal.SIGSEGV, receiveSignal)
    signal.signal(signal.SIGUSR2, receiveSignal)
    signal.signal(signal.SIGPIPE, receiveSignal)
    signal.signal(signal.SIGALRM, receiveSignal)
    signal.signal(signal.SIGTERM, receiveSignal)

Next, we add the process information for the current process, and detect the process id using the methode getpid() from the os module. In an endless while loop we wait for incoming signals. We implement this using two more Python modules - os and time. We import them at the beginning of our Python script, too:

import os  
import time  

In the while loop of our main program the print statement outputs "Waiting...". The time.sleep() function call makes the program wait for three seconds.

    # output current process id
    print('My PID is:', os.getpid())

    # wait in an endless loop for signals 
    while True:
        print('Waiting...')
        time.sleep(3)

Finally, we have to test our script. Having saved the script as signal-handling.py we can invoke it in a terminal as follows:

$ python3 signal-handling.py 
My PID is: 5746  
Waiting...  
...

In a second terminal window we send a signal to the process. We identify our first process - the Python script - by the process id as printed on screen, above.

$ kill -1 5746

The signal event handler in our Python program receives the signal we have sent to the process. It reacts accordingly, and simply confirms the received signal:

...
Received: 1  
...

Ignoring Signals

The signal module defines ways to ignore received signals. In order to do that the signal has to be connected with the predefined function signal.SIG_IGN. The example below demonstrates that, and as a result the Python program cannot be interrupted by CTRL+C anymore. To stop the Python script an alternative way has been implemented in the example script - the signal SIGUSR1 terminates the Python script. Furthermore, instead of an endless loop we use the method signal.pause(). It just waits for a signal to be received.

import signal  
import os  
import time

def receiveSignal(signalNumber, frame):  
    print('Received:', signalNumber)
    raise SystemExit('Exiting')
    return

if __name__ == '__main__':  
    # register the signal to be caught
    signal.signal(signal.SIGUSR1, receiveSignal)

    # register the signal to be ignored
    signal.signal(signal.SIGINT, signal.SIG_IGN)

    # output current process id
    print('My PID is:', os.getpid())

    signal.pause()

Handling Signals Properly

The signal handler we have used up to now is rather simple, and just reports a received signal. This shows us that the interface of our Python script is working fine. Let's improve it.

Catching the signal is already a good basis but requires some improvement to comply with the rules of the POSIX standard. For a higher accuracy each signal needs a proper reaction (see list above). This means that the signal handler in our Python script needs to be extended by a specific routine per signal. This works best if we understand what a signal does, and what a common reaction is. A process that receives signal 1, 2, 9 or 15 terminates. In any other case it is expected to write a core dump, too.

Up to now we have implemented a single routine that covers all the signals, and handles them in the same way. The next step is to implement an individual routine per signal. The following example code demonstrates this for the signals 1 (SIGHUP) and 15 (SIGTERM).

def readConfiguration(signalNumber, frame):  
    print ('(SIGHUP) reading configuration')
    return

def terminateProcess(signalNumber, frame):  
    print ('(SIGTERM) terminating the process')
    sys.exit()

The two functions above are connected with the signals as follows:

    signal.signal(signal.SIGHUP, readConfiguration)
    signal.signal(signal.SIGTERM, terminateProcess)

Running the Python script, and sending the signal 1 (SIGHUP) followed by a signal 15 (SIGTERM) by the UNIX commands kill -1 16640 and kill -15 16640 results in the following output:

$ python3 daemon.py
My PID is: 16640  
Waiting...  
Waiting...  
(SIGHUP) reading configuration
Waiting...  
Waiting...  
(SIGTERM) terminating the process

The script receives the signals, and handles them properly. For clarity, this is the entire script:

import signal  
import os  
import time  
import sys

def readConfiguration(signalNumber, frame):  
    print ('(SIGHUP) reading configuration')
    return

def terminateProcess(signalNumber, frame):  
    print ('(SIGTERM) terminating the process')
    sys.exit()

def receiveSignal(signalNumber, frame):  
    print('Received:', signalNumber)
    return

if __name__ == '__main__':  
    # register the signals to be caught
    signal.signal(signal.SIGHUP, readConfiguration)
    signal.signal(signal.SIGINT, receiveSignal)
    signal.signal(signal.SIGQUIT, receiveSignal)
    signal.signal(signal.SIGILL, receiveSignal)
    signal.signal(signal.SIGTRAP, receiveSignal)
    signal.signal(signal.SIGABRT, receiveSignal)
    signal.signal(signal.SIGBUS, receiveSignal)
    signal.signal(signal.SIGFPE, receiveSignal)
    #signal.signal(signal.SIGKILL, receiveSignal)
    signal.signal(signal.SIGUSR1, receiveSignal)
    signal.signal(signal.SIGSEGV, receiveSignal)
    signal.signal(signal.SIGUSR2, receiveSignal)
    signal.signal(signal.SIGPIPE, receiveSignal)
    signal.signal(signal.SIGALRM, receiveSignal)
    signal.signal(signal.SIGTERM, terminateProcess)

    # output current process id
    print('My PID is:', os.getpid())

    # wait in an endless loop for signals 
    while True:
        print('Waiting...')
        time.sleep(3)

Further Reading

Using the signal module and an according event handler it is relatively easy to catch signals. Knowing the meaning of the different signals, and to react properly as defined in the POSIX standard is the next step. It requires that the event handler distinguishes between the different signals, and has a separate routine for all of them.

Shannon -jj Behrens: PyCon Notes: PostgreSQL Proficiency for Python People

$
0
0
In summary, this tutorial was fantastic! I learned more in three hours than I would have learned if I had read a whole book!

Here's the video. Here are the slides. Here are my notes:

Christophe Pettus was the speaker. He's from PostgreSQL Experts.

PostgreSQL is a rich environment.

It's fully ACID compliant.

It has the richest set of features of any modern, production RDMS. It has even more features than
Oracle.

PostgreSQL focuses on quality, security, and spec compliance.

It's capable of very high performance: tens of thousands of transactions per second, petabyte-sized data sets, etc.

To install it, just use your package management system (apt, yum, etc.). Those systems will usually take care of initialization.

There are many options for OS X. Heroku even built a Postgres.app that runs more like a foreground app.

A "cluster" is a single PostgreSQL server (which can manage multiple databases).

initdb creates the basic file structure. PostgreSQL has to be up and running to run initdb.

To create a database:

sudo su - postgres
psql

create database this_new_database;

To drop a database:

drop database this_new_database;

Debian runs initdb for you. Red Hat does not.

Debian has a cluster management system. Use it. See, for instance, pg_createcluster.

Always create databases as UTF-8. Once you've created it, you can't change it.

Don't use SQLASCII. It's a nightmare. Don't use "C locale".

pg_ctl is a built-in command to start and stop PostgreSQL:

cd POSTGRES_DIRECTORY
pg_ctl -D . start

Usually, pg_ctl is wrapped by something provided by your platform.

On Ubuntu, start PostgreSQL via:

service postgresql start

Always use "-m fast" when stopping.

Postgres puts its own data in a top-level directory. Let's call it $PGDATA.

Don't monkey around with that data.

pg_clog and pg_xlog are important. Don't mess with them.

On most systems, configuration lives in $PGDATA.

postgresql.conf contains server configuration.

pg_hba.conf contains authentication settings.

postgresql.conf can feel very overwhelming.

Avoid making a lot of changes to postgresql.conf. Instead, add the following to it:

include "postgresql.conf.include"

Then, mess with "postgresql.conf.include".

The important parameters fall into these categories: logging, memory, checkpoints, and the planner.

Logging:

Be generous with logging. It has a very low impact on the system. It's your best source of info for diagnosing problems.

You can log to syslog or log CSV to files. He showed his typical logging configuration.

He showed his guidelines / heuristics for all the settings, including how to finetune things. They're really good! See his slides.

As of version 9.3, you don't need to tweak Linux kernel parameters anymore.

Do not mess with fsync or synchronous_commit.

Most settings require a server reload to take effect. Some things require a server restart. Some can be set on a per-session basis. Here's how to do that. This is also an example of how to use a transaction:

begin;
set local random_page_cost = 2.5;
show random_page_cost;
abort;

pg_hba.conf contains users and roles. Roles are like groups. They form a hierarchy.

A user is just a role with login privs.

Don't use the "postgres" superuser for anything application-related.

Sadly, you probably will have to grant schema-modification privs to your app user if you use migrations, but if you don't have to, don't.

By default, DB traffic is not encrypted. Turn on SSL if you are running in a cloud provider.

In pg_hba.conf, "trust" means if they can log into the server, they can access Postgres too. "peer" means they can have a Postgres user that matches their username. "md5" is an md5 hash password.

It's a good idea to restrict the IP addresses allowed to talk to the server fairly tightly.

The WAL

The Write-Ahead Log is key to many Postgres operations. It's the basis for replication, crash recovery, etc.

When each transaction is committed, it is logged to the write-ahead log.

Changes in the transaction are flushed to disk.

If the system crashes, the WAL is "replayed" to bring the DB to a consistent state.

It's a continuous record of changes since the last checkpoint.

The WAL is stored in 16MB segments in the pg_xlog directory.

Never delete anything from pg_xlog.

archive_command is a way to move the WAL segments to someplace safe (like a
different system).

By default, synchronous_commit is on, which means that commits do not return until the WAL flush is done. If you turn it off, they'll return when the WAL flush is queued. You might lose transactions in the case of a crash, but there's no risk of database corruption.

Backup and Recovery

Experience has shown that 20% of the time, your EBS volumes will not reattach when you reboot in AWS.

pg_dump is a built-in dump/restore tool.

It takes a logical snapshot of the database.

It doesn't lock the database or prevent writes to disk.

pg_restore restores the database. It's not fast.

It's great for simple backups but not suitable for fast recovery from major failures.

pg_bench is the built in benchmarking tool.

pg_dump -Fc --verbose example > example.dump

Without the -Fc, it dumps SQL commands instead of its custom format.

pg_restore --dbname=example_restored --verbose example.dump

pg_restore takes a long time because it has to recreate indexes.

pg_dumpall --globals-only

Back up each database with pg_dump using --format=custom.

To do a parallel restore, use --jobs=.

If you have a large database, pg_dump may not be appropriate.

A disk snapshot + every WAL segment is enough to recreate the database.

To start a PITR (point in time recovery) backup:

select pg_start_backup(...);

Copy the disk image and any WAL files that are created.

select pg_stop_backup();

Make sure you have all the WAL segments.

The disk image + all the WAL segments are enough to create the DB.

See also github.com/wal-e/wal-e. It's highly recommended.

It automates backups to S3.

He explained how to do a PITR.

With PITR, you can rollback to a particular point in time. You don't have to replay everything.

This is super handy for application failures.

RDS is something that scripts all this stuff for you.

Replication

Send the WAL to another server.

Keep the server up to date with the primary server.

That's how PostgreSQL replication works.

The old way was called "WAL Archiving". Each 16MB segment was sent to the secondary when complete. Use rsync, WAL-E, etc., not scp.

The new way is Streaming Replication.

The secondary gets changes as they happen.

It's all setup via recovery.conf in your $PGDATA.

He showed a recovery.conf for a secondary machine, and showed how to let it become the master.

Always have a disaster recovery strategy.

pg_basebackup is a utility for doing a snapshot of a running server. It's the easiest way to take a snapshot to start a new secondary. It's also useful for archival backups. It's not the fastest thing, but it's pretty foolproof.

Replication:

The good:

Easy to setup.

Schema changes are replicated.

Secondaries can handle read-only queries for load balancing.

It either works or it complains loudly.

The bad:

You get the entire DB cluster or none of it.

No writes of any kind to the secondary, not even temporary tables.

Some things aren't replicated like temporary tables and unlogged tables.

His advice is to start with WAL-E. The README tells you everything. It fixes a ton of problems.

The biggest problem with WAL-E is that writing to S3 can be slow.

Another way to do funky things is trigger-based replication. There's a bunch of third-party packages to do this.

Bucardo is one that lets you do multi-master setups.

However, they're fiddly and complex to set up. They can also fail quietly.

Transactions, MVCC, and Vacuum

BEGIN;
INSERT ...;
INSERT ...;
COMMIT;

By the way, no bank works this way ;)

Everything runs inside of a transaction.

If there is no explicit transaction, each statement is wrapped in one for you.

Everything that modifies the database is transactional, even schema changes.

\d shows you all your tables.

With a transaction, you can even rollback a table drop.

South (the Django migration tool) runs the whole migration in a single transaction.

Many resources are held until the end of a transaction. Keep your transactions brief and to the point.

Beware of "IDLE IN TRANSACTION" sessions. This is a problem for Django apps.

A tuple in Postgres is the same thing as a row.

Postgres uses Multi-Version Concurrency Control. Each transaction sees its own version of the database.

Writers only block writers to the same tuple. Nothing else causes blocking.

Postgres will not allow two snapshots to "fork" the database. If two people try to write to the same tuple, Postgres will block one of them.

There are higher isolation modes. His description of them was really interesting.

He suggested that new apps use SERIALIZABLE. This will help you find the concurrency errors in your app.

Deleted tuples are not usually immediately freed.

Vacuum's primary job is to scavenge tuples that are no longer visible to any transaction.

autovacuum generally handles this problem for you without intervention (since version 8).

Run analyze after a major database change to help the planner out.

If someone tells you "vacuum's not working", they're probably wrong.

The DB generally stabilizes at 20% to 50% bloat. That's acceptable.

The problem might be that there are long-running transactions or idle-in-transaction sessions. They'll block vacuuming. So will manual table locking.

He talked about vacuum issues for rare situations.

Schema Design

Normalization is important, but don't obsess about it.

Pick "entities". Make sure that no entity-level info gets pushed into the subsidiary items.

Pick a naming scheme and stick with it.

Plural or singular? DB people tend to like plural. ORMs tend to like singular.

You probably want lower_case to avoid quoting.

Calculated denormalization can sometimes be useful; copied denormalization is almost never useful.

Joins are good.

PostgreSQL executes joins very efficiently. Don't be afraid of them.

Don't worry about large tables joined with small tables.

Use the typing system. It has a rich set of types.

Use domains to create custom types.

A domain is a core type + a constraint.

Don't use polymorphic fields (fields whose interpretation is dependent on another field).

Don't use strings to store multiple types.

Use constraints. They're cheap and fast.

You can create constraints across multiple columns.

Avoid Entity-Attribute-Value schemas. They cause great pain. They're very inefficient. They make reports very difficult.

Consider using UUIDs instead of serials as synthetic keys.

The problem with serials for keys is that merging tables can be hard.

Don't have "Thing" tables like "Object" tables.

If a table has a few frequently-updated fields and a few slowly-updated fields, consider splitting the table. Split the fast-moving stuff out into a separate 1-to-1 table.

Arrays are a first-class type in PostgreSQL. It's a good substitute for using a subsidiary table.

A list of tags is a good fit for arrays.

He talked about hstore. It's much better than Entity-Attribute-Value. It's great for optional, variable attributes. It's like a hash. It can be indexed, searched, etc. It lets you add attributes to tables for users. Don't use it as a way to avoid all table modifications.

json is now a built in type.

There's also jsonb.

Avoid indexes on big things, like 10k character strings.

NULL it a total pain in the neck.

Only use it to mean "missing value".

Never use it to represent a meaningful value.

Let's call anything 1MB or more a "very large object". Store them in files. Store the metadata in the database. The database API is just not a good fit for this.

Many-to-many tables can get extremely large. Consider replacing them with array fields (either one way or both directions). You can use a trigger to maintain integrity.

You don't want more than about 250k entries in an array.

Use UTF-8. Period.

Always use TIMESTAMPTZ (which Django uses by default). Don't use TIMESTAMP. TIMESTAMPTZ is a timestamp converted to UTC.

Index types:

B-Tree

Use a B-Tree on a column if you frequently query on that column,
use one of the comparison operators, only get back 10-15% of the rows,
and run that query frequently.

It won't use the index if you're going to get back more than 15% of
the rows because it's faster to scan a table then scan an index.

Use a partial index if you can ignore most of the rows.

The entire tuple has to be copied into the index.

GiST

It's a framework to create indexes.

KNN indexes are the K-nearest neighbors.

GIN

Generalized inverted index. Used for full-text search.

The others either are not good or very specific.

Why isn't it using my index?

Use explain analyze to look at the query.

If it thinks it's going to require most of the rows, it'll do a table scan.

If it's wrong, use analyze to update the planner stats.

Sometimes, it can't use the index.

Two ways to create an index:

create index

create index concurrently

reindex rebuilds an index from scratch.

pg_stat_user_indexes tells you about how your indexes are being used.

What do you do if a query is slow:

Use explain or explain analyze.

explain doesn't actually run the query.

"Cost" is measured in arbitrary units. Traditionally, they have been "disk fetches". Costs are inclusive of subnodes.

I think explain analyze actually runs the query.

Things that are bad:

Joins between 2 large tables.

Cross joins (cartesian products). These often happen by accident.

Sequential scans on large tables.

select count(*) is slow because it results in a full table scan since you
have to see if the tuples are alive or dead.

offset / limit. These actually run the query and then throw away that many
rows. Beware that GoogleBot is relentless. Use other keys.

If the database is slow:

Look at pg_stat_activity:

select * from pg_stat_activity;

tail -f the logs.

Too much I/O? iostat 5.

If the database isn't responding:

Try connecting with it using psql.

pg_stat_activity

pg_locks

Python Particulars

psycopg2 is the only real option in Python 2.

The result set of a query is loaded into client memory when the query completes. If there are a ton of rows, you could run out of memory. If you want to scroll through the results, use a "named" cursor. Be sure to dispose of it properly.

The Python 3 situation is not so great. There's py-postgresql. It's pure Python.

If you are using Django 1.6+, use the @atomic decorator.

Cluster all your writes into small transactions. Leave read operations outside.

Do all your writes at the very end of the view function.

Multi-database works very nicely with hot standby.

Point the writes at the primary, and the reads at the secondary.

For Django 1.5, use the @xact decorator.

Sloppy transaction management can cause the dreaded Django idle-in-transaction problem.

Use South for database migration. South is getting merged into Django in version 1.7 of Django.

You can use manual migrations for stuff the Django ORM can't specify.

Special Situations

Upgrade to 9.3.4. Upgrade minor versions promptly.

Major version upgrades require more planning. pg_upgrade has to be run when the database is not running.

A full pg_dump / pg_restore is always the safest, although not the most practical.

Always read the release notes.

All parts of a replication set must be upgraded at once (for major versions).

Use copy, not insert, for bulk loading data. psycopg2 has a nice interface. Do a vacuum afterwards.

AWS

Instances can disappear and come back up without instance storage.

EBS can fail to reattach after reboot.

PIOPS are useful (but pricey) if you are using EBS.

Script everything, instance creation, PostgreSQL, etc. Use Salt. Use a VPC.

Scale up and down as required to meet load. If you're just using them to rent a server, it's really expensive.

PostgreSQL RDS is a managed database instance. Big plus: automatic failover! Big minus: you can't read from the secondary. It's expensive. It's a good place to start.

Sharding

Eventually, you'll run out of write capacity on your master.

postgres-xc is an open source fork of PostgreSQL.

Bucardo provides multi-master write capability.

He talked about custom sharding.

Instagram wrote a nice article about it.

Pooling

Opening a connection is expensive. Use a pooler.

pgbouncer is a pooler.

pgPool II can even do query analysis. However, it has higher overhead and is more complex to configure.

Tools

Monitor everything.

check_postgres.pl is a plugin to monitor PostgreSQL.

pgAdmin III and Navicat are nice clients.

pgbadger is for log analysis. So is pg_stat_statements.

Closing

MVCC works by each tuple having a range of transaction IDs that can see that
tuple.

Failover is annoying to do in the real world. People use HAProxy, some pooler, etc. with some scripting, or they have a human do the failover.

HandyRep is a server-based tool designed to allow you to manage a PostgreSQL "replication cluster", defined as a master and one or more replicas on the same network.

Shannon -jj Behrens: JJ's Mostly Adequate Summary of the Django Meetup: When *Not* To Use the ORM & Goodbye REST: Building GraphQL APIs with Django

$
0
0

The Django meetup was at Prezi. They have a great space. They are big Django users.

Goodbye REST: Building APIs with Django and GraphQL

Jaden Windle, @jaydenwindle, lead engineer at Jetpack.

https://github.com/jaydenwindle/goodbye-rest-talk

They moved from Django REST Framework to GraphQL.

It sounds like a small app.

They're using Django, React, and React Native.

I think he said they used Reason and moved away from it, but I could be wrong.

They had to do a rebuild anyway, so they used GraphQL.

He said not to switch to GraphQL just because it's cool, and in general, avoid hype-driven development.

GraphQL is a query language, spec, and collection of tools, designed to operate over a single endpoint via HTTP, optimzing for perf and flexibility.

Key features:

  • Query for only the data you need.
  • Easily query for multiple resources in a single request.
  • Great front end tooling for handling caching, loading / error states, and updates.

(I wonder if he's talking about Apollo, not just GraphQL. I found out later, that they are using Apollo.)

GraphQL Schema:

  • Types
  • Queries
  • Mutations
  • *Subscriptions

Graphene is the goto framework for GraphQL in Django.

He showed an example app.

You create a type and connect it to a model. It's like a serializer in DRF.

(He's using VS Code.)

It knows how to understand relationships between types based on the relationships in the models.

He showed the query syntax.

He showed how Graphene connects to the Django model. You're returning raw Django model objects, and it takes care of serialization.

There's a really nice UI where you can type in a query, and it queries the server. It has autocomplete. I can't tell if this is from Apollo, Graphene, or some other GraphQL tool.

You only pass what you need across the wire.

When you do a mutation, you can also specify what data it should give back to you.

There is some Mutation class that you subclass.

The code looks a lot like DRF.

Subscriptions aren't fully implemented in Graphene. His company is working on implementing and open sourcing something. There are a bunch of other possible real-time options--http://graphql-python/graphql-ws is one.

There's a way to do file uploads.

Important: There's this thing called graphene-django-extras. There's even something to connect to DRF automatically.

Pros:

  • Dramatically improves front end DX
  • Flexible types allow for quick iteration
  • Always up-to-date documentation
  • Only send needed data over the wire

Cons:

  • Graphene isn't as mature as some other GraphQL implementations (for instance, in JS and Ruby)
  • Logging is different when using a single GraphQL endpoint
  • REST is currently better at server-side caching (E-Tag, etc.)

Graphene 3 is coming.

In the Q&A, they said they do use Apollo.

They're not yet at a scale where they have to worry about performance.

He's not entirely sure whether it's prone to the N+1 queries problem, but there are GitHub issues related to that.

You can do raw ORM or SQL queries if you need to. Otherwise, he's not sure what it's doing behind the scenes.

You can add permissions to the models. There's also a way to tie into Django's auth model.

Their API isn't a public API. It's only for use by their own client.

The ORM, and When Not To Use It

Christophe Pettus from PostgreSQL Experts.

He thinks the ORM is great.

The first ORM he wrote was written before a lot of the audience was born.

Sometimes, writing code for the ORM is hard.

Database agnosticism isn't as important as you think it is. For instance, you don't make the paint on your house color-agnostic.

Libraries have to be DB agnostic. Your app probably doesn't need to be.

Reasons you might want to avoid the query language:

  • Queries that generate sub-optimal SQL
  • Queries more easily expressed in SQL
  • SQL features not available via the ORM

Django's ORM's SQL is much better than it used to be.

Don't use __in with very large lists. 100 is about the longest list you should use.

Avoid very deep joins.

It's not hard to chain together a bunch of stuff that ends up generating SQL that's horrible.

The query generator doesn't do a separate optimization pass that makes the query better.

It's better to express .filter().exclude().filter() in SQL.

There are so many powerful functions and operators in PostgreSQL!

SomeModel.objects.raw() is your friend!

You can write raw SQL, and yet still have it integrate with Django models.

You can even get stuff back from the database that isn't in the model definition.

There's some WITH RECURSIVE thing in PostgreSQL that would be prohibitively hard to do with the Django ORM. It's not really recursive--it's iterative.

You can also do queries without using the model framework.

The model framework is very powerful, but it's not cheap.

Interesting: The data has to be converted into Python data and then again into model data. If you're just going to serialize it into JSON, why create the model objects? You can even create the JSON directly in the database and hand it back directly with PostgreSQL. But make sure the database driver doesn't convert the JSON back to Python ;) Return it as raw text.

There are also tables that Django can't treat as model tables. For instance, there are logging tables that lack a primary key. Sometimes, you have weird tables with non-Django-able primary keys.

The ORM is great, though. For instance, it's great for basic CRUD.

Interfaces that require building queries in steps are better done with SQL--for instance, an advanced search function.

Summary:

  • Don't be afraid to step outside the ORM.
  • SQL isn't a bug. It's a feature. It's code like everything else.
  • Do use the ORM for operations that it makes easier.
  • Don't hesitate to use the full power of SQL.

Q&A:

Whether you put your SQL in model methods or managers is a matter of taste. Having all the code for a particular model in one place (i.e. the model or manager) is useful.

Write tests. Use CI.

Use parameter substitution in order to avoid SQL injection attacks. Remember, all external input is hostile. You can't use parameter substitution if the table name is dynamic--just be careful with what data you allow.

If you're using PostGIS, write your own queries for sure. It's really easy to shoot yourself in the foot with the GIS package.

 

 

gamingdirectional: Create a game over scene for pygame project

$
0
0

In this article we are going to create a game over scene for the pygame project, we will use back the start scene class which we have created previously to render our game over scene by slightly modify it for the multi-scenes uses. Here is the modify version of the start scene class, as you can see we have passed in a state variable to the start scene class’s draw method which will be used...

Source

Talk Python to Me: #188 Async for the Pythonic web with Sanic

$
0
0
What do most web servers do most of the time? They wait. They wait on external systems while processing a request.

James Bennett: On degrees

$
0
0

Lately there’s been a recurring discussion on various social-media outlets about the relevance of academic degrees to a career in programming. Specifically: is a degree in computer science (or some other field perceived as related) a necessity for a career as a programmer? If not, is it still a good idea to have one?

I’ve jumped into a few such threads on Twitter, but I have complex thoughts on the topic, and 140-character or (now) 280-character ...

Read full entry


Davy Wybiral: Getting Started with GPS Modules | Tracking Device | Tutorial

$
0
0
Today I took a GPS module and a microcontroller for a walk in the park and wrote a few lines of code to show my GPS coordinates in real-time.

Davy Wybiral: Experimenting with Electric Paint

$
0
0
Just recently I got my hands on some electrically conductive paint and decided to put it to the test. It functions well for creating resistive elements like custom potentiometers, and also seems to work really well for capacitive touch elements. At the end of the video I painted a small patch of it on the side of my monitor to add a touch button that can be programmed using a small Python script (included in the video description).

Anyway, here it is:

Yasoob Khalid: Email Security & Privacy

$
0
0

Hi everyone! I hope all of you are doing well. Things have been super exciting on my side. Just got done with a file system checker project in C for my Operating Systems class. It was mentally draining but equally rewarding. This blog post is not about Python but rather about Emails.

This week I also gave a department wide talk at Colgate University on Email security and privacy and some of the SMTP features which are available at your disposal. It went fairly successful. This was a result of my own independent research on the topic. I tried to cover as much base as possible while also trying to keep the talk engaging.

If you want to learn more about the topic, you can take a look at the slides I used and do some research of your own. If something seems wrong please let me know in the comments below so that we all can benefit. You can also access (I think?) the speaker notes for more context behind the content on the slides.

If you are pressed for time, skip to the last slide to learn about some interesting attacks. One of them makes use of Cyrillic script. If you haven’t heard of Cyrillic script before you would love that slide.

I would love to hear from you all. Please share your views in the comments below about the talk or anything in general.

Weekly Python StackOverflow Report: (cliv) stackoverflow python report

$
0
0

Python Bytes: #106 Fluent query APIs on Python collections

gamingdirectional: Create a next level scene for pygame project

$
0
0

In this article we will create a next level scene which will be displayed whenever the player has gained 30 points. Just like the previous chapter we are going to use back the start scene class as well as modify the game manager class and the overlap class First of all we will create the next scene surface and draw it on the game scene whenever the state of the game has changed to the next level...

Source

PyBites: PyBites Twitter Digest - Issue 37, 2018

$
0
0

Digital Ocean Developer Trends of the quarter

Submitted by @BryanKimani

A great article on learning things in general

Submitted by @dgjustice

A conversation with Guido!

music21 - a Python-based toolkit for computer-aided musicology

Submitted by @Dan Shorstein

Shameless plug because we can! CodeChalleng.es can now help your company run coding interviews with Interview Bites!

Thanks to everyone that participated in the TalkPython Black Friday sale!

Multiple assignment and tuple unpacking improve Python code readability

Submitted by @tryexceptpass

Nice, TinyDB!

Too handy not to share. Print a list of reverse DNS records for a subnet

Check out Jake VanderPlas's Python Data Science Handbook

A great message: Document Everything!

Some AMAZING responses to the question: "What's the most fun thing you have ever done with data?"

Tracking all of your SQL Queries in Git

Serverless APIs with Python, AWS Lambda & API Gateway

So old...


>>>frompybitesimportBob,JulianKeepCalmandCodeinPython!

Vasudev Ram: Tower of Hanoi program in Higher-Order Perl book, ported to Python

$
0
0

Hi, readers,

I was reading the book Higher Order Perl (Wikipedia entry). It is by Mark Jason Dominus, a well-known figure in the Perl community. I've only read a bit of it so far, but it already looks very good. There are also many reviews of it, all of which say that it is a good book, including one by Damian Conway, another well-known Perl figure.

Early in the book, in chapter 1 - Recursion and Callbacks, there is a nice example program showing how to solve the Tower of Hanoi problem. I had come across the problem earlier, but had not seen a Perl solution before. It's not really any different from the same kind of solution written in other languages, except for some points related to Perl's syntax and semantics, such as the use of local (Perl's my) vs. global variables, specifically with regard to a recursive function, which is what Mark uses for Hanoi.

Anyway, since I had not implemented Hanoi before, I thought of porting Mark's hanoi function from Perl to Python. It's an easy recursive algorithm. (The Wikipedia article has multiple algorithms for Hanoi, some non-recursive too.)

I wrote two versions: the first is hard-coded to solve it for a tower with n equal to 3, where n is the number of disks, and the second does it in a loop for n ranging from 1 to 4, with a prompt and a pause after each one.

If you press q at any of the prompts, the program quits. If you press any other key, it goes on to the next iteration, until the last one. For reading a keypress without needing to press Enter, I used the getch function (get character) from the msvcrt module that comes with Python. MSVCRT stands for Microsoft Visual C Run Time.

So this program is Windows-specific. There are equivalent ways of reading a character without needing to press Enter, for Linux too, but they usually involve some low-level fiddling with the tty / terminal settings, termios, etc. I think ActiveState Code has a recipe for a getch-like function that works on Linux.

Here is the first version, hanoi_01.py:
from __future__ import print_function

"""
This is the code for the Perl version by Mark Jason Dominus:

sub hanoi {
my ($n, $start, $end, $extra) = @_;
if ($n == 1) {
print "Move disk #1 from $start to $end.\n"; # Step 1
} else {
hanoi($n-1, $start, $extra, $end); # Step 2
print "Move disk #$n from $start to $end.\n"; # Step 3
hanoi($n-1, $extra, $end, $start); # Step 4
}
}
"""

# This is my Python version ported from the Perl version above:

def hanoi(n, start, end, extra):
"""
Function hanoi(n, start, end, extra)
Solve Tower of Hanoi problem for a tower of n disks,
of which the largest is disk #n. Move the entire tower from
peg 'start' to peg 'end', using peg 'extra' as a work space.
"""
if n == 1:
print("Move disk #1 from {} to {}.\n".format(start, end))
else:
# Recursive call #1
hanoi(n - 1, start, extra, end)
print("Move disk #{} from {} to {}.\n".format(n, start, end))
# Recursive call #2
hanoi(n - 1, extra, end, start)

# Call to hanoi() with 3 disks, i.e. n = 3:
hanoi(3, 'A', 'C', 'B')
Here is the output on running it:
$ python hanoi_01.py
Move disk #1 from A to C.

Move disk #2 from A to B.

Move disk #1 from C to B.

Move disk #3 from A to C.

Move disk #1 from B to A.

Move disk #2 from B to C.

Move disk #1 from A to C.
Here is the second version, hanoi_02.py:
from __future__ import print_function
from msvcrt import getch

"""
Tower of Hanoi program
---
By Vasudev Ram
Web site: https://vasudevram.github.io
Blog: https://jugad2.blogspot.com
Twitter: https://mobile.twitter.com/vasudevram
Product store: https://gumroad.com/vasudevram
---
Translated to Python from the Perl version in the book Higher-Order Perl
by Mark Jason Dominus: https://hop.perl.plover.com/book/ , with some
minor enhancements related to interactivity.
"""

def hanoi(n, start, end, extra):
"""
Function hanoi(N, start, end, extra)
Solve Tower of Hanoi problem for a tower of N disks,
of which the largest is disk #N. Move the entire tower from
peg 'start' to peg 'end', using peg 'extra' as a work space.
"""
assert n > 0
if n == 1:
print("Move disk #1 from {} to {}.".format(start, end))
else:
hanoi(n - 1, start, extra, end)
print("Move disk #{} from {} to {}.".format(n, start, end))
hanoi(n - 1, extra, end, start)

for n in range(1, 5):
print("\nTower of Hanoi with n = {}:".format(n))
hanoi(n, 'A', 'C', 'B')
if n < 4:
print("\nPress q to quit, any other key for next run: ")
c = getch()
if c.lower() == 'q':
break

Here is the output on running it - I pressed q after 3 iterations:
$ python hanoi_02.py

Tower of Hanoi with n = 1:
Move disk #1 from A to C.

Press q to quit, any other key for next run:

Tower of Hanoi with n = 2:
Move disk #1 from A to B.
Move disk #2 from A to C.
Move disk #1 from B to C.

Press q to quit, any other key for next run:

Tower of Hanoi with n = 3:
Move disk #1 from A to C.
Move disk #2 from A to B.
Move disk #1 from C to B.
Move disk #3 from A to C.
Move disk #1 from B to A.
Move disk #2 from B to C.
Move disk #1 from A to C.

Press q to quit, any other key for next run:
Note that getch does not echo the key that is pressed to the screen. If you want that, use getche, from the same module msvcrt.

Viewing the output, I observed that for 1 disk, it takes 1 move, for 2 disks, it takes 3 moves, and for 3 disks, it takes 7 moves. From this, by informally applying mathematical induction, we can easily figure out that for n disks, it will take 2 ** n - 1 moves. The Wikipedia article about the Tower of Hanoi confirms this.

That same Wikipedia article (linked above) also shows that the Tower of Hanoi has some interesting applications:

Excerpts:

[ The Tower of Hanoi is frequently used in psychological research on problem solving. There also exists a variant of this task called Tower of London for neuropsychological diagnosis and treatment of executive functions. ]

[ Zhang and Norman[10] used several isomorphic (equivalent) representations of the game to study the impact of representational effect in task design. ]

[ The Tower of Hanoi is also used as a Backup rotation scheme when performing computer data Backups where multiple tapes/media are involved. ]

[ The Tower of Hanoi is popular for teaching recursive algorithms to beginning programming students. ]

And this application is possibly the most interesting of the lot - ants solving the Tower of Hanoi problem :)

[ In 2010, researchers published the results of an experiment that found that the ant species Linepithema humile were successfully able to solve the 3-disk version of the Tower of Hanoi problem through non-linear dynamics and pheromone signals. ]

Also check out the section in that Wikipedia article with the title:

"General shortest paths and the number 466/885"

Not directly related, but similar to the above ratio, did you know that there is an easy-to-remember and better approximation to the number pi, than the ratio 22/7 that many of us learned in school?

Just take the 1st 3 odd numbers, i.e. 1, 3, 5, repeat each of them once, and join all those 6 digits, to get 113355, split that number in the middle, and divide the 2nd half by the 1st half, i.e 355/113, which gives 3.1415929203539823008849557522124. (See the section titled "Approximate value and digits" in the Wikipedia article about pi linked above, for this and other approximations for pi).

Pressing the pi button in Windows CALC.EXE gives 3.1415926535897932384626433832795, and comparing that with the ratio 355/113 above, shows that the ratio is accurate to 6 decimal places. I've read somewhere that this ratio was used by mathematicians in ancient India.

Here are some interesting excerpts from the Wikipedia article about pi:

[ The number pi is a mathematical constant. Originally defined as the ratio of a circle's circumference to its diameter, it now has various equivalent definitions and appears in many formulas in all areas of mathematics and physics. It is approximately equal to 3.14159. It has been represented by the Greek letter "p" since the mid-18th century, though it is also sometimes spelled out as "pi". It is also called Archimedes' constant. ]

[ Being an irrational number, pi cannot be expressed as a common fraction (equivalently, its decimal representation never ends and never settles into a permanently repeating pattern). Still, fractions such as 22/7 and other rational numbers are commonly used to approximate pi. The digits appear to be randomly distributed. In particular, the digit sequence of pi is conjectured to satisfy a specific kind of statistical randomness, but to date, no proof of this has been discovered. Also, pi is a transcendental number; that is, it is not the root of any polynomial having rational coefficients. ]

[ Being an irrational number, pi cannot be expressed as a common fraction (equivalently, its decimal representation never ends and never settles into a permanently repeating pattern). Still, fractions such as 22/7 and other rational numbers are commonly used to approximate p. The digits appear to be randomly distributed. In particular, the digit sequence of pi is conjectured to satisfy a specific kind of statistical randomness, but to date, no proof of this has been discovered. Also, pi is a transcendental number; that is, it is not the root of any polynomial having rational coefficients. This transcendence of pi implies that it is impossible to solve the ancient challenge of squaring the circle with a compass and straightedge. ]

[ Ancient civilizations required fairly accurate computed values to approximate pi for practical reasons, including the Egyptians and Babylonians. Around 250 BC the Greek mathematician Archimedes created an algorithm for calculating it. In the 5th century AD Chinese mathematics approximated pi to seven digits, while Indian mathematics made a five-digit approximation, both using geometrical techniques. The historically first exact formula for p, based on infinite series, was not available until a millennium later, when in the 14th century the Madhava–Leibniz series was discovered in Indian mathematics. ]

(Speaking of Indian mathematics, check out this earlier post by me:

Bhaskaracharya and the man who found zero.)

[ Because its most elementary definition relates to the circle, pi is found in many formulae in trigonometry and geometry, especially those concerning circles, ellipses, and spheres. In more modern mathematical analysis, the number is instead defined using the spectral properties of the real number system, as an eigenvalue or a period, without any reference to geometry.

It appears therefore in areas of mathematics and the sciences having little to do with the geometry of circles, such as number theory and statistics, as well as in almost all areas of physics.

The ubiquity of pi makes it one of the most widely known mathematical constants both inside and outside the scientific community. Several books devoted to pi have been published, and record-setting calculations of the digits of pi often result in news headlines. Attempts to memorize the value of pi with increasing precision have led to records of over 70,000 digits. ]

- Enjoy.


- Vasudev Ram - Online Python training and consulting


I conduct online courses on Python programming, Unix / Linux commands and shell scripting and SQL programming and database design, with course material and personal coaching sessions.

The course details and testimonials are here.

Contact me for details of course content, terms and schedule.

Try FreshBooks: Create and send professional looking invoices in less than 30 seconds.

Getting a new web site or blog, and want to help preserve the environment at the same time? Check out GreenGeeks.com web hosting.

Sell your digital products via DPD: Digital Publishing for Ebooks and Downloads.

Learning Linux? Hit the ground running with my vi quickstart tutorial.

I wrote it at the request of two Windows system administrator friends who were given additional charge of some Unix systems. They later told me that it helped them to quickly start using vi to edit text files on Unix. Of course, vi/vim is one of the most ubiquitous text editors around, and works on most other common operating systems and on some uncommon ones too, so the knowledge of how to use it will carry over to those systems too.

Check out WP Engine, powerful WordPress hosting.

Sell More Digital Products With SendOwl.

Get a fast web site with A2 Hosting.

Creating online products for sale? Check out ConvertKit, email marketing for online creators.

Teachable: feature-packed course creation platform, with unlimited video, courses and students.

Posts about: Python * DLang * xtopdf

My ActiveState Code recipes

Follow me on:


Podcast.__init__: Keeping Up With The Python Community For Fun And Profit with Dan Bader

$
0
0
Keeping up with the work being done in the Python community can be a full time job, which is why Dan Bader has made it his! In this episode he discusses how he went from working as a software engineer, to offering training, to now managing both the Real Python and PyCoders properties. He also explains his strategies for tracking and curating the content that he produces and discovers, how he thinks about building products, and what he has learned in the process of running his businesses.

Summary

Keeping up with the work being done in the Python community can be a full time job, which is why Dan Bader has made it his! In this episode he discusses how he went from working as a software engineer, to offering training, to now managing both the Real Python and PyCoders properties. He also explains his strategies for tracking and curating the content that he produces and discovers, how he thinks about building products, and what he has learned in the process of running his businesses.

Preface

  • Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great.
  • When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so check out Linode. With 200 Gbit/s private networking, scalable shared block storage, node balancers, and a 40 Gbit/s public network, all controlled by a brand new API you’ve got everything you need to scale up. Go to podcastinit.com/linode to get a $20 credit and launch a new server in under a minute.
  • Visit the site to subscribe to the show, sign up for the newsletter, and read the show notes. And if you have any questions, comments, or suggestions I would love to hear them. You can reach me on Twitter at @Podcast__init__ or email hosts@podcastinit.com)
  • To help other people find the show please leave a review on iTunes, or Google Play Music, tell your friends and co-workers, and share it on social media.
  • Join the community in the new Zulip chat workspace at podcastinit.com/chat
  • Your host as usual is Tobias Macey and today I’m interviewing Dan Bader about finding, filtering, and creating resources for Python developers at Real Python, PyCoders, and his own trainings

Interview

  • Introductions
  • How did you get introduced to Python?
  • Let’s start by discussing your primary job these days and how you got to where you are.
    • In the past year you have also taken over management of the Real Python site. How did that come about and what are your responsibilities?
    • You just recently took over management of the PyCoders newsletter and website. Can you describe the events that led to that outcome and the responsibilities that came along with it?
  • What are the synergies that exist between your various roles and projects?
    • What are the areas of conflict? (e.g. time constraints, conflicts of interest, etc.)
  • Between PyCoders, Real Python, your training materials, your Python tips newsletter, and your coaching you have a lot of incentive to keep up to date with everything happening in the Python ecosystem. What are your strategies for content discovery?
    • With the diversity in use cases, geography, and contributors to the landscape of Python how do you work to counteract any bias or blindspots in your work?
  • There is a constant stream of information about any number of topics and subtopics that involve the Python language and community. What is your process for filtering and curating the resources that are ultimately included in the various media properties that you oversee?
  • In my experience with the podcast one of the most difficult aspects of maintaining relevance as a content creator is obtaining feedback from your audience. What do you do to foster engagement and facilitate conversations around the work that you do?
  • You have also built a few different product offerings. Can you discuss the process involved in identifying the relevant opportunities and the creation and marketing of them?
  • Creating, collecting, and curating content takes a significant investment of time and energy. What are your avenues for ensuring the sustainability of your various projects?
  • What are your plans for the future growth and development of your media empire?
  • As someone who is so deeply involved in the conversations flowing through and around Python, what do you see as being the greatest threats and opportunities for the language and its community?

Keep In Touch

Picks

Links

The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA

gamingdirectional: Create a win scene and the level manager class for pygame project

$
0
0

In this article we will create a pop up win scene when the player has won all the levels which it will then ask the player whether he wants to start the game all over again or not? We will also create a framework for the level manager class which we will further add in more features in the next chapter. First of all, we will modify the start scene class again to include in a win scene graphic.

Source

Techiediaries - Django: CSS Grid Layout Tutorial—Styling a Django Template

$
0
0

Throughout this tutorial, we'll learn about CSS Grid Layout. We'll be using a Django template.

There are many popular techniques for creating responsive layouts. In this tutorial, we'll be building a simple Django web application with a modern UI styled with CSS Grid layout.

By building the UI using CSS Grid, you'll learn about many useful and easy techniques for achieving popular requirements such as centering and spanning items, switching layouts depending on the screen size and creating responsive UIs.

Before, we dive into practical steps, let's first introduce CSS Grid.

What is CSS Grid?

CSS Grid is a modern 2-dimentionnal system for creating HTML layouts. It's now supported by most web browsers. It makes creating professional and complex layouts more easier than ever!

CSS Grid Layout allows you to build advanced grid layouts in CSS instead of HTML like the case for tables for example.

Unlike CSS floats and HTML tables you can create a grid layout in a straightforward way. You simply need to use an HTML element with its display property set to grid or inline-grid. This way, any elements contained in the parent element or the container becomes grid items. If you don't specify any other CSS Grid property, the items will be automatically arranged in a grid structure thanks to a powerful grid algorithm.

If you need more control over the grid items, you can use different CSS Grid properties to specify the different options for the items including position.

Prerequisites

If you want to follow this tutorial, step by step to build a CSS Grid UI for your Django application. You will need a few requirements. You need to have;

  • A development environment ready with Python 3 and PIP installed,
  • Basic knowledge of Python,
  • Working knowledge of Django.

Creating a Virtual Environment & Installing Django

Now let's start by creating a virtual environment that will allow you to isolate your project's dependencies from the other Python dependencies installed on your system.

Go to your terminal and run the following command to create a virtual environment using the venv module included in Python 3.

$ cd ~
$ python3 -m venv env

Next, you need to activate this environment using:

$ source env/bin/activate

Now, you can install Django using pip:

$ pip install django

Creating a Django Project

After installing Django, you now need to create a project. Head back to your terminal and run the following command:

$ django-admin start css
$ django-admin.py startproject cssgriddemo

This command will generate a project named cssgriddemo.

Creating a Django Application

Now that you have created a Django project, you need to create a Django application. Head over to your terminal and run the following command to create the app:

$ python manage.py startapp gridui 

Next, add it to the list of installed apps in the settings.py file:

INSTALLED_APPS=[# [...]'gridui']

Next, we need to add a template and a view function.

Open the gridui/views.py file and add the following import:

fromdjango.views.generic.baseimportTemplateView```Nextaddthefollowingviewclass:```pythonclassHome(TempateView):template_name='gridui/index.html'

Next, add the gridui/index.html template with the following code:

<!DOCTYPE html><htmllang="en"><head><metacharset="utf-8"><title>Django UI with CSS Grid Layout</title></head><body><divclass="header"><h1>Django UI with CSS Grid Layout</h1></div><divclass="sidebar">
.sidebar
</div><divclass="main"><divclass="item"><span>1</span></div><divclass="item"><span>2</span></div><divclass="item"><span>...</span></div><divclass="item"><span>12</span></div></div><divclass="footer">
Copyright 2018
</div></body></html>

We create an HTML template with the header, sidebar, main and footer sections. In the main section, we include a set of items.

Finally, let's add an URL to our urls.py file

fromdjango.urlsimportpathfromgriduiimportviewsurlpatterns=[# [...]path('',views.Home.as_view())]

You can now migrate your database and run the Django server:

$ python manage.py migrate
$ python manage.py runserver

You application will be running from localhost:8000.

This is a screenshot of our interface at this point:

CSS Grid

Next, let's add basic CSS styling. Add a <style> tag and add the following styles:

<style>body{background:#478dee;margin:5px;padding:0px;font-family:-apple-system,BlinkMacSystemFont,SegoeUI,Roboto,Oxygen,Ubuntu,Cantarell,FiraSans,DroidSans,HelveticaNeue,sans-serif;}div.header{text-transform:uppercase;}</style>

This is a screenshot of the interface:

CSS Grid

Adding CSS Grid

Now that we have created a simple HTML structure, we can proceed by styling the UI using CSS Grid.

First we need to make the <body> element a grid container and define the grid columns, rows and areas:

body{//[...]display:grid;grid-gap:0.2vw;height:100vh;grid-template-columns:150px1fr;grid-template-rows:120px1fr61px;grid-template-areas:"header   header""sidebar content""footer footer";}

We use display:grid pair to make <body> a grid container. We also set a grid gap of _0.2vw.

We make use of grid-template-columns to define two columns, the first column has a fixed width of 150px and the second column has the remaining width.

fr is a fractional unit and 1fr means the element should take 1 part of the available space.

Next, we use grid-template-rows to define three rows:

  • The first row has a fixed height of 120px ,
  • The third row has a fixed height of 61px,
  • The second row has the remaining space (1fr).

We finally use grid-template-areas for assigning the virtual cells, resulted from the intersection of columns and rows, to areas.

Now we need to define those areas specified in the areas template using grid-area:

div.header{grid-area:header;}div.sidebar{grid-area:sidebar;background:#0769f3;}div.footer{grid-area:footer;background:#3581eb;}div.main{grid-area:content;}

This is the screen shot of the result now:

CSS Grid

Adding a Nested CSS Grid

Now that we have created the main layout of the page, let's use CSS Grid to create a nested grid in the content area.

Grid children can also be Grid containers.

Let's define the content area as a grid container. In the <style> tag, add the following CSS:

div.main{display:grid;grid-gap:0.2vw;grid-template-columns:repeat(auto-fill,minmax(150px,1fr));grid-template-rows:repeat(auto-fill,minmax(150px,1fr));}

Here we use repeat, auto-fill and minmax to define columns and rows.

Also add the following style to use an image background for items:

div.item{background-image:url('https://source.unsplash.com/daily');}

This is the screen shot of the result after adding more <div class="item"> elements: :

CSS Grid

CSS Grid grid-column, grid-row and span

CSS Grid enables you to control the position of grid elements using grid-column and grid-row.

You can also use span to define how much columns or rows a specified element takes.

Now back to our project. Let's span the second item of the content area across five columns and two rows and position it from column line 2 and row line 1 (default location).

div.main>div.item:nth-child(2){grid-column:2/span5;grid-row:1/span2;}

This is a screen shot of the result:

CSS Grid

Conclusion

In this tutorial you've seen how to create a beautiful UI for a Django application using CSS Grid Layout.

Mike Driscoll: PyDev of the Week: Erika Fille Legara

$
0
0

This week we welcome Erika Fille Legara (@eflegara) as our PyDev of the Week. Erika is a professor and program director at the Asian Institute of Management. She has spoken at PyCon Philippines. You can check out her website to see what else she has been up to or watch her talk below:

Let’s take a few moments to get to know her better!

Can you tell us a little about yourself (hobbies, education, etc):

Hi, I’m Erika. I am a physicist by training. I am your typical grad school (assoc.) professor/administrator who’s always trying her best to strike the appropriate balance between teaching, research, and fulfilling certain administrative duties. At the moment, the research projects I am engage in are mostly industry-driven.

Outside work, I enjoy traveling and travel photography. With the recent career transition, however, leisure travels have been minimized. Nowadays, I spend most of my free time reading, listening to music, and yes, binge-watching. I also love highway driving, every now and then, on weekends; it helps the mind relax. I like the fact that in long drives I get to listen to awesome road trip playlists without interruption.

Why did you start using Python?

I started writing scripts in C++ for my undergrad research. My thesis was on the study of complex systems; network science, in particular. One of my colleagues then at the lab introduced me to Python when he saw me writing really, really long scripts in C++ to build complex network models. He showed me how I can reduce the 50 or so (or longer) lines of code I wrote in C++ to only a few lines, less than 10 actually, in Python, thanks to all python scientific libraries developers and contributors. Since then, I’ve never looked back.

What other programming languages do you know and which is your favorite?

Other languages are C/C++, R, and MATLAB. I am also familiar with PHP.

But my one and only favorite is Python, of course.

What projects are you working on now?

I am currently working on a couple of research projects involving urban systems modeling, transport systems modeling, analysis and modeling of the spread of (mis/dis)information in social systems, and building predictive models for location analysis.

Which Python libraries are your favorite (core or 3rd party)?

NumPy and SciPy, Matplotlib, PANDAS, scikit-learn, NetworkX, graph-tool, Keras, TensorFlow, H2Oai

Is there anything else you’d like to say?

Never stop learning! The world, especially with today’s technologies, has so much to offer.

Thanks for doing the interview, Erika!

Viewing all 24370 articles
Browse latest View live


<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>