DataCamp: IPython Or Jupyter?

Sage Notebook

The Sage notebook as a browser-based system was first released mid-2000s and then in 2007, a new version was released that was more powerful, had user accounts and could be used to make documents public. It resembled the Google docs UI design since the layout of the Sage notebook was based on the layout of Google notebooks.

The creators of the Sage notebooks have confirmed that they were avid users of the Mathematica notebooks and Maple worksheets. Other motivations or drivers that were important when you consider the development of the Sage notebooks were the facts that the developers had close contact with the team behind IPython, they had experienced failed attempts at GUIs for IPython, and the rise of “AJAX” = web applications, which didn’t require users to refresh the whole page every time you do something.

IPython and Jupyter Notebook

In late 2001, twenty years after Guido van Rossum began to work on Python at the National Research Institute for Mathematics and Computer Science in the Netherlands, Fernando Pérez starts developing IPython. The project was heavily influenced by the Mathematica notebooks and Maple worksheets, just like the Sage notebook and many other projects that followed.

In 2005, both Robert Kern and Fernando Pérez attempted building a notebook system. Unfortunately, the prototype had never become fully usable.

Fast forward two years: the team had kept on working, and in 2007, they formulated another attempt at implementing a notebook-type system. By October 2010, there was a prototype of a web notebook and in the summer of 2011, this prototype was incorporated and it was released with 0.12 on December 21, 2011. In subsequent years, the team got awards, such as the Advancement of Free Software for Fernando Pérez on 23 of March 2013 and the Jolt Productivity Award, and funding from the Alfred P. Sloan Foundations, among others.

Lastly, in 2014, Project Jupyter started as a spin-off project from IPython.

The last release of IPython before the split contained an interactive shell, the notebook server, the Qt console, etc. The project was big, with tools that were increasingly becoming more and more distinct projects that happened to pertain to the same project. After the Jupyter project started, the language-agnostic parts of the IPython project, such as the notebook format, message protocol, Qt console, notebook web application, etc. were put into the Jupyter project.

This is called “The Big Split”.

IPython has now only two roles to fulfill: being the Python backend to the Jupyter Notebook, which is also known as the kernel, and an interactive Python shell. But this is not all: within the IPython ecosystem, you’ll also find a parallel computing framework. You’ll read more about this later on!

And just like IPython, Project Jupyter is actually one name for a bunch of projects: the three applications that it harbors are the Notebook itself, a Console and a Qt console, but there are also subprojects such as Jupyterhub to support notebook deployment, nbgrader for educational purposes, etc. You can see an overview of the Jupyter architecture here.

Note that it’s exactly the evolution of this project that explains the confusion that many Pythonistas have when it comes to IPython and Jupyter: since one came out of the other (quite recently), there are some who still have difficulties adopting the right names for the concepts. But what might be the even more complicating factor is the evolution: since one came out of the other, there is a considerable overlap between the IPython and the Jupyter Notebook features that are sometimes hard to distinguish!

How to distinguish between the two will become clear in the next sections of this post.

If you want to know more details about how the development of IPython came about, check out the personal accounts of Fernando Pérez and William Stein about the history of their notebooks.

R Notebooks

R Markdown and Jupyter notebooks share the delivering of a reproducible workflow, the weaving of code, output, and text together in a single document, supporting interactive widgets and outputting to multiple formats.

However, the two also differ: the former focuses on reproducible batch execution, plain text representation, version control, production output and offers the same editor and tools that you use for R scripts. The latter focus on outputting inline with code, caching the output across sessions, sharing code and outputting in a single file. Notebooks have an emphasis on an interactive execution model. They don’t use a plain text representation, but a structured data representation, such as JSON.

That all explains the purpose of RStudio’s notebook application: it combines all the advantages of R Markdown with the good things that computational notebooks have to offer.

To learn more about how you can work with R notebooks and what the exact differences are between Jupyter and R Markdown notebooks in terms of notebook sharing, project management, version control and more, check out DataCamp’s Jupyter and R: Notebooks with R post.

Other Data Science Notebooks

Of course, there are even more notebooks that you can consider when you’re getting into data science. In recent years, a lot of new alternatives have found their way to data scientists and data science enthusiasts: think not only of Beaker Notebook, Apache Zeppelin, Spark Notebook, DataBricks Cloud, etc., but also of other tools such as the Rodeo IDE which also make your data science analyses interactive and reproducible.

Plotting

One major feature of the IPython kernel is the ability to display plots that are the output of running code cells. The kernel is designed to work seamlessly with the matplotlib data visualization library to provide this functionality. To make use of it, use the magic command %matplotlib.

As such, your plot will be displayed in a separate window by default. Additionally, you can also specify a backend, such as inline or qt, the output of the plotting commands will be shown inline or through a different GUI backend. You can read more about it here.

FileSystem Navigation

The kernel’s magic commands also provide a way to navigate through your file system. The magics %cd and %bookmark can be used to either change directory or to bookmark a folder to have faster access to directories you often use.

Debugger Access

Next, you can also use magics to call up a Python debugger %pdb every time there is an uncaught exception. This will direct you through the part of the code that triggered the exception, which will make it possible rapidly find the source of a bug.

You can also use the %run magic command with the -d option to run scripts under the Python debugger’s control. It will automatically set up initial breakpoints for you. Lastly, you can also use the %debug magic for even easier debugger access.

IPython Extensions

You can use the %load_ext magic to load an IPython extension by its module name. IPython extensions are Python modules that modify the behaviour of the shell: extensions can register magics, define variables, and generally modify the user namespace to provide new features for use within code cells.

Here are some examples:

• Use %load_ext oct2py.ipython to seamlessly call M-files and Octave functions from Python,
• Use %load_ext rpy2.ipython to use an interface to R running embedded in a Python process,
• Use %load_ext Cython to use a Python to C compiler,
• Use sympy.init_printing() to pretty pritn Sympy Basic objects automatically, and
• To use Fortran in your interactive session, you can use %load_ext fortranmagic.
• … There are many more! You can create an register your own IPython extensions and register them on PyPi: this also means that there are many other user-defined extensions and magics out there! One example is ipython_unittest, but also check out this Extensions Index.

One of the other extensions that you should keep an eye out on is sparkmagic, a set of tools for interactively working with remote Spark clusters through Livy, a Spark REST server, in Jupyter notebooks. The sparkmagic library provides a %%spark magic that you can use to easily run code against a remote Spark cluster from a normal IPython notebook.

# Load in sparkmagic
%load_ext sparkmagic.magics

# Set the endpoint
%manage_spark

# Ask for help
%spark?

Go here for more examples of how you can make use of these magics to work with Spark cluster interactively.

Note that IPython has two other magics besides %load_ext that allow you to manage the extensions from within your Jupyter Notebook: %reload_ext and %unload_ext to unload, reimport and load an extension and to unload the extension, respectively.

Different Kernels, Other Magics

However, in other languages, the syntax element in the magic commands might have a meaning.

The R kernel, IRKernel, doesn’t have a magics sytem. To execute bash commands, for example, you’ll use R functions such as system() to invoke an OS command. An example would be system("head -5 *.csv", intern=TRUE). Note that by including the intern argument, you specify that you want to capture the output of the command as a character vector in R. To display markdown input, you make use of display_markdown(), to which you pass the Markdown code as a character vector.

Likewise, the Julia Kernel IJulia also doesn’t use “magics”. Instead, other syntaxes to accomplish the same goals are more natural in Julia, work in environments outside of IJulia code cells, and are often more powerful. However, the developers of the IJulia kernel have made sure that whenever you enter an IPython magic command in an IJulia code cell, you will see a printout with help that explains how you can achieve a similar effect in Julia if possible.

For example, the analog of IPython’s %load in IJulia is IJulia.load().

On the other hand, there are kernels such as the Scala kernel IScala that do support magic commands, similarly to IPython. However, the set of magics is different as it has to match the specifics of Scala and the JVM. Magic commands consist of percent sign % followed by an identifier and optional input to a magic. Some of the most notable magics are:

# Type Information
%type 1

# Library Management 
%libraryDependencies
%update

As you have read above, the sparkmagic library also provides a set of Scala and Python kernels that allow you to automatically connect to a remote Spark cluster, run code and SQL queries, manage your Livy server and Spark job configuration, and generate automatic visualizations. And this without needing any code!

For example, you can easily execute SparkSQL queries with %%sql or access Spark application information and logs via %%info magic.

If you’re working with another kernel and you wonder if you can make use of magic commands, it might be handy to know that there are some kernels that build on the metakernel project and that will use, in most cases, the same magics that you’ll also find in the IPython kernel. You can find a list of the metakernel magics here. The metakernel is a Jupyter/IPython kernel template which includes core magic functions.

Some examples:

• The MATLAB kernel matlab_kernel,
• The Octave kernel octave_kernel,
• The Java9 kernel java9_kernel,
• The Wolfram kernel wolfram_kernel,
• The SAS kernel sas_kernel.
• … And many more!

This means that, for example, when you’re using the MATLAB kernel, you’ll have the following magics available:

Available line magics:
%cd  %connect_info  %download  %edit  %get  %help  %html  %install  %install_magic  %javascript  %kernel  %kx  %latex  %load  %ls  %lsmagic  %magic  %parallel  %plot  %pmap  %px  %python  %reload_magics  %restart  %run  %set  %shell  %spell

Available cell magics:
%%debug  %%file  %%help  %%html  %%javascript  %%kx  %%latex  %%processing  %%px  %%python  %%shell  %%show  %%spell 

If you look at the chunk that is printed above, you’ll see that some of those magic commands seem very familiar. For those who don’t know the magics so well, compare the above chunk to the magics that you have available in the IPython kernel by default and you’ll see that some of the magics are the same:

Available line magics:
%alias  %alias_magic  %autocall  %automagic  %autosave  %bookmark  %cat  %cd  %clear  %colors  %config  %connect_info  %cp  %debug  %dhist  %dirs  %doctest_mode  %ed  %edit  %env  %gui  %hist  %history  %killbgscripts  %ldir  %less  %lf  %lk  %ll  %load  %load_ext  %loadpy  %logoff  %logon  %logstart  %logstate  %logstop  %ls  %lsmagic  %lx  %macro  %magic  %man  %matplotlib  %mkdir  %more  %mv  %notebook  %page  %pastebin  %pdb  %pdef  %pdoc  %pfile  %pinfo  %pinfo2  %popd  %pprint  %precision  %profile  %prun  %psearch  %psource  %pushd  %pwd  %pycat  %pylab  %qtconsole  %quickref  %recall  %rehashx  %reload_ext  %rep  %rerun  %reset  %reset_selective  %rm  %rmdir  %run  %save  %sc  %set_env  %store  %sx  %system  %tb  %time  %timeit  %unalias  %unload_ext  %who  %who_ls  %whos  %xdel  %xmode

Available cell magics:
%%!  %%HTML  %%SVG  %%bash  %%capture  %%debug  %%file  %%html  %%javascript  %%js  %%latex  %%perl  %%prun  %%pypy  %%python  %%python2  %%python3  %%ruby  %%script  %%sh  %%svg  %%sx  %%system  %%time  %%timeit  %%writefile

In essence, you can use one question to distinguish which magics are specific to IPython and which ones can be used in other kernels: is this functionality Python-specific or is it a general thing that can also be used in the language that you’re working with?

For example, %pdb or the Python Debugger or %matplotlib is something specific to Python that wouldn’t make sense when you’re working with the JavaScript kernel. However, changing directories with %cd is typically something that is very general and that should work in any language, as it is such a “general” command. So, chances are that this will be a magic that can be used in other kernels. Of course, you’ll still need to know whether your kernel makes use of the magics.