In this article by Thomas Bitterman, the author of the book Mastering IPython 4.0, we will look at the tools the IPython interactive shell provides. With the split of the Jupyter and IPython projects, the command line provided by IPython will gain importance.

This article covers the following topics:

• What is IPython?
• Installing IPython
• Starting out with the terminal
• IPython beyond Python
• Magic commands

(For more resources related to this topic, see here.)

What is IPython?

IPython is an open source platform for interactive and parallel computing. It started with the realization that the standard Python interpreter was too limited for sustained interactive use, especially in the areas of scientific and parallel computing.

Overcoming these limitations resulted in a three-part architecture:

• An enhanced, interactive shell
• Separation of the shell from the computational kernel
• A new architecture for parallel computing

This article will provide a brief overview of the architecture before introducing some basic shell commands. Before proceeding further, however, IPython needs to be installed.

Those readers with experience in parallel and high-performance computing but new to IPython will find the following sections useful in quickly getting up to speed. Those experienced with IPython may skim the next few sections, noting where things have changed now that the notebook is no longer an integral part of development.

Installing IPython

The first step in installing IPython is to install Python. Instructions for the various platforms differ, but the instructions for each can be found on the Python home page at http://www.python.org. IPython requires Python 2.7 or ≥ 3.3. This article will use 3.5. Both Python and IPython are open source software, so downloading and installation are free.

A standard Python installation includes the pip package manager. pip is a handy command-line tool that can be used to download and install various Python libraries. Once Python is installed, IPython can be installed with this command:

pip install ipython

IPython comes with a test suite called iptest. To run it, simply issue the following command:

iptest

A series of tests will be run. It is possible (and likely on Windows) that some libraries will be missing, causing the associated tests to fail. Simply use pip to install those libraries and rerun the test until everything passes.

It is also possible that all tests pass without an important library being installed. This is the readline library (also known as PyReadline). IPython will work without it but will be missing some features that are useful for the IPython terminal, such as command completion and history navigation. To install readline, use pip:

pip install readline
pip install gnureadline

At this point, issuing the ipython command will start up an IPython interpreter:

ipython

IPython beyond Python

No one would use IPython if it were not more powerful than the standard terminal. Much of IPython’s power comes from two features:

• Shell integration
• Magic commands

Shell integration

Any command starting with ! is passed directly to the operating system to be executed, and the result is returned. By default, the output is then printed out to the terminal. If desired, the result of the system command can be assigned to a variable. The result is treated as a multiline string, and the variable is a list containing one string element per line of output. For example:

In [22]: myDir = !dir

In [23]: myDir
Out[23]:
[' Volume in drive C has no label.',
 ' Volume Serial Number is 1E95-5694',
 '',
 ' Directory of C:\Program Files\Python 3.5',
 '',
 '10/04/2015  08:43 AM    <DIR>          .',
 '10/04/2015  08:43 AM    <DIR>          ..',]

While this functionality is not entirely absent in straight Python (the OS and subprocess libraries provide similar abilities), the IPython syntax is much cleaner. Additional functionalities such as input and output caching, directory history, and automatic parentheses are also included.

History

The previous examples have had lines that were prefixed by elements such as In[23] and Out[15]. In and Out are arrays of strings, where each element is either an input command or the resulting output. They can be referred to using the arrays notation, or “magic” commands can accept the subscript alone.

Magic commands

IPython also accepts commands that control IPython itself. These are called “magic” commands, and they start with % or %%. A complete list of magic commands can be found by typing %lsmagic in the terminal.

Magics that start with a single % sign are called “line” magics. They accept the rest of the current line for arguments. Magics that start with %% are called “cell” magics. They accept not only the rest of the current line but also the following lines.

There are too many magic commands to go over in detail, but there are some related families to be aware of:

• OS equivalents: %cd, %env, %pwd
• Working with code: %run, %edit, %save, %load, %load_ext, %%capture
• Logging: %logstart, %logstop, %logon, %logoff, %logstate
• Debugging: %debug, %pdb, %run, %tb
• Documentation: %pdef, %pdoc, %pfile, %pprint, %psource, %pycat, %%writefile
• Profiling: %prun, %time, %run, %time, %timeit
• Working with other languages: %%script, %%html, %%javascript, %%latex, %%perl, %%ruby

With magic commands, IPython becomes a more full-featured development environment. A development session might include the following steps:

1. Set up the OS-level environment with the %cd, %env, and ! commands.
2. Set up the Python environment with %load and %load_ext.
3. Create a program using %edit.
4. Run the program using %run.
5. Log the input/output with %logstart, %logstop, %logon, and %logoff.
6. Debug with %pdb.
7. Create documentation with %pdoc and %pdef.

This is not a tenable workflow for a large project, but for exploratory coding of smaller modules, magic commands provide a lightweight support structure.

Creating custom magic commands

IPython supports the creation of custom magic commands through function decorators. Luckily, one does not have to know how decorators work in order to use them. An example will explain.

First, grab the required decorator from the appropriate library:

In [1]: from IPython.core.magic import(register_line_magic)

Then, prepend the decorator to a standard IPython function definition:

In [2]: @register_line_magic
   ...: def getBootDevice(line):
   ...:         sysinfo = !systeminfo
   ...:         for ln in sysinfo:
   ...:                 if ln.startswith("Boot Device"):
   ...:                        return(ln.split()[2])
   ...:
Your new magic is ready to go:
In [3]: %getBootDevice
Out[3]: '\Device\HarddiskVolume1'

Some observations are in order:

• Note that the function is, for the most part, standard Python. Also note the use of the !systeminfo shell command. One can freely mix both standard Python and IPython in IPython.
• The name of the function will be the name of the line magic.
• The parameter, “line,” contains the rest of the line (in case any parameters are passed).
• A parameter is required, although it need not be used.
• The Out associated with calling this line magic is the return value of the magic.
• Any print statements executed as part of the magic are displayed on the terminal but are not part of Out (or _).

Cython

We are not limited to writing custom magic commands in Python. Several languages are supported, including R and Octave. We will look at one in particular, Cython.

Cython is a language that can be used to write C extensions for Python. The goal for Cython is to be a superset of Python, with support for optional static type declarations. The driving force behind Cython is efficiency. As a compiled language, there are performance gains to be had from running C code. The downside is that Python is much more productive in terms of programmer hours. Cython can translate Python code into compiled C code, achieving more efficient execution at runtime while retaining the programmer-friendliness of Python.

The idea of turning Python into C is not new to Cython. The default and most widely used interpreter (CPython) for Python is written in C. In some sense then, running Python code means running C code, just through an interpreter. There are other Python interpreter implementations as well, including those in Java (Jython) and C# (IronPython).

CPython has a foreign function interface to C. That is, it is possible to write C language functions that interface with CPython in such a way that data can be exchanged and functions invoked from one to the other. The primary use is to call C code from Python. There are, however, two primary drawbacks: writing code that works with the CPython foreign function interface is difficult in its own right; and doing so requires knowledge of Python, C, and CPython.

Cython aims to remedy this problem by doing all the work of turning Python into C and interfacing with CPython internally to Cython. The programmer writes Cython code and leaves the rest to the Cython compiler. Cython is very close to Python. The primary difference is the ability to specify C types for variables using the cdef keyword. Cython then handles type checking and conversion between Python values and C values, scoping issues, marshalling and unmarshalling of Python objects into C structures, and other cross-language issues.

Cython is enabled in IPython by loading an extension. In order to use the Cython extension, do this:

In [1]: %load_ext Cython

At this point, the cython cell magic can be invoked:

In [2]: %%cython
   ...: def sum(int a, int b):
   ...:     cdef int s = a+b
   ...:     return s

And the Cython function can now be called just as if it were a standard Python function:

In [3]: sum(1, 1)
Out[3]: 2

While this may seem like a lot of work for something that could have been written more easily in Python in the first place, that is the price to be paid for efficiency. If, instead of simply summing two numbers, a function is expensive to execute and is called multiple times (perhaps in a tight loop), it can be worth it to use Cython for a reduction in runtime.

There are other languages that have merited the same treatment, GNU Octave and R among them.

Summary

In this article, we covered many of the basics of using IPython for development. We started out by just getting an instance of IPython running. The intrepid developer can perform all the steps by hand, but there are also various all-in-one distributions available that will include popular modules upon installation. By default, IPython will use the pip package managers. Again, the all-in-one distributions provide added value, this time in the form of advanced package management capability.

At that point, all that is obviously available is a terminal, much like the standard Python terminal. IPython offers two additional sources of functionality, however: configuration and magic commands.

Magic commands fall into several categories: OS equivalents, working with code, logging, debugging, documentation, profiling, and working with other languages among others. Add to this the ability to create custom magic commands (in IPython or another language) and the IPython terminal becomes a much more powerful alternative to the standard Python terminal.

Also included in IPython is the debugger—ipdb. It is very similar to the Python pdb debugger, so it should be familiar to Python developers.

All this is supported by the IPython architecture. The basic idea is that of a Read-Eval-Print loop in which the Eval section has been separated out into its own process. This decoupling allows different user interface components and kernels to communicate with each other, making for a flexible system.

This flexibility extends to the development environment. There are IDEs devoted to IPython (for example, Spyder and Canopy) and others that originally targeted Python but also work with IPython (for example, Eclipse). There are too many Python IDEs to list, and many should work with an IPython kernel “dropped in” as a superior replacement to a Python interpreter.

Resources for Article:

---

Further resources on this subject:

• Python Data Science Up and Running [article]
• Scientific Computing APIs for Python [article]
• Overview of Process Management in Microsoft Visio 2013 [article]

---