(For more resources related to this topic, see here.)
The Python threading module
The Python threading module offers a layer of abstraction to the module _thread, which is a lower-level module. It provides functions that help the programmer during the hard task of developing parallel systems based on threads. The threading module’s official papers can be found at http://docs.python.org/3/library/threading.html?highlight=threading#module-threadin.
The Python multiprocessing module
The multiprocessing module aims at providing a simple API for the use of parallelism based on processes. This module is similar to the threading module, which simplifies alternations between the processes without major difficulties. The approach that is based on processes is very popular within the Python users’ community as it is an alternative to answering questions on the use of CPU-Bound threads and GIL present in Python. The multiprocessing module’s official papers can be found at http://docs.python.org/3/library/multiprocessing.html?highlight=multiprocessing#multiprocessing.
The parallel Python module
The parallel Python module is external and offers a rich API for the creation of parallel and distributed systems making use of the processes approach. This module promises to be light and easy to install, and integrates with other Python programs. The parallel Python module can be found at http://parallelpython.com. Among some of the features, we may highlight the following:
- Automatic detection of the optimal confi guration
- The fact that a number of worker processes can be changed during runtime
- Dynamic load balance
- Fault tolerance
- Auto-discovery of computational resources
Celery – a distributed task queue
Celery is an excellent Python module that’s used to create distributed systems and has excellent documentation. It makes use of at least three different types of approach to run tasks in concurrent form—multiprocessing, Eventlet, and Gevent. This work will, however, concentrate efforts on the use of the multiprocessing approach. Also, the link between one and another is a configuration issue, and it remains as a study so that the reader is able to establish comparisons with his/her own experiments.
The Celery module can be obtained on the official project page at http://celeryproject.org.
Summary
In this article, we had a short introduction to some Python modules, built-in and external, which makes a developer’s life easier when building up parallel systems.
Resources for Article:
Further resources on this subject:
- Getting Started with Spring Python [Article]
- Python Testing: Installing the Robot Framework [Article]
- Getting Up and Running with MySQL for Python [Article]
