Python Geospatial Development
Build a complete and sophisticated mapping application from scratch using Python tools for GIS development
- Build applications for GIS development using Python
- Analyze and visualize Geo-Spatial data
- Comprehensive coverage of key GIS concepts
- Recommended best practices for storing spatial data in a database
- Draw maps, place data points onto a map, and interact with maps
- A practical tutorial with plenty of step-by-step instructions to help you develop a mapping application from scratch
The “Slippy Map” Stack
The “slippy map” is a concept popularized by Google Maps: a zoomable map where the user can click and drag to scroll around and double-click to zoom in. Here is an example of a Google Maps slippy map showing a portion of Europe:
(Image copyright Google; map data copyright Europa Technologies, PPWK, Tele Atlas).
Slippy maps have become extremely popular, and much of the work done on geo-spatial web application development has been focused on creating and working with slippy maps.
The slippy map experience is typically implemented using a custom software stack, like this:
Starting at the bottom, the raw map data is typically stored in a Shapefile or database. This is then rendered using a tool such as Mapnik, and a tile cache is used to speed up repeated access to the same map images. A user-interface library such as OpenLayers is then used to display the map in the user’s web browser, and to respond when the user clicks on the map. Finally, a web server is used to allow web browsers to access and interact with the slippy map.
Let’s take a closer look at each of these pieces.
Spatially-enabled Databases
In a sense, almost any database can be used to store geo-spatial data: simply convert a geometry to WKT format and store the results in a text column. But while this would allow you to store geo-spatial data in a database, it wouldn’t let you query it in any useful way. All you could do is retrieve the raw WKT text and convert it back to a geometry object one record at a time.
A spatially-enabled database, on the other hand, is aware of the notion of space, and allows you to work with spatial objects and concepts directly. In particular, a spatially-enabled database allows you to:
- Store spatial data types (points, lines, polygons, and so on) directly in the database, in the form of a geometry column.
- Perform spatial queries on your data. For example: select all landmarks within 10 km of the city named “San Francisco”.
- Perform spatial joins on your data. For example: select all cities and their associated countries by joining cities and countries on (city inside country).
- Create new spatial objects using various spatial functions. For example: set “danger_zone” to the intersection of the “flooded_area” and “urban_area” polygons.
As you can imagine, a spatially-enabled database is an extremely powerful tool for working with your geo-spatial data. By using spatial indexes and other optimizations, spatial databases can quickly perform these types of operations, and can scale to support vast amounts of data simply not feasible using other data-storage schemes.
Map Rendering
Mapnik is an example of a Python library for generating good-looking maps. Within the context of a web application, map rendering is usually performed by a web service which takes a request and returns the rendered map image file. For example, your application might include a map renderer at the relative URL /render which accepts the following parameters:
- minX, maxX, minY, maxY: The minimum and maximum latitude and longitude for the area to include on the map.
- width, height: The pixel width and height for the generated map image.
- layers: A comma-separated list of layers which are to be included on the map. The available predefined layers are: coastline, forest, waterways, urban, and street.
- formatThe desired image format. Available formats are: PNG, JPEG, GIF.
This hypothetical /render web service would return the rendered map image back to the caller. Once this has been set up, the web service would act as a black box providing map images upon request for other parts of your web application.
As an alternative to hosting and configuring your own map renderer, you can choose to use an openly available external renderer. For example, OpenStreetMap provides a freely-available map renderer for OpenStreetMap data at: http://dev.openstreetmap.de/staticmap
Tile Caching
Because creating an image out of raw map data is a time- and processor-intensive operation, your entire web application can be overloaded if you get too many requests for data at any one time. While there is a lot you can do to improve the speed of the map-generation process, there are still limits on how many maps your application can render in a given time period.
Because the map data is generally quite static, you can get a huge improvement in your application’s performance by caching the generated images. This is generally done by dividing the world up into tiles, rendering tile images as required, and then stitching the tiles together to produce the desired map:
Tile caches work in exactly the same way as any other cache:
- When a tile is requested, the tile cache checks to see if it contains a copy of the rendered tile. If so, the cached copy is returned right away.
- Otherwise, the map rendering service is called to generate the tile, and the newly-rendered tile is added to the cache before returning it back to the caller.
- As the cache grows too big, tiles which haven’t been requested for a long time are removed to make room for new tiles.
Of course, tile caching will only work if the underlying map data doesn’t change. You can’t use a tile cache where the rendered image varies from one request to the next.
One interesting use of a tile cache is to combine it with map overlays to improve performance even when the map data does change. Because the outlines of countries and other physical features on a map don’t change, it is possible to use a map generator with a tile cache to generate the base map onto which changing features are then drawn as an overlay:
The final map could be produced using Mapnik, by drawing the overlay onto the base map, which is accessed using a RasterDataSource and displayed using a RasterSymbolizer. If you have enough disk space, you could even pre-calculate all of the base map tiles and have them available for quick display. Using Mapnik in this way is a fast and efficient way of combining changing and non-changing map data onto a single view — though there are other ways of overlaying data onto a map, for example using vector and raster layers within the OpenLayers library.
User Interface Libraries
While it is easy to build a simple web-based interface in HTML, users are increasingly expecting web applications to compete with desktop applications in terms of their user interface. Selecting objects by clicking on them, drawing images with the mouse, and dragging-and-dropping are no longer actions restricted to desktop applications.
AJAX (Asynchronous JavaScript and XML) is the technology typically used to build complex user interfaces in a web application. In particular, running JavaScript code on the user’s web browser allows the application to dynamically respond to user actions, and make the web page behave in ways that simply can’t be achieved with static HTML pages.
While JavaScript is ubiquitous, it is also hard to program in. The various web browsers in which the JavaScript code can run all have their own quirks and limitations, making it hard to write code that runs the same way on every browser. JavaScript code is also very low level, requiring detailed manipulation of the web page contents to achieve a given effect. For example, implementing a pop-up menu requires the creation of a
element that contains the menu, formatting it appropriately (typically using CSS), and making it initially invisible. When the user clicks on the page, the pop-up menu should be shown by making the associated
element visible. You then need to respond to the user mousing over each item in the menu by visually highlighting that item and un-highlighting the previously-highlighted item. Then when the user clicks, you have to hide the menu again before responding to the user’s action.
All this detailed low-level coding can take weeks to get right—especially when dealing with multiple types of browsers and different browser versions. Since all you want to do in this case is have a pop-up menu that allows the user to choose an action, it just isn’t worth doing all this low-level work yourself. Instead, you would typically make use of one of the available user interface libraries to do all the hard work for you.
These user interface libraries are written in JavaScript, and you typically add them to your web site by making the JavaScript library file(s) available for download, and then adding a line like the following to your HTML page to import the JavaScript library:
<script type="text/javascript" src="library.js">
If you are writing your own web application from scratch, you would then make calls to the library to implement the user interface for your application. However, many of the web application frameworks that include a user interface library will write the necessary code for you, making even this step unnecessary.
Web Servers
In many ways a web server is the least interesting part of a web application: the web server listens for incoming HTTP requests from web browsers and returns either static content or the dynamic output of a program in response to these requests:
There are many different types of web servers, ranging from the pure-python SimpleHTTPServer included in the Python Standard Library, through more fully-featured servers such as CherryPy, and of course the most popular industrial-strength web server of them all: Apache.
One of the main consequences of your choice of web server is how fast your application will run. Obviously, a pure-Python web server will be slower than a compiled high-performance server such as Apache. In addition, writing CGI scripts in Python will cause the entire Python interpreter to be started up every time a request is received—so even if you are running a high performance web server, your application can still run slowly if you don’t structure your application correctly. A slow web server doesn’t just affect your application’s responsiveness: if your server runs slowly, it won’t take many requests to overload the server.
Another consequence of your choice of web server is how your application’s code interacts with the end user. The HTTP protocol itself is stateless—that is, each incoming request is completely separate, and a web page handler has no way of knowing what the user has done previously unless you explicitly code your pages in such a way that the application’s state is passed from one request to the next (for example, using hidden HTML form fields).
Because some web servers run your Python code only when a request comes in, there is often no way of having a long-running process sitting in the background that keeps track of the user’s state or performs other capabilities for your web page handlers. For example, an in-memory cache might be used to improve performance, but you can’t easily use such a cache with CGI scripts as the entire interpreter is restarted for every incoming HTTP request.
While the choice of web server will often have been made for you, for example by the server you are running your system on, or by the web application framework you are using, it is still worthwhile understanding the issues and consequences involved in the choice of a web server, and how your application design can affect the performance and scalability of your overall system.
Summary
While you could choose to use Google Maps to create a slippy map experience for your users, it is just as easy to combine your own geo-spatial data source such as a spatially-enabled database with a map renderer, a tile cache, a user-interface library and a web server to create your own custom slippy map stack. Implementing your own stack means you aren’t limited by Google’s licensing requirements, and can use your own map data and render the maps in whatever way you like. You also have far more flexibility in terms of what information gets displayed, and can go beyond what is possible using Google Maps, for example by implementing geo-spatial editing and analysis tools directly within your slippy map.
Further resources on this subject:
- Python Graphics: Animation Principles [Article]
- Python 3: Object-Oriented Design [Article]
- Animating Graphic Objects using Python [Article]
- Python Testing: Beginner’s Guide [Book]
