When you begin using Python after using other languages, it’s easy to bring a lot of idioms with you. Though they may work, they are not the best, most beautiful, or fastest ways to get things done with Python—they’re not pythonic. I’ve put together some tips on basic things that can provide big performance improvements, and I hope they’ll serve as a starting point for you as you develop with Python.
Use comprehensions
Comprehensions are great. Python knows how to make lists, tuples, sets, and dicts from single statements, so you don’t need to declare, initialize, and append things to your sequences as you do in Java. It helps not only in readability but also on performance; if you delegate something to the interpreter, it will make it faster.
def do_something_with(value):
return value * 2
# this is an anti-pattern
my_list = []
for value in range(10):
my_list.append(do_something_with(value))
# this is beautiful and faster
my_list = [do_something_with(value) for value in range(10)]
# and you can even plug some validation
def some_validation(value):
return value % 2
my_list = [do_something_with(value) for value in range(10) if some_validation(value)]
my_list
[2, 6, 10, 14, 18]And it looks the same for other types. You just need to change the appropriate surrounding symbols to get what you want.
my_tuple = tuple(do_something_with(value) for value in range(10))
my_tuple
(0, 2, 4, 6, 8, 10, 12, 14, 16, 18)
my_set = {do_something_with(value) for value in range(10)}
my_set
{0, 2, 4, 6, 8, 10, 12, 14, 16, 18}
my_dict = {value: do_something_with(value) for value in range(10)}
my_dict
{0: 0, 1: 2, 2: 4, 3: 6, 4: 8, 5: 10, 6: 12, 7: 14, 8: 16, 9: 18}Use Generators
Generators are objects that generate sequences one item at a time. This provides a great gain in performance when working with large data, because it won’t generate the whole sequence unless needed, and it’s also a memory saver. A simple way to use generators is very similar to the comprehensions we saw above, but you encase the sentence with () instead of [] for example:
my_generator = (do_something_with(value) for value in range(10))
my_generator
<generator object <genexpr> at 0x7f0d31c207e0>The range function itself returns a generator (unless you’re using legacy Python 2, in which case you need to use xrange).
Once you have a generator, call next to iterate over its items, or use it as a parameter to a sequence constructor if you really need all the values:
next(my_generator)
0
next(my_generator)
2To create your own generators, use the yield keyword inside a loop instead of a regular return at the end of a function or method. Each time you call next on it, your code will run until it reaches a yield statement, and it saves the state for the next time you ask for a value.
# this is a generator that infinitely returns a sequence of numbers, adding 1 to the previous
def my_generator_creator(start_value):
while True:
yield start_value
start_value += 1
my_integer_generator = my_generator_creator(0)
my_integer_generator
<generator object my_generator_creator at 0x7f0d31c20708>
next(my_integer_generator)
0
next(my_integer_generator)
1The benefits of generators in this case are obvious—you would never end up generating numbers if you were to create the whole sequence before using it. A great use of this, for example, is for reading a file stream.
Use sets for membership checks
It’s wonderful how we can use the in keyword to check for membership on any type of sequence. But sets are special. They’re of a mapping kind, an unordered set of values where an item’s positions are calculated rather than searched.
When you search for a value inside a list, the interpreter searches the entire list to see whether the value is there. If you are lucky, the value is the first of the sequence; on the other hand, it could even be the last in a long list.
When working with sets, the membership check always takes the same time because the positions are calculated and the interpreter knows where to search for the value. If you’re using long sets or loops, the performance gain is sensible. You can create a set from any iterable object as long as the values are hashable.
my_list_of_fruits = ['apple', 'banana', 'coconut', 'damascus']
my_set_of_fruits = set(my_list_of_fruits)
my_set_of_fruits
{'apple', 'banana', 'coconut', 'damascus'}
'apple' in my_set_of_fruits
True
'watermelon' in my_set_of_fruits
False
Deal with strings the right way
You’ve probably done or read something like this before:
# this is an anti-pattern
"some string, " + "some other string, " + "and yet another one"
'some string, some other string, and yet another one'It might look easy and fast to write, but it’s terrible for your performance. str objects are immutable, so each time you add strings, trying to append them, you’re actually creating new strings.
There are a handful of methods to deal with strings in a faster and optimal way.
To join strings, use the join method on a separator with a sequence of strings. The separator can be an empty string if you just want to concatenate.
# join a sequence of strings, based on the separator you want
", ".join(["some string", "some other string", "and yet another one"])
'some string, some other string, and yet another one'
''.join(["just", "concatenate"])
'justconcatenate'To merge strings, for example, to insert information in templates, we have a classical way; it resembles the C language:
# the classical way
"hello %s, my name is %s" % ("everyone", "Gabriel")
'hello everyone, my name is Gabriel'And then there is the modern way, with the format method. It is quite flexible:
# formatting with sequencial strings
"hello {}, my name is {}".format("everyone", "Gabriel")
'hello everyone, my name is Gabriel'
# formatting with indexed strings
"hello {1}, my name is {2} and we all love {0}".format("Python", "everyone", "Gabriel")
'hello everyone, my name is Gabriel and we all love Python'Avoid intermediate outputs
Every programmer in this world has used print statements for debugging or progress checking purposes at least once. If you don’t know pdb for debugging yet, you should check it out immediately.
But I’ll agree that it’s really easy to write print statements inside your loops to keep track of where your program is. I’ll just tell you to avoid them because they’re synchronous and will significantly raise the execution time.
You can think of alternative ways to check progress, such as watching via the filesystem the files that you have to generate anyway.
Asynchronous programming is a huge topic that you should take a look at if you’re dealing with a lot of I/O operations.
Cache the most requested results
Caching is one of the greatest performance tunning tweaks you’ll ever find.
Python gives us a handy way of caching function calls with a simple decorator, functools.lru_cache. Each time you call a function that is decorated with lru_cache, the interpreter checks whether that call was made recently, on a cache that is a dictionary of parameters-result pairs. dict checks are as fast as those of set, and if we have repetitive calls, it’s worth looking at this cache before running the code again.
from functools import lru_cache
@lru_cache(maxsize=16)
def my_repetitive_function(value):
# pretend this is an extensive calculation
return value * 2
for value in range(100):
my_repetitive_function(value % 8)The decorator gives the method cache_info, where we can find the statistics about the cache. We can see the eight misses (for the eight times the function was really called), and 92 hits. As we only have eight different inputs (because of the % 8 thing), the cache size was never fully filled.
my_repetitive_function.cache_info()
CacheInfo(hits=92, misses=8, maxsize=16, currsize=8)Read
In addition to these six tips, read a lot, every day. Read books and other people’s code. Make code and talk about code. Time, practice, and exchanging experiences will make you a great programmer, and you’ll naturally write better Python code.
About the author
Gabriel Marcondes is a computer engineer working on Django and Python in São Paulo, Brazil. When he is not coding, you can find him at @ggzes, talking about rock n’ roll, football, and his attempts to send manned missions to fictional moons.
