In my latest book, Fast Python, I bust a lot of speed myths in Python. Some of the most basic advice people give about optimizing Python code is wrong, likely because no one ever bothered to test it. In this book, I go back to the basics and verify (or bust) all of the advice and assumed knowledge that we share as a programming community.
In this post, I’ll show you five ways to flatten a list in Python, and then show you the test results. I consider these results to be a myth-buster, because they are not at all what I expected based on what we consider best practice in the Python community.
# Use long-form list addition def slow_add_flatten_lists(the_lists): result =  for _list in the_lists: result = result + _list return result # Use list addition with the += operator def add_flatten_lists(the_lists) result =  for _list in the_lists: result += _list return result # Use List.extend def extend_flatten_lists(the_lists): result =  for _list in the_lists: result.extend(_list) return result # Use List.append def append_flatten_lists(the_lists) result =  for _list in the_lists: for val in _list: result.append(val) return result # Use a nested list comprehension def comprehension_flatten_lists(the_lists) return [val for _list in the_lists for val in _list]
Most people would expect the third or fifth option to be fastest, because they are the most concise and best utilize Python’s existing data structures to achieve the desired outcome, but this is not the case. For some reason, the second method, using list addition with the
+= operator, is the fastest by a large margin. Here are the profiles.
Most people assume that the
+= operator on lists must be slow, because
a += b is likely just syntactical sugar for
a = a + b, which is provably slow. This isn’t the case. The
+= operator is optimized in some other ways for Python lists, as is proven by the code profiles, and it is not at all equivalent to
a = a + b. Surprisingly enough, it is not only optimized, but it ends up being the fastest known method the flatten a list.
Check out Fast Python if you are interested in busting more speed myths and learning how to optimize your Python code. Writing this book has been very helpful to my own programming, because it has given me access to empirical evidence for my speed-based design decisions.