I don't think the author realizes how appropriate the Jedi tone for this article is. Comprehensions are the gateway drug to the dark side, away from imperative programming and toward languages that treat everything as expressions which snap together more freely. It hints at an idea of making a `for` loop and an `if` statement return a value (see CoffeeScript). But it also hints at the idea that useful idioms like the List Comprehension can merge/simplify existing constructs into a new, easier syntax-- and that there are languages that allow you to do this freely (see Lisp).
Python showed me the Force, but I'm with the Dark Side now.
One of the most extraordinary APIs for comprehensions can be found in Clojure, where the "for" expression really changed how I view data. A single expression can generate many complex data sequences with such elegant beauty.
List comprehensions are awesome. Not only that - python does them insanely beautifully. Clojure and ES6 are examples that I think aren't as readable, though equally powerful more or less. But in python they don't have any clunkiness. Simple, expressive. Love them.
Nesting them can get ugly, but it's easy to avoid: Just use generator expressions and chain them. No real runtime overhead that way.
Well with regards to clunkiness and awesomeness, are they not still less 'naturally' composable and (as a result) less readable in composition than collection pipelines? I don't see a good reason to prefer them over:
collection
.map(x => x * 2)
...
.filter(isOdd)
.reduce(blargh)
... style syntax that most other modern, C-style languages offer now (ES6, Rust, Ruby, C# ...)
It might be a small thing, but in the first one I feel the prime 'p' becomes the center piece, whereas in the second, 'p' is burrowed somewhat in the expression.
Both are fairly unintelligible to me (probably partly due to my lack of interest in prime numbers), so it seems like splitting hairs to draw comparisons, but if I had to figure out what was going on, I'd rather be staring at this (~ES6):
range(2, 100).filter(p => {
// I'd probably stick a comment here to explain this ...
// I'd split out this whole section into a named function once I understood it.
return range(2, p)
.map(q => p % q != 0)
.all();
})
I don't feel like I could iterate on understanding this problem so well in Python because I have to reach for one of 2 seemingly inferior solutions (list comprehensions or borked lambdas).
I think there are values in both styles. I've written a lot of both and for simpler things (which the vast majority if not all of code should be), I definitely prefer list comprehensions. They feel even more functional because they don't depend on map/filter/reduce defined on your objects.
I think chaining syntax from e.g. Elixir gives you more flexibility than map/filter/reduce defined on objects too. Big fan of that, though it does interact oddly with elixirs optional parentheses for function calls (which is a mistake of the language imo)
This is nothing new at all, it is called the "Fluent Interface" design. Many Python libraries implement this design, for instance the Pandas library is a popular example.
I hate this design approach deep in my soul. It makes for very brittle code that creates lots of backward compatibility issues. If you're working on some legacy code that has some nonsense like
it is maddening! You have to untangle just what exactly gets returned by every step of the chain, so that you can ensure you're in the right context to know exactly what the next call of the chain is doing.
In that example, say someone changes `foo.get_status()` to return some new kind of "status" object, and it alters the `dispatch_handler` and so on. Of course one can implement this in a way where the chain of downstream calls doesn't break, but the point isn't so much that, through huge engineering effort it is possible, but rather that it is extremely brittle and adds a layer of complexity that's not needed.
When the intermediate points of the chain are just functions, instead of member functions of a class, it means you can easily experiment with them and figure out what's going on without needing to recreate the entire set of context along the whole chain.
`run_dispatcher` in my example would be a hell of a lot easier to unit test and throw some mocked example class into for debugging or refactoring than if it is `some_class.run_dispatcher` ... and then if `some_class` has child classes that specialize the behavior, you're just hosed.
The problem is composability. People think that the fluent interface makes things composable because from some arbitrary point in the middle of the chain of calls, they have easy attribute-like access to the next operation they want to do. This artificially feels easy and convenient.
But contrast this to a functional language like Haskell, where none of these things need to be member functions of an object, and hence the context of the object doesn't have to be created at any point in the fluent chain. Then you can write something even better:
We can even easily refer to this whole chain of events with a single function name:
let statusDispatchLog = (close . logError . dispatchHandler . getStatus)
(And, of course, we get lots of nice type checking in statically typed languages to ensure that the composition actually makes sense -- which not only protects you at run time, but is also a huge help to clue you in to your design flaws. If you're trying to shoehorn some stuff into a fluent interface and it's not working, it probably means you have thought clearly about how the methods should "flow" in the call chain.)
To do the same thing in a fluent interface, we need a horrible lambda or a whole new function definition, exactly because the fluent interface is only sweeping the composability issues under the rug.
The difference is subtle, but important. Instead of making a new function that is explicitly the composition of other functions, you are making a function that just happens to access other functions as attributes, and if you set it up correctly then it acts as a sequence of composition.
In Python this is particularly a shame because functions are first class objects. Of course, you can write helper functions / decorators that sort of do function composition (if you're willing to throw away useful argument signatures), or you can use flaky hacks like the common Infix pattern in Python, and then live with ugly "<< . >>" or "|.|" misleading syntax.
It always makes me sad that Python lacks an extremely short function composition infix operator that provides some information about the function signatures of the functions being composed.
Because not even a comprehension can help you when you need to do the fluent interface stuff in Python.
Barry Warsaw proprosed List Comprehensions 16 years ago in PEP 202.
The original concept was that it provided a more concise way to write loops so that:
t = [expr(x, y) for x in s1 for y in s2]
was just a short way to write:
_t = []
for x in s1:
for y in s2
_t.append(expr(x, y))
t = _t
del _t
The original implementation reflected that design. I later added the LIST_APPEND opcode to give the list comprehensions a speed advantage over the unrolled code.
The question of whether to expose the loop induction variable didn't get much discussion until I proposed Generator Expressions in PEP 279 and decided to give them the behavior of hiding the loop induction variables so that the behavior would match that of a normal unrolled generator function.
When set and dict comprehensions (displays) came along afterwards, they were given the latter behavior because there was precedent, because there was a mechanism to implement that precedent, and to provide a short-cut for then common practice of creating dicts and sets with generator comprehensions:
s = {expr(x) for x in t}
was a short-form for:
s = set(expr(x) for x in t)
The advent of Python 3 gave us an opportunity to make a four forms (listcomps, genexps, set and dict displays) consistent about hiding the loop induction variable.
The current state in Python 3 has the advantage of being consistent between all four variants and matches how mathematicians treat bound and free variables.
There are some disadvantages as well. List comprehensions can no longer be cleanly explained as being equivalent to the unrolled version. The disassembly is harder to explore be cause you need to drill into the internal code object. Tracing the execution with PDB is no fun because you go up and down the stack. It is more difficult to explain scoping -- formerly, all you had was locals/globals/builtins, but now we have locals/nonlocals/globals/builtins plus variables bound in list comps, genexps, set/dict displays plus exception instances that are only visible inside the except-block.
> Pretty sure it was a bug that they kept for backwards compat, not intentional
Yep, as demonstrated by only list comprehensions exposing this behaviour (on python 2) whereas none of generator comprehensions, dictionary comprehensions or set comprehensions do:
>>> [x for x in range(5)]
[0, 1, 2, 3, 4]
>>> x
4
>>> {y for y in range(5)}
set([0, 1, 2, 3, 4])
>>> y
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'y' is not defined
>>> {y:y for y in range(5)}
{0: 0, 1: 1, 2: 2, 3: 3, 4: 4}
>>> y
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'y' is not defined
>>> list(y for y in range(5))
[0, 1, 2, 3, 4]
>>> y
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'y' is not defined
>>>
Hah, like there is any doubt that she'll become a Jedi. It's basically Episode IV all over again. Han Solo instead of Obi Wan, Rey instead of Luke, First Order instead of the Empire, Kylo Ren instead of Darth Vader.
We like hearing a story we already know. Rey is well on her way to become a Jedi, probably in two-movies' time.
1: Seems pretty clear that The Force Awakens is an "...origin story of a female Jedi." [1]
2: At some point while writing this I realised I would spend all evening if I had to round up all the planets, and chose to note that the lists are "non-exhaustive" instead :)
[::-1] is an idiomatic way to reverse a string in Python that is the obvious way for a habitual user of the language to do it e.g.:
def palindrome(s):
return s == s[::-1]
It could be discussed whether ''.join(reversed(s)) is more readable for a novice programmer learning Python. In general, Python prefers words over punctuation.
Also, there are objects that can be reversed() that are not sequences.
List comprehension in Python is great. However, that's Padawan territory. If you want to be a Jedi then APL is the only way. It's not "list comprehension" it's the way of the force when you use APL.
It's fun switching from Python to Scala where for/yield comprehensions are idiomatic and very natural (nested comprehensions being a good example). Oh, and type safe and fast. It's especially powerful to build off those constructs with currying, partial application, pattern matching, etc...
I love Python and write code in it daily but as a (pseudo-)functional language it feels very awkward to me.
> planets_flat = [planet for episode in episodes.values() for planet in episode['planets']]
Can somebody explain this one to me (I understand list comprehensions)? I'm having trouble understanding how the second part uses something defined in the first part, but the first part can't stand on its own, so
> Notice how the for loops in the comprehension goes in the same order as in the imperative code.
Thanks, that's a sane way to explain the order. Up to right now, it was always "the opposite of what you'd expect", which was a memory rule that always failed me.
It also helps on how ifs should be inserted, for example:
ys = []
for x in xs:
if P(x):
for y in Y(x):
if Q(x,y):
ys.append(y)
Becomes
[y
for x in xs
if P(x)
for y in Y(x)
if Q(x,y)
]
Surely combining this many for/if's may often be the wrong idea. Just like making a depth 4 iterative loop isn't always ideal. It does make the order easier to remember though :)
Which is why you should probably avoid nesting list comprehensions. A normal for loop with a single list comprehension inside it works just as well, and is far easier to read.
Once you know how it translates to regular for loops, this syntax is pretty natural. I always put each for loop on its own line, which makes things more readable.
Nope. Well - I could if I tried but that particular form is fairly abhorrent to me. My brain hurts just reading it so this is the stage I would find a clearer way to express the algorithm.
Comprehensions are great, but they can hurt readability and maintainability when taken too far. Most of the time you shouldn't be playing code golf with your code, iteratively seeking to pack more and more work into a single line of code. That makes the code harder to understand and harder to re-use.
While many of the examples in this helpful article are good, the first example with the octets is an excellent example of how not to do it. Look at the octet parsing code we ended up with in the article:
# Snippet 1
octets = [bbs[i:i+8] for i in range(0, len(bbs), 8)]
It's nice and tight. What does it do? I'd need to peer at it a moment and decode it, executing it in my head. This is subjective, but I think code should be self-explanatory; it's up to the computer to execute code, not people in their heads. Is there an off-by-one error in there? Here's another way to do it that'd be better.
octets = chunks(bbs, 8)
That function chunks() is something that I keep in an iterutils package which I end up using all the time. It's intuitive, and it has a doctest that shows that we definitely don't have an off-by-one error. It's also easier to re-use than the first one. Maybe it also bears mentioning that chunks() works on an iterator, while the first solution needs to keep the whole thing in memory at once. Here's the chunks method I use:
def chunks(collection, chunk_size):
"""Divides list l into chunks of up to n elements each.
>>> l = range(75)
>>> chunks(l,10)
[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14, 15, 16, 17, 18, 19],
[20, 21, 22, 23, 24, 25, 26, 27, 28, 29],
[30, 31, 32, 33, 34, 35, 36, 37, 38, 39],
[40, 41, 42, 43, 44, 45, 46, 47, 48, 49],
[50, 51, 52, 53, 54, 55, 56, 57, 58, 59],
[60, 61, 62, 63, 64, 65, 66, 67, 68, 69],
[70, 71, 72, 73, 74]]
"""
for i in xrange(0, len(collection), chunk_size):
yield collection[i : i + chunk_size]
When you catch yourself playing code golf and trying to pack more and more meaning into a single line, look for ways that you can break the problem down into multiple components that use each other. This kind of functional decomposition is one of the things that makes functional programming so wonderful. Lots of times the intermediate steps in a complex expression have meaning and are useful on their own.
However, I would reject your `chunks` function in a code review and tell you to use `grouper` form the itertools recipes [1].
More generally,any time I've ended up with a long list comprehension, the answer has been to "check itertools and see how you would describe this ugly comprehension in those terms"
All of the recipes in the itertools documentation should pretty much be considered folklore imho. If you always use the 'zip((iterator,)n)' approach, people will recognize it and know immediately what you mean.
Cool! When you need it to work on an input that's an iterator, please submit a patch with a doctest. That's one of the important benefits of reusable code.
One thing I find interesting is that everyone loves to hate CoffeeScript, but its individual features/syntax are consistently lauded in conversations about other languages.
(Not to mention half of ES6 existed in CoffeeScript first, but that's a gripe for another day)
The composition of features is more important than just the individual features - CoffeeScript is a great example of when feature composition goes wrong.
I just wish that python could stop being such a butt about not being like the rest of the languages
list of items --> filter list --> operate on list
becomes
operate on list <-- (list of items --> filter list)
And because this is how it was decided to tackle map/filter problems, we'll always have a weird gimped anon-function operator instead, to discourage the map/filter patterns of every other language
List comprehensions are just syntactic sugar over some common higher-order functions. Are we really getting excited over syntactic sugar? How about making Python's higher-order functions not suck instead?
I don't want to be a jedi Python programmer and exploit every neat trick of the language. I want to be a very good programmer and write code that's easy for others to read, debug, and maintain.
Yes let's make the reputation of Python more cryptic and culty. Because cryptic and culty things are better right?
Han Solo said it best "Hokey religions and ancient weapons are no match for a good blaster at your side, kid."
Lets keep it explicit alright? It's better then implicit.
Edit: read the article, it neither makes python cryptic or culty. The Jedi thing is just a cool SW reference. That said list comprehensions are a little cryptic to me since I haven't written python in awhile. Personally I think an important attribute of elegance in programming is how little you need to read the docs to understand something, and the more one-liners we do the more times we are likely to have to look at the documentation before reading it (not necessarily a bad thing, but sometimes a time sink and sometimes people won't look up the docs when they should!).
Eh, while comprehensions with multiple for-ins push the limits of good taste, I don't think:
planets_set = {
planet for episode in episodes.values()
for planet in episode['planets']
}
is less maintainable in a Python shop than say...
planets = set()
for episode in episodes.values():
planets.update(episode['planets'])
Although the latter will likely make perfect sense to most non-python developers. The former is faster and has a smaller memory footprint and might be preferred when dealing with a larger or more irregular data sets.
The former also has the advantage of not leaking the "episode" variable into the function/method scope, which could introduce a subtle bug if that variable gets conditionally reused. So while it's harder to understand for a less-experienced python developer, the set comprehension solution is inherently safer due to python's design.
Additionally, generator comprehensions can be far more efficient than any simple for loop. If you had a generator with a billion star coordinates being read from some file, you'd never be able to load it all into memory. So, instead of manually making a generator function, you could just do
coordinates = (star.x, star.y, star.z for star in star_map)
Gosh. If I was told off for using list comprehensions I'd start considering my employment options fairly quickly. Whilst they can be abused and pushed beyond the limits of reasonable use - in most cases I've seen in the wild they are the clearest and most Pythonic way to do it.
Nested list comprehensions with multiple filtering if statements? Scald away.
But the simpler examples there are perfectly readable, understandable and maintainable if you understand list comprehensions. And if you need to alter it to the point where it needs to be a little more verbose in order to convey what you're doing, then sure, you can break it out into some other structure. But just because the complexity might increase later on doesn't mean that you shouldn't use a list comprehension.
But what do you gain from using nested list comprehension like that? There's no performance gain (or very, very little). The only tangible benefit I see is saving a few lines of code.
And for those few lines of code you've traded the ability for new programmers to understand it easily. To me that's a net loss.
New programmers should learn how to use list comprehensions. They're not just some obscure Python concept, they appear in multiple other languages, as well as mathematics.
It's actually much more sensible code a lot of the time, especially for people who didn't get a degree that consisted of a lot of for-looping in class.
good_jedis = [jedi for jedi in universe if jedi.alignment == "good"]
vs.
good_jedis = []
for jedi in universe:
if jedi.alignment == "good":
good_jedis.append(jedi)
And then when you get on stuff like dictionary comprehensions and generator expressions, it starts becoming an amazing tool that is able to accomplish things that are straight-up cumbersome with old loops.
There's a computer science course out there that a friend was telling me about, that begins by showing students how to "map" a function over an iterable, way way before a "for loop" ever shows up. I think this is the way it should be done, "for loops" as a three-line-structure only seem natural to people who grew up programming for loops.
Actually, these are pretty idiomatic and understandable.
List comprehensions (and inline generators, which you get by using parenthesis) make it trivial to handle fairly complex sequence processing, and are one of the reasons Python gets a lot of love from the FP community.
Sure, it's not really an FP language, but these give it an almost LISPy feel.
Regardless,it would be less Pythonic to build a loop and iterate.
Might be easier for complete novices, but I don't suppose people are just hiring novices these days, right?
I just learned about this, so they are unreadable to me.
But it's not relevant. What relevant is, do they create a readability problem for a python developer who spent significant time with them? I honestly don't know.
I 'got' list comprehensions fairly immediately and find them clearer in most cases than map/filter for an explicit loop. I'm cautious when nesting them (as long as your code formatting is clear - a single nested comprehension is pretty acceptable) and I never use the multiple 'for' form as it's just not intuitive to me.
I am rather fond of dictionary comprehensions as mentioned in the article:
colors = [jedi['lightsaber_color'] for jedi in jedis]
frequencies = {color: colors.count(color) for color in set(colors)}
print(frequencies)
# {'green': 6, 'red': 5, 'blue': 6}
Really? I am pretty much mind-blown when the first time I learned about this. Given that my background is Mathematics, I thought it was pretty elegant and also looked very intuitive.
That's not really "cute" code though. Short list comprehension isn't any less readable (and certainly not less maintainable WTF?) than a multi-line "for" loop. In fact, I find short list comprehension more readable. I don't have to mentally keep track of place in a multistep loop.
Python showed me the Force, but I'm with the Dark Side now.