I don't understand this argument. Clojure shouldn't have transducers because the word sounds scary? Programming language designers should avoid adding powerful, higher-order abstractions because they are hard to understand? This sentiment is incredibly anti-intellectual. And like you said, if you don't understand it you don't have to use it.
Language design requires tradeoffs: the power of a feature, vs. how much more complex it makes the language to read/learn/etc. Making something easy to use doesn't commit one to anti-intellectualism; it just shifts the field of intellectual challenge elsewhere.
See Yegge's Perl essays[1] for examples of core-language features that are 'powerful' but problematic; hopefully that convinces you that at least the form of argument is legitimate. It could be that those Perl features are bad whereas transducers are great. I happen to share some skepticism of transducers because they overly-resemble existing features (partial application) and can be implemented pretty easy with existing Clojure tools. Also the marquee use case (sharing transformations between sequences and channels) so far seems exceedingly rare to me, though maybe things are moving in a direction where that's more important.
> And like you said, if you don't understand it you don't have to use it.
True to an extent, but (a) the high prominence given to the feature may well lead to a situation where you can't read other peoples' code without learning the concept, and (b) it's easy to create transducers by accident.
Most languages with map/reduce don't have transducers (and they're certainly not central features). Also, don't forget that while Clojure is functional, it is also very much imperative (it's a functional-imperative language rather than a pure-functional language).
> Most languages with map/reduce don't have transducers
That's true but mostly irrelevant to the point that transducer usage isn't complex once one understands map/reduce, which are common, whether or not transducers are.
> Programming language designers should avoid adding powerful, higher-order abstractions because they are hard to understand?
In general? Absolutely![1] Making algorithms easier for humans to understand is the whole purpose of abstractions.
Programming is based two things: algorithms and abstractions, with algorithms being fundamental to computation and abstractions are usability features designed to help people write code -- that is their one and only purpose (computers and even theoretical computational models don't care about abstractions). Unlike algorithms, abstractions are not useful in isolation, and their utility is not a function of mathematical "power" but of psychological benefit. Their utility is measured by how much they help the human programmer write and read an algorithm[2]. Another way to look at it is that algorithms tackle essential complexity and abstractions tackle accidental complexity[3].
More specifically, abstractions help human programmers in two ways: they increase code reuse and improve code readability.
But here's the thing. There are things other than powerful abstractions that help humans program -- for example, a clear execution model (what happens when), debuggability etc.. This means that the more powerful (i.e. abstract) abstractions become they do not necessarily perform their function -- namely, assisting developers -- better. So every abstraction must be carefully weighed: how much wasted code does it save, how much more readable it makes code, vs. how much does it hurt understanding or debuggability.
I'd say transducers are just about the point where the abstraction starts hurting more than it helps. It's a borderline case. Now, I am not saying Clojure shouldn't have transducers (again, borderline), but that they most certainly shouldn't be emphasized.
> if you don't understand it you don't have to use it.
That doesn't quite work. I said, don't use them right away. Once a programming language and its libraries use an abstraction, you must learn it sooner or later. After all, most code you'll read isn't your own. This is why every language feature has a cost, and why good language developers are hesitant about introducing new features (transducers aren't a language feature, but they do have a prominent place in the standard library).
---
[1]: For example, Java and Go are both languages whose designers intentionally and radically reduced the use of many of the abstractions available in other popular languages of their time. Java in particular drastically removed abstractions possible by the most popular language at the time it was introduced, and attained tremendous success, partly because of that (well, at the time it was already apparent that C++'s power -- in addition to its lack of safety -- was greatly detrimental to code maintenance in most parts of the industry).
[2]: Abstractions are secondary to algorithms. Also note how almost all sub-disciplines of computer science deal with algorithms, and just one, rather small, discipline -- PL research -- is concerned with abstractions.
[3]: Even algorithms are often not judged in isolation; there are lots and lots of useful algorithms that aren't used because they are too hard to implement and maintain correctly -- regardless of abstractions used.
For example, Java and Go are both languages whose designers intentionally and radically reduced the use of many of the abstractions available in other popular languages of their time.
Java is a perfect example of proliferation of accidental complexity caused by the unwillingness to provide facilities for composing abstractions in the core language. The Java community has resorted to massive external XML-based configuration files to provide code reuse where it is impossible to achieve inside the language.
The whole point of Lisp (and Clojure is a Lisp) is the power to compose functions and lists together, building higher and higher-level abstractions to allow concise expression of logic to solve a problem. The point of Lisp is to be expressive and powerful, not popular and readable. BASIC and COBOL were popular and readable.
> Java is a perfect example of proliferation of accidental complexity caused by the unwillingness to provide facilities for composing abstractions in the core language.
Well, those are all tradeoffs, and the fact is that since the addition of annotations and later lambdas, all those "external XML-based configuration files" are receding, to the point they no longer exist in almost any of the newer libraries (or new versions of old libraries).
Java didn't start out with insufficient abstractions that were later added by other languages. Java started out as a reaction to languages with overly-powerful abstractions that hindered maintenance. You may not like the result and think it aesthetically unpleasing (although it's been getting better and better for quite a while now), but it is a fact that Java codebases are extremely maintainable. This is not a guess or a gut feeling. Those legacy Java codebases exist as a living proof of that. Other languages legacy code was either thrown away or frozen, unmaintained. And if you think "good, codebases shouldn't live for too long", well, that's a nice sentiment, but the fact is that long lived codebases save the industry a lot of time and money (even if young developers think they could have done it better if they'd just started from scratch). Again, we know that because we've seen the alternative.
BASIC and COBOL were never nearly as maintainable as Java (I know becas. I love Lisp (Scheme was among my first languages, and Clojure is my favorite application-programming language), but the point of a language designed for the industry shouldn't be expressiveness or power, nor readability, but usefulness and maintainability. Any other property should serve that. Professional software is written to serve a purpose -- they're not pieces of art (or not just pieces of art). You must remember that the average lifespan of a codebase is about a decade, and the cost of the project is spread -- unevenly -- over that decade. A language for the industry -- as well as other related tools -- is meant to reduce that cost.
I think Java does an excellent job of that, and I believe Clojure can do an excellent job -- we just don't have the data yet. But if you design a language for the industry, you must always look at the big picture -- at those ten years of the codebase as a goal -- that's the challenge. Coming up with a language that's powerful and expressive is easy. Doing that in a way that really serves the industry's need is much harder. Rich Hickey is a pragmatist, and he gets that. I think that the emphasis on transducers was a stumble, because he may have lost sight of the real goal.
