6a) .. because instead of using a language designed in part to address turnover and difficulty of keeping avant garde coders in banal and soul sucking jobs, the management passed the business decision of "what tools do we use?" to the said workers.
On the other hand, the language chosen might ensure high turnover of the sorts of coders you'd really like to keep, assuming you could ever attract them in the first place.
Management has choices here, including simply making the best business will stay good enough, or even if it doesn't, prioritizing keeping an essential core of its technology people short of anything but completely closing down the unit.
One thing I've noticed that's very common in the long term success or failure of high tech companies is whether they kept their core technologists. Compare Microsoft to Lotus and perhaps Ashton-Tate (dBase). Companies like banks may not appear to be "high tech companies", but their "production" is done with computer systems so....
> On the other hand, the language chosen might ensure high turnover of the sorts of coders you'd really like to keep, assuming you could ever attract them in the first place.
I'm sure Bell Labs never had that problem :) I was reading someone's PhD dissertation on Chill [1] and what was interesting to note was how many of the players ended up outsourcing their software development to 3rd parties.
Software is hard.
There other day I was talking with someone in tech management from a national chain (think malls). They are rolling their own commerce platform since "nothing" out there can satisfy their ecommerce requirements. I didn't say this upfront -- a potential client -- but the idea that this chain will actually get into serious software development, develop an A team, and keep them, is complete and utter wishful thinking.
My advice to anyone running a non-tech company is own the data and the apis and off load the rest to a[n] actual tech company.
These are both libraries that you can choose to use. Both Typed Clojure and Schema are more powerful than Java's type system. By powerful I mean you can declare types and constraints that aren't expressible in most type systems (eg: an object can not be NULL, or a Map or Array must have specific keys or type'd values). Schema is run-time (we only leverage it in development and testing), while Typed Clojure is more akin to compile time.
I'm ignorant here. From what you say it sounds like Typed Clojure is traditionally what we think of as 'types', whereas schema is more along the line of asserts.
Yup; but asserts that a) describe an entire 'object' (usually a map or list) b) can be composed together and c) be used to generate e.g. documentation or validated forms
By powerful I mean you can declare types and constraints that aren't expressible in most type systems (eg: an object can not be NULL, or a Map or Array must have specific keys or type'd values)
Hmm. Can it do anything better than that?
Java has optional nullity annotations that tools like Findbugs and (more usefully) IntelliJ can use to highlight nullity bugs. Kotlin has nullity integrated into the language's type system in a much better way. Stating that a Map or Array must have specific types in it is the whole purpose of generics, which Java had since 1.5, no?
Don't get me wrong. The Java type system is not that strong and has some frustrating holes in it. But typed arrays and nullity tracking doesn't seem like some advanced Clojure-only tech, to me.
If you're interested in learning about them, those two links are going to do a better job of explaining what both Typed Clojure and Schema are that I am across a few comments.
Wrt the typing of Maps and Arrays, what Clojure supports goes way beyond what Java's type system supports: you can specify that the first element of an array has to be of type X, the second is of type Y, the third can be either NULL or Z. For maps, you can specify that a key must have a particular type and must be in the map, with a specific value while other keys may be present (required or not) and you can compose any of the constraints mentioned on the values the keys may take.
This is different from Generics in that Generics constrain to homogeneity for an entire collection.
You could create formal objects in your Java code to express similar constraints, but you're not achieving the same result: a Java object actually has the property (even if it's null) while a map (or an array) with an optional member will only have it if it's present. With Java you'd need many classes to model all the specific combinations. Java's current type system is significantly less expressive and doesn't have the same power. I realize those are subjective. By expressive and power I am referring to whether you can declare an idea in your code or whether you have to write the imperative logic to implement an idea (the former meeting my definition of expression or power).
The Null annotations approach runs into problems quite fast, though. How about a list of values? How do you annotate that the values are not null? How about a list of lists?
Quickly you discover you need a type system. (I'm not familiar with Kotlin but I'm sure it's fine.)
The answer is that core.typed's type system is proper space-age tech, but it's also not exactly ready for production use.
> I always thought critical applications like banking needed compile time type checking.
They need quality, trustworthy software. Compile time type checking is a tool that can be used to help get there (but how useful it is in getting there depends on how robust the type system is, and Java's is not particularly robust).
Clear and concise code that is readily understood, avoids visual noise so that the programmer can focus on function, also is a tool that can help with that.
Clojure focuses more on the latter than the former, but that doesn't necessarily make it worse, and may make it better.
Technology adoption is almost always driven by the people, not the technology. The OP is a big fan of Clojure, having used it elsewhere, so its no surprise they brought Clojure to their new job.
To be clear, I'm not suggesting that Clojure is a bad language. I really enjoy using Lisp-like languages! I also enjoy modern statically typed languages (and on balance prefer them). My experience is that having good people and avoiding bad technology choices (e.g. choosing Ruby for high throughput and low latency environments) is the secret to success. Stories like these are invariably sold as technology successes (and, equally, technology is blamed when things go bad.) I believe the reality is more that better technology attracts better people. It's the better people and chance to avoid legacy that gives success.
Note that one of the largest and reliable systems -- smart phone to internet gateways often run Erlang, another functional and language with dynamic (but strong) typing.
So when you watch cat videos on your smart phone, there is about 50% some non-statically compiled functional code has done its job in setting up the necessary signaling.
WhatsApp has also recently been the poster child for running a successful distributed platform, with a tiny number of engineers (about 10 or 20) by also leveraging a non-statically compiled and functional language.
In general static type checking is nice, I like it. But clearly not a deal breaker. O often hear "I won't touch anything unless it has static typing" or "large applications should have static types".
But maybe the question to ask, should applications be that large? Lately we have been all upvoting microservices. And if anything those encourage decoupling large monstrous application into smaller independent services. At that point there are just a soup of dynamically typed components.
From what I've seen, Clojure has a strong type system, but is dynamic. These terms aren't orthogonal. If I understand correctly (+ 'a' 1) is not allowed. Immutability only strengthens it.
I ask this because I've been eager to get my hands dirty with Clojure. I have experience with other functional languages (Haskell, OCaml, F# and some Emacs Lisp), so the paradigm isn't new.
I've looked at core.typed, but doesn't seem as neat as the syntax of Typed Racket. It's something, though. Is it checked at compile time?
What irks me is that static typing and type inference can make code feel really robust, and for the lack of a better word, safe. In Clojure I see a lot of nice things but the lack of typing, though common for Lisps, always bothers me a bit. I don't like runtime errors that happen because the compiler wasn't able to tell me that this object doesn't have that method. JavaScript's undefined is not a function or its kin in Python are examples of this behaviour, which I don't like.
Though, I see that Clojure's answer to this is rapid REPL development--which is great!--and unit testing, and what I've recently discovered, pre- and post-conditions. But it still feels somehow inferior.
I am torn between learning Scala or Clojure. Knowing OCaml and F#, Scala doesn't look that interesting, messy and multi-paradigm. Clojure appeals to me because its a Lisp and has one paradigm, but on the other hand I'm scared by runtime exceptions.
> Should I just ignore my trepidations and proceed?
Yap give it a try.
From what I understand Core.typed while not as rigorous as static type system in Haskell or OCaml can let you gradually add typing to your application. They more you add the greater the benefit.
But also personally haven't used it.
I used Dialyzer in Erlang, which is a similar concept. You annotate your code with types and the more you do the more type errors it will find for you:
I think the WhatsApp Erlang angle is a bit overblown.
While they may have a small number of engineers (20 or max 20?), their engineers are probably laser focus in delivering the product (re: no web-app for the longest of time, no RoR vs Django vs Clojure vs Java vs PHP, no React vs Backbone vs etc etc), just back-end: Erlang, front-end: various phones that needed to be supported.
I would attribute their success at their skill set and experience. If you watch Rick Reed presentation in MeetBSD, a few key points to note: Rick Reed is very smart and knows full-stack (minus the browser): from the OS (internal, driver, etc) up all the way to Erlang VM and the WhatsApp backend.
The team had to modify FreeBSD and patch Erlang VM whenever necessary. How many companies have _that_ kind of talent? (e.g.: modify Linux and JVM to make them run faster? or Ruby implementation and PostgreSQL/MySQL while supporting the actual product?). Rick himself has tons of experience writing distributed systems at Yahoo! (using C++ nonetheless...).
While Erlang helps them but in reality, their skill + experience matter more.
Java doesn't have particularly good compile-time type checking. It's mostly there for performance rather than safety guarantees.
While Clojure is a dynamic language, it actually does have optional type checking, courtesy of the core.typed library. This isn't perfect, but it is considerably more sophisticated than Java's inbuilt type system.
Even without static type checking, I'd argue that Clojure is the safer language by default, since it mostly avoids mutability.
This. The immutable nature of Clojure is what, IMHO, makes it such a better choice than Java for me. I frequently see discussions/debates around type safety on HN, but in my experience, it has been the mutable nature of Java systems that has created more bugs than anything else. Clojure being immutable by default has drastically reduced the pain caused by these bugs. Clojure's collections, FP style, and meta-programming (via macros) all make it a great choice for me, but I think being immutable by default is the biggest thing.
The arguments I've heard supporting this are usually weaker than their proponents think because they simply don't understand the alternative. Even if this is true though, "banking" is much too broad a category to make such general statements. Consider:
> Here are some specifics – our new code is going to be an order of magnitude less in volume than the old and this is being conservative.
Java's static typing would help catch mistakes but it's not magic. And by moving this piece of their system to Clojure, they've managed to simplify it so much that there's much less room for mistakes to hide.
The verbosity of Java isn't necessary in all statically-typed languages. For instance Haskell is no more verbose than, say, Python, and has an even more expressive static type system than Java.
I didn't mean to imply that the verbosity/complexity of their original code base was caused by the type system. The article is light on details, so we can only speculate: maybe they'd be better off with Haskell but just failed to consider it, or maybe they considered all options and decided it would be easier to retrain their team and integrate with the existing code based if they used a JVM Lisp. What is clear is that they have a much smaller and more maintainable code base now than they had before, and so they're better off than they were.
Most of the times, they don't. Static typing helps with catching certain types of errors long before the compiled code arrives in production but it's just one tiny factor that contributes to overall software quality. Lack of expressiveness or lack of developer happiness can be much more detrimental to software quality than lack of static typing.
> Lack of expressiveness or lack of developer happiness can be much more detrimental to software quality than lack of static typing.
Except 5 years later, when the happy developers have gone on to express themselves elsewhere, and left behind them a mess. Now, you can make a mess in any language, but a mess in a dynamic language is considerably harder to refactor.
I also dispute the claim that a strong type system is a "tiny factor" in code quality. Being able to express invariants with types makes code much robust, and self-documenting.
However, I've also read studies that show the opposite, but the lack of rigurosity and the possible confounds that show up for both sides seems to render this as an open question.
> However, I've also read studies that show the opposite, but the lack of rigurosity and the possible confounds that show up for both sides seems to render this as an open question.
This doesn't look particularly convincing, indeed (though studies about programming languages rarely are).
A language having static or dynamic typing is only a part of the puzzle. Having tools that let you understand a codebase better through static typing is more important. With a statically typed language you have the compiler as a baseline, but it enables IDEs to do far more powerful things.
Hopefully they had tests. And unless they've mathematically proved the code always does what it expects (I've only seen that in avionics systems).
Also depends on application of course. Large concurrent and distributed applications benefit a bit less from static typing in traditional languages. Or rather, they are so hard, that type error are not as much of a significance. How concurrency, communication and failure is handled is more critical.
But say a game or a large desktop application with millions of lines of code, could get a larger benefit from static type checking, no doubt.
Now for a bit of personal experience. I have programmed in Java, C#, Python, Erlang, C++. I have found that when working with large or unknown code bases C# for example is great. Just having the IDE and generics support in C# during compile time is awesome. But if I program something from scratch, I can make a lot faster progress in Python. I often put more work into both unit tests and integration tests because I just have more time available.
Also failures during run-time, even in Python, in my systems a very rarely type errors (those are caught pretty early one). But they are often logic or concurrency errors.
> Static typing helps with catching certain types of errors long before the compiled code arrives in production but it's just one tiny factor that contributes to overall software quality.
If you use them right, expressing the constraints of your system in the type system, you can improve huge areas of quality. It doesn't give you much for free, but it does give you a tool that lets you check your own correctness properties more efficiently and maintainably than any alternative.
> Lack of expressiveness or lack of developer happiness can be much more detrimental to software quality than lack of static typing.
Agreed - but good static typing makes a language more expressive, not less.
Exactly. Not to mention that in any large scale system your functional and integration tests are the most useful for producing high quality software than simply having static types.
No-one is arguing that you should just have static types and nothing else. The argument is that static types and testing are better than just testing. After all, static types are a kind of test that the compiler verifies for you, leaving you free to focus on the rest of the tests. I'd say in any large scale system, dynamic typing is a risk (though there are ways of mitigating this risk, of course).
I second this. I'm currently creating a new dynamic website, which will see around 10k users a day.
We could have done it in a week with rails scaffolding capabilities and the vast gem environment, but we are doing everything manually in Scala, because type safety and performance. It's so frustrating that I'm looking forward to quit very soon.
The fact that Rails has scaffolding and whatever Scala framework your team uses doesn't has little to do with type safety (there are plenty of web frameworks in dynamic languages without scaffolding as well).
There are several languages which require a comparable amount of typing to a dynamic program, while adding considerable type safety.
The Play! Framework http://www.playframework.com has been pretty sweet for us so far and we use Java with it and that plays second fiddle to Scala users who really enjoy the full benefit of the framework.
I'd enjoy getting paid to go slow and learn Scala.
I really despise the "let's finish this in one week" that is typically going on for me. I'd rather learn best practices, even if they're in a language, and have some experience building long-lasting services.
Feels like the difference in working at a mobile home shop vs a custom log home company.
and Niall O'Connor from the bank (who spoke at PyCon IE 2013 - http://python.ie/pycon/2013/speakers/niall_oconnor/ ) confirmed via a comment on my post, that they were not "beginning" to do it, but that it had "already happened".
Edit: The reddit thread quoted by the parent comment, also mentions BoA and Niall.
Goldman's core risk management system (SecDB) is written in python, as is JPMorgan's Athena and BofAs Quartz. I believe the main people responsible for SecDB implemented Athena at JP and then Quartz at BofA.
And the whole system is in part credited with helping them survive the recent financial crisis ... although simply listening to their risk people was more important. But it's all intertwingled, I'm sure, because good risk people would want a system like SecDB so they know the company's positions in quasi-real time with a great deal of assurance vs. hours and days and through e.g. scraping spreadsheets, often manually, etc.
I'd feel better building complex financial system in Clojure rather than C# (the 2 languages I'm most familiar with at this point).
With contractual based programming, pre/post assertions, schema validation, etc... you can add all the type checking you want. That along with the other well known benefits of functional programming I think would make it highly desirable to build large complex systems in.
I always thought this was what SPARK/Ada was for, along with one of the real-time OSes. But it turns out that languages with the stick that far up their backsides are only used for actually critical applications, like air-traffic control, medical implants, crypto libraries, and spacecraft/satellites.
If a banking program fails, they can just ask for a mulligan from the central bank/government. They're just playing with people's life savings, not any actual lives, right?
If they're going to suck up all the best brains in the industry anyway, they ought to be able to do SPARK/Ada.
If I were making banking/trading software, I'd start with an OS that combined the security paranoia of OpenBSD with the deterministic performance of LynxOS, build some tools that validate source code by automated proof rather than testing, and hire a bunch of people smarter than me to build software that is secure, reliable, and profitable, in that order. No bank in existence would be crazy enough to pay me to try.
Above a certain level I don't think your thesis is currently valid.
For a "bank bank", "playing with people's life savings", the lower level transnational stuff, yeah. Although I don't get the impression that the current stuff doing that is prone to disastrous failures, and I believe the system has some slack here and there to reverse errors, i.e. transfers of money vs. buying and selling financial instruments. The current infamous failure in this area, which was WRT to the latter, was at Knight Capital.
But above a certain level, at which point we're not so much "playing with people's life savings" (or, rather, if they have a clue, only a small portion of them), the situation can be a lot more fluid. E.g. changing trading strategies, which can be driven by the capriciousness of governments. The current state of the art doesn't make it practical to embed the US tax code into source code amenable to automated proof, especially before it changes again, right??
I've been trying to come up with a common definition for language safety, and in scoring both C# and Clojure, Clojure came out a much more "safe by default" language.
Ok, then use Prismatic Schema & core.typed. Not to mention other tactics to help correctness, like simulation testing with Simulant. (Or what don't you like about them?)
One of the reasons I went with Scala instead of Clojure is that you don't "just have to make a jump". You can write one class, in code that clearly corresponds 1:1 to what you'd write in Java, and the interop is smooth enough that other classes don't notice that this class is in Scala. Later on you (hopefully) are writing a very different style of code, more declarative and immutable and putting much more of the business logic into the type system. But it doesn't have to be a big-bang switchover; you can be productive from day 1, hour 1 even.
I think you missed the overall point of the author, which was that in order make a significant improvement at all, you need to make a radical shift in the modelling of the system. One of the problems mentioned was the inherent complexity of using a type system for modelling data. Scala is very type oriented and so as a result you will end up with a lot of types for everything there as well.
Yes it's certainly easier to make a transition into Scala, but I don't think using Scala will solve the actual problem, even if it does have FP capabilities and encourage immutability.
As I read the article, though, it's not just types. The fundamental problem was that nobody there had a good idea how to architect a Java application that large. So they threw out the Java, and replaced it with Clojure, which they had even less experience architecting. Sure, Clojure may have resulted in 100,000 lines instead of a million, but that's still a large enough system that architecture matters. Did they create another large mess? It seems reasonably likely that they did.
Java's types aren't broken enough to force you into a terrible architecture. Trying to use them badly (that is, in the way that everybody seems to think that you're supposed to use them for enterprise apps) seems to be the actual culprit.
What I found is writing Scala that is not striving for ~100% immutability and functional style does not bring much benefit. It is just a different syntax, that is nicer than Java on one hand, but also big and not straightforward on the other (in the C++ sense, so 'Scala' does not have a conventional coding style, it ranges from Java to Scalaz). Also, making core 'half-functional' does not bring any benefits in terms of composability, concurrency, reasonability (about the code).
All the functional goodness of easier to reason about code. A much more concise way to express common patterns, and a type system powerful enough to abstract away things that you have to repeat in Java. More dynamism than is possible to use safely in a dynamically-typed language, because I can write code that simply "does the right thing" in a principled, safe way that the reader can confirm by checking the types, whereas in something like Python it would be impossible for a reader to see whether that code was buggy.
> It is just a different syntax, that is nicer than Java on one hand, but also big and not straightforward on the other
Actually the syntax is often more straightforward than Java's. E.g. {} just means a block in a sense that's handled consistently everywhere in Scala, whereas in Java {} are required on a try/catch or a method but optional on an if/else.
> making core 'half-functional' does not bring any benefits in terms of composability, concurrency, reasonability (about the code).
Of course it does, or at least if it doesn't then the whole functional programming effort is a mistake. Any "pure" core is a) relative to some notion of equivalence that might or might not correspond to the things that are important to your particular problem and b) doesn't do anything without an impure interpreter. See e.g. http://conal.net/blog/posts/the-c-language-is-purely-functio...
I would argue that Clojure-Java interop is better than Scala-Java. E.g. Clojure's collections are Iterable, can be used in Java without conversion. 'reify' and protocols are other good things in this area.
Keeping old programming style sure allows to lessen frustration compared to diving right into working with immutable collections but the difference won't be that big if you are learning anyway.
This article conflates OO and strong typing a little too much. These are too very different things, but agree that the combination can cause problems, not to mention that so many Java architectures are pathological in terms of struggling to find the code that actually does something.
It would be great to see examples of the design/modeling of the system. I really like functional languages have done since my university days, but there seems to be a dearth of examples on how to actually lay such large projects out in Clojure/Scala etc.
Having something like the J2EE Petstore (for all its sins) would go a long way to allowing more projects to make the switch than more articles on how great things are.
> when you have REPL you are TDD’ing in a different way as opposed to constructing persistent unit-test classes that could potentially form a layer of cement.
Lately I've been using the console for development in rails and javascript more than anything and it is a really good way to iterate on a solution quickly.
Especially in a functional language, if you architect your code well, e.g. avoiding large (lengthy) functions. You write a function, test it in the REPL, and forget about it. Until, of course, someone hands it data you weren't expecting. But here Clojure helps you a bit with it's effort to provide consistent interfaces to its collection types, e.g. lists, vectors and maps. And the most expressive and fast to write code in, the more time you can devote to testing.
I'll bet in practice this sort of TDD works very well, as long as your coders are diligent. For the projects I've done this way, including a couple in C using a very good but now defunct interpretive environment for it (CenterLine's CodeCenter/ObjectCenter), it's been superb. In an extreme case, I spent around a week programming in C the engine of a high speed document imaging scanning system, at which point I got the driver that let me speak directly to the scanner. The first time I tried it end-to-end it worked ... until I ran out of file descriptors (had forgotten to close() each file after writing out its contents).
There was of course a whole lot of experience that made that feat possible, but I found it very impressive how the environment allowed me to write code with so few flaws (that was, in fact, the only major one I can remember once I declared a function finished and ready to try out for real).
1) Bring in new blood to improve a legacy platform
2) The new blood decides to throw out everything and do a complete rewrite
3) The new system is super successful because it can focus on a much smaller subset of problems
4) The bank runs into profitability issues and can no longer pay developers well
5) The good developers leave en masse before everything is completely rewritten.
6) New developers come in and spends the entire time fighting fires and patching the system instead of building new stuff.