For those short on time: "It doesn't go as planned" is explained in the middle of the article:
> When everything compiled, I shipped it. And to everyone's surprise, apart from a few bugs I had created while rewriting thousands of lines of code... it worked. It just did. No explosions, no obscure bugs, no memory leak, no performance degradation. That was rather unexpected.
...followed by
> Then we saw the JVM CPU usage rise to alarming heights, with unusual patterns. And no obvious culprit in the thread dumps...
I don't think there's anything wrong with an evocative title. In my mind, clickbait is when you never actually get what's promised, or it's vastly oversold. This title tells me, "we had a deploy that caused problems for a surprising reason," and I feel like needing to retune the JVM delivers on that. It's a weird, surprising issue. I'm sure there are old-hands who have deployed JVM apps at scale might be unsurprised and unimpressed, but I was surprised.
I think showmanship is okay if it's tasteful. The problem with clickbait is that it's manipulative and tacky and dishonest. After you read the article you feel like you've been had.
>In my mind, clickbait is when you never actually get what's promised, or it's vastly oversold.
Clickbait is trying to bait you into clicking. This definitely falls under that. Is it the worst case of clickbait I've seen? No. But it is definitely clickbait.
By this definition any catchy or interesting title is clickbait, regardless of how descriptive or accurate. "Nixon Resigns" will certainly bait me into clicking the article for more details, even though it's basically leading with the punchline. For the term to not be meaningless I think some level of deception or withholding important info has to be involved, "Nixon Made a Shocking Decision....You'll Never Guess What It Is!!!" is clickbait.
Based on your first example, I know what happened and can make an informed decision about whether I need or want to know the details of how it happened. I'd argue that that is not click bait, as it leads with the most important piece of information first. The title could further be improved by writing "Nixon resigned because of XYZ", but that would be nitpicking.
The most beneficial title from the reader's perspective is a title which allows the reader to quickly decide whether they need to know the information and informs the reader with the necessary information as efficiently as possible. Hence the now long dead practice of writing in the form of an inverted pyramid (https://en.wikipedia.org/wiki/Inverted_pyramid_(journalism)).
The inverted pyramid isn't dead. Blog posts are more like feature articles. Feature articles never followed the inverted pyramid.
Do you feel this article did something wrong? Because from what I'm hearing it seems like you're saying it wasn't to your taste, which is entirely valid but a different matter.
I'm making an argument that the article is click bait, which it is. I'm not sure where you see a value judgement about whether click bait is wrong in my comments, or why you are defensive about it on behalf of the author?
I don't like click bait. I voiced that opinion. That's all.
> I'm sure there are old-hands who have deployed JVM apps at scale might be unsurprised and unimpressed, but I was surprised.
I've never done anything nontrivial with the JVM, so I'm totally willing to believe I was surprised because of my JVM ignorance, but I've never had to fiddle with my language VMs.
Regardless, I argue there's room for tasteful showmanship without it being clickbait.
Yes it's clickbait, so I didn't upvote it; an accurate title would have been "Lessons learned upgrading Lichess from Scala 2 -> 3 and the need to tune the JVM". A drama-free title like that would have been fine, and I would have upvoted.
Personally I'd never click on your proposed title as I have 0 interest in Scala, but I clicked on this one and was pleasantly surprised by interesting story. Making everything sound robotic is not the way.
Do you agree that title is clickbait? You are ok with titles that misrepresent or exaggerate to boost clicks (even if most of those viewers bounce upon discovering that)?
But the upgrade....didnt go as planned? In what way was it a misrepresentation? It was certainly generic but definitely not exaggerated. Felt fine for an editorialized title
INNOCENT dev did THIS, and what happened next was <truncated text, click to follow article, followed by being bombarded by 20 ads which magically bypass ublock origin>
It sounds like clickbait when read in the Hacker News context. But if I'd come across this in the Lichess blog directly, I would've immediately known it was kinda tongue-in-cheek or at least nothing serious. Context matters.
Yeah despite being an interesting post about Scala, I came for the harrowing tale of a big deployment gone wrong. Downtime, angry customers, seat of the pants debugging and straight to prod gambles to save a sinking ship.
Instead, a smug tale of a huge upgrade that went unexpectedly smoothly - with a performance problem swiftly solved. Still a good story, and kudos to the team. Just not the horror show I came for.
Is it clickbait when the top contributor/creator posts on his website? The same website that makes a strong promotion against ad tracking: https://lichess.org/ads
I clearly don't have the same definition of clickbait.
it seems there is no universal definition of clickbait. So I based my comment on this:
1. the article was posted on Thibault's own website that doesn't generate revenue per click. Isn't clickbait associated with the possibility of additional ad revenue?
2. Thibault recently called for help on the issue. This post recaps what he went through. He clearly expressed that the code upgrade process didn't go as planned for him. the post and sub-title feels genuine.
3. Clickbait (to me at least) indicates that there's either intent on generating revenue or self-promotion. I fail to see those conditions here
So that's my logic. So why would this be considered clickbait?
Clickbait is a sensationalized or misleading title meant to draw extra attention (in the form of clicks).
Both "big upgrade" and "didn't go as planned" are technically true, but overpromise. A big upgrade could be new features, improved UI, something that affects the user experience.
"Updated Lichess language version, had to tweak cache settings" is a title that isn't trying to sell itself.
I noticed this as well. There was no integration testing? No manual testing?
I don’t pay for Lichess, love the UX, and I know it’s a labor of love, but come on. Surely Lichess refactoring aren’t so urgent and user critical that they should just go straight to prod?
Yeah, there should be some kind of shakedown test.
But Nah, not every issue can be tested for within the manpower/$ budget. Issues of load being one of them.
A valid strategy is: send it!, work with the beast a bit, if too unruly roll it back. In this case, they tamed the new beast. Not like Lichess is going to lose billions...
Load testing would have to be quite advanced to find this, doing more than just simulating users for the most common features. It took more than 24 hours of real-world usage to visit sufficiently many niche code paths to fill the code cache.
Lichess being a project that is 85% written by one guy, and all other changes approved and read by him, he may have felt confident (until now) that he could maintain all the complexity of the software in his own head, and recall what to do and where to do it whenever he made changes. Seems bonkers to most of us writing software in teams, but compare this choice of workflow to ones made by others in similar situations: programmers in academia, who often work entirely alone, or Linus Torvalds in earlier versions of the kernel. Skipping testing is a common choice, especially when working alone and with a typed language. I could see Thibault bringing tests in now however. They would not have caught much in this case, because it seems like the problem came to light only when the software started running with real users.
If I understand correctly, not being able to tune .NET is a shortcoming. It is like having a JVM which does not have tuning options, it works optimally only for some workloads, and fails severely for others.
I mean, I'm not a JVM fan, but this sort of thing happens in lots of languages, and even operating systems. Upgrade from version X to X + 1, especially with lots of changes in everything else, and you're bound to hit different limits or explore the interface with limits you were hitting and didn't realize.
The other option is to never update anything; which is valid, but painful.
Of course _real programmers_ write forged microcode to define just the machine code instructions they need, and they flash the ucode update live in prod to deploy their app.
But if you're weak willed and need some hand holding, the official ISA's machine code is tolerated too.
Shit the ROMs on the Apollo mission were hand-woven wire looms. 4K wires weaving between and around magnets. Why hold a magnet when you can use a marlin spike?
Pathetic. If you can't fit your code into L1, are you even a real programmer? No one has time for your Electron shit. Get real, bruh, what's your deadlift?
As an anecdotal case, a company a friend of mine works at has a backend workforce of primarily Java-only devs, so that's the only thing they can really maintain. There's a lot of sunk cost and inertia on the enterprise side when it comes to Java.
But yeah for starting something new, with an open language choice? No way anyone sane would go for it these days. Especially after Oracle basically closed sourced it, making JRE distribution illegal.
> We want types, not boilerplate. Sometimes it's best to let the compiler figure out what things are by itself.
At first I thought type inference was cool too, but more and more I don't like it. I prefer being able to see a variable's type easily, even when skimming over code. Type inference makes code less readable to me.
Type inference locally is great. Doing `val something = MyClass()` or `val something = someFunc()` . No ambiguity, just less typing and clutter. At bounds however (function input/output for instance) I prefer explicit types, less chance for mistakes and the signature acts as documentation to what the function does.
> Type inference locally is great. Doing `val something = MyClass()` or `val something = someFunc()` . No ambiguity, just less typing and clutter.
As much as I can get behind val something = new MyClass(), where the type is explicitly mentioned on the line, it's a whole other story for val something = someFunc().
You have no way to know the type of that variable without looking at the function itself which is more work than should be needed.
Static typing removes freedom I am addicted to with duck typing, however, I only ship TDD code which also serve as coding documentation. Both are protective but if you're not writing tests then by all means use static typing.
I'm finding this hard to relate to. Our circumstances and experiences must be pretty different. But I'm afraid to me that sentence reads a bit like "I have a lathe so I don't need a bicycle". You or I might not do bicycle jobs, or lathe jobs, but it's a bit of a stretch to imply one renders the other irrelevant.
I find it interesting that you characterise static typing as about safety. I think it's more about communication. (for me this includes "communication to myself when I wrote this seven years ago and didn't think about it since", which I appreciate may be a bit of an edge case). Tests are also a great medium of communication, but a different one - in which case the metaphor becomes swap a sonnet for a sonata? I'd like both please, but maybe not always at the same time :-D
Duck typing (structural typing) is orthogonal to dynamic vs static. Typescript being an example of a statically typed language whose type system is centers on structural typing
> if you're not writing tests then by all means use static typing
I know you're not saying this, but your wording implies it to a bit... Tests to cover <the things covered by static typing> are only one of the uses for automated testing. There's plenty of places (most of them) where automated testing is just as useful with a statically typed language.
Type inference everywhere can make code less readable, but type inference when it's clearly the code author's intent that the type of the variable should not be of much interest to anyone improves readability.
A common example of this is a variable that's initialized in one line, fed into another function in the very next line, and then never used again. [1] If the function call that produces it, and the function call that consumes it is named well, the variable's type is often, but not always superfluous.
It's like naming for loop variables. Sometimes, you want a specific, detailed name for them. Other times, you just call it 'i', because what it is... Is not very important. It's a counter, counter goes up until we hit the end of a list, don't think about it too much.
[1] The two function calls could have been mashed together, but that would make the code less readable. The full variable type could have also been included, but its not very relevant to the intent of the code - which is simply[2] feeding the output of one function as an input into the other.
[2] Obviously, when contextual information (like real-world side effects of creating a particular variable - say, a thread, or a record...) that complicates the "Create -> Use" case is relevant, or if the APIs in question have a lot of function overloading, or poorly-named interfaces, or if non-trivial casting is going to take place, it may still be beneficial to explicitly provide the variable type.
Type inference w/ overlays is usually enough. The only thing I’m missing with VSCode rust overlays is they sometimes don’t show the lifetime on an inferred type which makes it a little more annoying.
One of the things that's made me the most frustrated learning rust is all the type coercion and inferencing. Sometimes code will just work when I have the wrong mental model because types are coerced but then it ends up biting me later when suddenly everything breaks because it can't coerce anymore and I find out nothing was the type I thought it was. I wish there was a way to turn it off while I'm learning just so I can understand what's really happening.
Nobody forces you to rely on type inference. It’s perfectly valid Rust where you type everything if you so wish. This might actually be a good way to learn the ins and outs of the type system.
EDIT: Also, what do you mean by type ‘coercion’? As far as I know the types in Rust are never silently coerced to anything. They just are what they are.
wow, I never contemplated before whether that kind of language design could be evil, but you make me think it might be. Maybe not in the ordinary course of sensible use, but in some horrible "I didn't think of that". Then I feel a bit hypocritical because I'm sure I've relied on a hundred more problematic abstractions without worrying about it.
Far from evil, it's amazingly useful, and is why things like `let x: Vec<_> = foo.collect();` work (the return type of the `collect` method is generic). Rust stops type inference at function boundaries, so the precise type of something is never far away if you need it.
I concur. Rust does it very sensible and elegantly. I will note that the other thing rust does is let you under specify the lifetimes of a function signature but when compile errors happen they tell you what they ended up inferring and you can always correct it when you need something slightly different. The two features are critical in making Rust code easy to read and write as the types and lifetimes can get tedious and most of you have to specify it literally everywhere.
True, although coercion in Rust is quite limited (it doesn't even coerce smaller ints into larger ones), and the coercions that exist are pretty harmless IMO (like letting you call a function that accepts a `&foo` if all you have is a `&mut foo`). I suspect they're referring to something else, as I've never seen coercion highlighted as a pain point before.
You should consider configuring your editor to display types inline, it's awesome. It'll provide annotations next to the code with relevant types, so you don't have to click or anything to display the type. I found it also sped up my learning process by increasing the bandwidth of the feedback loop between me and the language, so I developed an intuition for type inference rules & the return types of stdlib functions faster.
Most editors (or plugins for them) that support a LSP, like rust analyzer[0], can do it. This includes afaik (neo)vim (via coc.nvim), vscode (via rust-analyzer extension), emacs and Clion.
Thank you for pointing these out. I personally use coc but vim lsp was also a good experience. This lost is by far not complete as many other editprs also support LSP servers, like helix.
It could, but then you put off grumpy people like me with old editors that don't do that kind of thing. Plus you need fancy tools to look at patches. And none of it is going to work inside tokio's select macro (or other useful macros), which could be a lot of your code, depending on what you're writing.
I'd rather work in a language where nobody has type information. It's a much more even playing field. ;p
IMO a decent editor is a must for statically typed languages - it's such a productivity boost to navigate by type information, type safe refactoring, etc.
Main benefit of static typing is enabling tooling to reason about your code - not using IDEs is throwing a huge chunk of it away. Historically it was easy to get projects that would be too large for real-time tooling - but these days the IDEs got better and you can get 32 or even 64 GB ram into a workstation trivially - I haven't seen a scenario like that in years.
I've also noticed GitHub supports type navigation in some languages, but yeah for nontrivial reviews I'll usually do a checkout anyway.
I use joe. I've been using it for 25 years at this point, and it works enough, and I'm not interested in changing. About 15 years ago, I upgraded to a version of joe with syntax highlighting which was pretty cool, but I'm not much interested in more than that. I do occasionally poke around with things in fancy IDEs (it's easier to fiddle with Arduino projects with an IDE than otherwise, because nobody writes about how to do the pieces individually... but maybe I can drive platform.io from the command line in the future), but I'm much happier with a simple as dirt editor, and another terminal to run Make in (or whatever, using Rust for work now, so running cargo instead of make).
I get what you mean but it does come across a bit as "why would anyone want C compiler, ASM provides a much more even playing field". All of the "old" editors that people still use have LSP plugins these days, so feel free to treat yourself any day now. :)
Non-sarcastically: type systems have their place but are not quite there yet. I say this as someone currently doing Advent of Code in Haskell, which surely cannot be accused of being lacking in type safety. What I would love to see in a type system is the flexibility that (for example) Rails' ActiveRecord adds to a class while still being type-safe. AFAIK, no such system exists (yet) but I would love to be proven wrong sometime in the future.
Rather than turn off type coercion and inference, it sounds like you want a way to see the coerced/inferred types right in your editor. Rust Analyzer does this for you — there's a plugin for VSCode that will show inferred types as inline hints in your editor next to each expression, so you're never left guessing.
Personally, I agree, Rust is nigh impossible to write without types in your editor; thankfully Rust Analyzer provides them.
I already know Rust, and unexpected implicit refs and derefs in match/if let expressions or method calls still trip me up when writing or reading new code (especially now that match ergonomics can make `match &Option<T> { Some(x)` compile but not the way you expect). Nowadays I avoid higher-order generic functions like .filter(|...| ...) so don't hit unexpected &&T etc. there, but matching on Entry and regex searches still trips me up.
My IDE can show me inferred types as an overlay in my editor (even in languages with dynamic types like Ruby, jetbrains inference is pretty good when the code is simple), perhaps look for an extension for your preferred editor?
Why Github? I'm assuming you mean their code review interface, which is so incredibly lacking I don't even bother with it anymore and I just pull the branch locally and use my IDE.
class class1<T extends Comparable<T> >
implements SmallestLargest<T>
T[] values;
This is a spot where inference doesn't work, but has the same problem as the relatively tame example you used.
The solution here is not to eliminate the type signatures. The solution is not to wait until compile time to figure out the type of a thing. Nominal typing, as in named types, is the solution here. If you have a 'thing', which also happens to be an application.someHierarchy.someLongAssPackage.someLongThing implements application.someHierarchy.someLongAssPackage.someInterface, anotherLongAssDefinitionForTheReturnTypes but that's an implementation detail, not a type description.
It's an Address, or a customer impression, or a similar product, or a fuel tank. Just fucking call it what it is instead of pretending like you're working on your PhD thesis. Your coworkers aren't robots, or academics, they're human beings who've got real work to do and kissing your architectural astronaut butt/complementing your farts are not on the list. Name things like a human. A wise human, but a human.
Not really, that is solved with aliases. The problem really comes to life when using strongly typed generic functions that must be generic over their return type. This most notably occurs with Rust iterators. Try writing out the type of this thing explicitly:
let pos_nums = v.iter().map(|s| s.parse::<u64>().unwrap()).filter(|n| n > 0);
Hint: you literally can't in current Rust because closures have Voldemort types, you can't name their type. And even if you could, the resulting type would be horrendous.
Explicit types are useful in some places. Type inference doesn't mean you can't use them, just that you don't have to spell them out everywhere even when they aren't useful.
yeah, to be fair I only know it through the C++ `auto` keyword, but I dislike its overuse. It's great for weird templated iterator types but I prefer to see the concrete type if it's a reasonable length.
With C++’s implicit conversions, it makes that less of an issue though. Like a function returning long and assigning to an int “just works” and can silently trunctate. With auto, both are long
Is it a bug if the compiler thinks that maybe the value could grow too large to fit in an integer so it promotes it to a floating point type. But then you compare it to a value that is converted out of a string of ASCII digits and the comparison is off by 1 because 4 != 3.99999999996 and this causes your code to iterate one additional loop which doesn't cause an immediate problem except about 1 in every 256 times it causes a boundary condition where the next packet will have a corrupt first byte?
Technically the compiler technically did nothing wrong, but you are still left with a head scratcher because all of the logic seems sound and yet it produces occasional garbage.
Scala's compiler used to widen prematurely and apply implicit conversions similarly to what you described.
Scala 3 doesn't do that [1]. The new inference algorithm will, however, prefer a generic method over the same named method without a generic argument in scope when no typing hints are present in the source code when there are two possible matches for an inference completion. This can then cause problems when the generic expects some kind of given parameter:
def foo: String = ...
def foo[A:SomeRequiredThing]:A = ...
//in some other place
foo shouldBe("hello") // foo[String](using SomeRequiredThing[String])
// gets preferentially selected by the scala 3
// inference algorithm, which can lead to surprising
// though not "incorrect" results [also 1]
What compiler does that? What you're describing sounds like a weirdly designed type system, not a compiler that implements it.
That said, this is an example of the compiler being wrong.
If a value is changed to a data representation with different comparison semantics, and later compared to a different value of a different representation, it means the original optimization (hand wave over the definition of that) was unsound in the first place. AOT compilers would classify this as a bug that needs to be fixed imminently, and JIT compilers that do this kind of thing use something called dynamic deoptimization to remove the change once its been invalidated.
Yes, promoting an integer to a float is 100% wrong and should result in a compilation error (if you dislike overflows) or an integer overflow (if you dislike compilation errors) or even a promotion to a BigInt type (if you dislike both other alternatives). Promotion to a float is always wrong.
> Technically the compiler technically did nothing wrong
I don't know a single language where integers are "promoted" to floats when they get large; certainly no statically typed language. You could argue JavaScript does this, but it's neither compiled nor does it actually have integers. So yes, if a compiler did that it would be a bug.
type inference isn't magic. It would absolutely be a bug for the compiler to change the type of a variable from an int to a float because it thinks there will be overflow.
There's a lot of ignorance in this thread around the JVM, which is forgivable I suppose since it's not the hot new thing atm :)
The parameters they are tuning are available in any good VM with a JIT. You won't see these tuning knobs with Go because there's no JIT. These are trade offs. Scala is also known to create more garbage than Java.
The reason for the CPU spikes are code is getting compiled via JIT which takes CPU. If isn't enough space for JIT compiled code then you'll be recompiling code often, burning CPU.
This is similar to database cache tuning. Too little cache and you burn CPU and IOPS. Too much cache and you don't leave enough memory for other operations.
These parameters have good defaults but large apps can surpass them. It's important it's not automatic as this could have dire consequences (oom etc).
> If isn't enough space for JIT compiled code then you'll be recompiling code often, burning CPU.
This sounds like a bug to me. If you are out of space for compiled code stop compiling. Or at least start slowly raising the threshold at which you recompile something so that the compilation cost is amortized over time.
If your recompiling is using more CPU than was being used before it started something is wrong. You should only be compiling code when it is expected to save CPU long-term.
> To be able to tell if Scala 3 itself is faster, we would have to rollback to Scala 2 and try it with the proper JVM tuning. I'm not willing to do that, sorry! Once you've tried Scala 3, there's no going back.
Pardon my ignorance. Why can't you set up a performance test in both environments and measure the speed?
Setting up reliable and reproducible performance environments is as much work as running good quality prod systems. It requires monitoring thing rarely available to monitor, extending prod metrics, creating snapshots, running repeatable tests on different scales. This isn't work for a single person. I'm in a team of 7 working to acquire next customer who will be on average 5x bigger than peaks of our current biggest customer. We've been designing, configuring and creating metrics on EKS since September, alongside training devs and developing performance engineering framework in k6. Lichess doesn't need that much effort and accuracy, but it's still lots of work. Did you know Fargate will schedule you Intel Skylake or Broadwell and there's even 30% difference in performance between them on certain tasks?
> Pardon my ignorance. Why can't you set up a performance test in both environments and measure the speed?
Everyone has a performance test environment, not everyone has one that's separate from production. In this case, generating realistic test patterns is probably very difficult, so you get one environment for everything.
I think it's a matter of effort. Many performance bottlenecks only really show up when you scale up to massive levels. While it's possible to replicate that on a test environment, it takes time and effort to set that up. While doing that is worth it for some applications, I can see why the Lichess developers didn't want to bother.
Time, effort and spare resources that Lichess had better allocate somewhere else.
If Thibault enjoys Scala 3 and feels more productive, while the overall performance on the same hardware is approximately equal, it's already a win for the project.
> That's where things got a bit hairy. Lila is built on Play Framework which is not yet ported to Scala 3.
> So I forked it and butchered it to remove everything we don't need - which is actually most of the framework.
I guess there is hope that Play framework itself will be migrated to Scala 3 and that the dependency on the fork can be removed, but this is taking on a risk - what if there are security updates to the upstream in the mean time?
I think the plan might be to get rid of the play framework altogether and rather use a couple small, independent libraries to achieve the same.
The likelihood of getting security fixes in the future is about the same as getting new security holes created by whatever updates. Butchering away everything they didn't need certainly didn't harm security.
Play has been donated to the community one year ago, which has been reasonably active lately. They secured some funding and development efforts have picked up a good pace.
Is there any other runtime that needs as much tuning as the JVM?
It seems like a failure of the JVM that someone capable of porting thousands of lines of code with little trouble then fails to tune the JVM. It's like, "we wrote all the code, now the hard part starts, tuning the JVM".
I took just the opposite lesson from this: the JVM offers enough turing knobs that in the rare, extreme cases where the defaults don't work for you (I suspect few single-server workloads in the world have the combination of complexity and request rate that Lichess does), there are still ways to do something about it. If he'd been using, say, Go, he'd probably have had to give up and roll back, or patch the runtime code.
I’ve been using Go for 7’ish years across a large codebase, comprising multiple services interacting with millions of people.
Every release Go gets better and requires 0 changes to the code. I have never needed to fine-tune the GC. I have never needed to spend a month rewriting my code to work with Go 2.0. I have never been nervous to update to a new Go version. I write the code once, and it runs great in prod for years. I love and value these things. I also suspect that Lichess’s use case would perform extremely well in Go out-of-the-box, since it’s just a web app.
I certainly hope the JVM team would like Lichess and other web apps to run well without needing arcane configuration knowledge gathered over years of experience and battle scars in production.
> I’ve been using Go for 7’ish years across a large codebase, comprising multiple services interacting with millions of people.
> Every release Go gets better and requires 0 changes to the code. I have never needed to fine-tune the GC. I have never been nervous to update to a new Go version.
I've had the same experience with the JVM, over a longer time period. You hear about this because it's exceptional and interesting, not because it's the norm. "Just a web app" is an incredibly reductive take.
The same problem exists in the Go runtime (it's fundamental to the problem space), and since they don't let you tune it presumably they either guess or hardcode what the value should be; good luck if they get it wrong. Sure, you probably won't hit it with your code - the overwhelming majority of Java users don't hit this kind of problem with their code either.
A huge spike in CPU usage would be treated as a regression in other language runtimes and would likely be addressed. The fact that this is solvable with advanced JVM knobs is both good and bad. Good in the way you say. Bad because the complexity of maintaining all those knobs makes it difficult if not impossible to improve the runtime defaults, and every runtime problem becomes a tuning problem.
Perhaps it is different now, but I've always hated how to figure out how much memory a java app needs. You can certainly give it 30GB of ram and it will happily use it all up and then start making garbage collection calls. But does it really need all that ram? I think the best practice of the time was to continually lower your max heap amounts until you started getting allocation errors, then bump your number up by 20%-50% (or something like that).
No, that's not now you should tune the modern JVM.
GC will always happen no matter how much heap size you allocate: large heap size will make GC happen less frequently, but depending on the algorithm, it will also increase how much time is needed for each collection. The key point for GC tuning is to keep total GC time and pause time under control with as little memory as possible.
* First, you have to setup monitoring for the garbage collection time: turn on metrics collection and details garbage collection logging.
* Second, tune your total GC time so that it's under 5% or less. Start with a reasonably small max heap size, says 256MB, and keep increasing it if the GC time is still too large. Try to keep the max heap size under 32GB to take advantage of "Compressed OOPs"
* Third, you only need to use more advance flags if you detect large GC pause in your GC logging. Otherwise, you're done.
I write Java for 15 years and the only knob I ever had to change is Xmx. I think that I also had to adjust permgen size in old JVM for development, it's not needed anymore. My apps are not world scale like lichess but small country scale.
Java was the first mainstream garbage-collected language. Not the first GC language, but the first one to get serious traction. It started the post-C++ era.
That was a pretty bold design decision for the time, and one that worked out. The VM was another big one, and it also worked out.
I did say "almost" every single language decision.
GC wasn't a failed experiment. OOP extremism was. And the latter is a bigger Java standout than GC.
Python predates Java and is still mainstream, and while (mostly) not garbage collected, but reference counted, language design wise it's not a big difference. There have been GCd Python implementations, proving that.
Being successful is not really a language design choice, so I wouldn't count that.
> The VM was another big one, and it also worked out.
As I explain in the blog post, the VM was a colossally bad mistake. The inspiration for the VM sounds like it was better. So looks like they took the ball and started running in the wrong direction.
3. Bad API is not being actively deprecated. StringBuffer, Vector, Date, Calendar, File. Replacements exist, but old classes are still in core.
Also I think that Java community is crazy with their love of Spring Boot. Like Spring is not magic enough, let's put more magic to autoconfigure its magic. But that's not a failure of Java.
Also, JAVA rise to popularity coincided with the rise of "design-patterns-everywhere", nothing against them where they fit. But there was a time where 'design-patterns' were the true-gospel and Java was where you go to design-pattern-church and practise your religion, everyday, everywhere !
Are you running the whole lichess on one machine, or is this one shard only? 30 GB RAM for one instance of application seems very high.
(sorry did not have time to read the whole article yet)
Would be interesting to see if this application would benefit to switching the JVM from openjdk to Azuls “high performance” JVM.
Rather than use a big bang deployment. Setup an A/B test, and gather a sampling of data.
I haven’t been able to truly test azul’s claims yet. Java apps that I manage/deploy were on a small scale in regards to requests/sec and I personally didn’t see much difference.
Azul Zing is quit expensive at minimum $15K/year. While Lichess might be able to get it for free under some promotion deals, it would not make sense to tune an OSS projects for such an expensive option since most users won't be able to use it.
Got sucked into reading the article because of the "didn't go as planned" but it left my thirst for failure stories unquenched. Should have known.
That aside, that thing with opaque types in Scala sounds interesting - but how would the compiler know that the parameter was a UserId and not just a random string?
You have to explicitly call something like UserId(...), just like with any other type. (You can actually do this technique in C by using a 1-element struct, although it's a lot more cumbersome than Scala since the type system is so much more limited).
As far as I know, the only available approaches are brands and just wrapping it in a one element object (but that's even worse). But the branding is a lot easier if you just use something like zod [1] for all your types.
Is there an automatic refactoring that changes type-inferred variable declarations and changes then into typed declarations?
I can see the value when writing code in the early stages of a project to both have flexibility and save keystrokes with type inference to help prototype and explore the problem space. Later on, I can see the benefit of switching to typed variable declarations for readability as time goes on and that part of the code base slips away from my working memory.
Yes, there is. That's one of the great things about Scala's typesystem. It makes those things easy to even write yourself, but there is already something builtin:
However, from my experience it's actually better to not annotate most things. I annotate methods are that used very often or are "borders" of my API or interface. Otherwise it is usually not needed because it rarely makes the code more clear and IDEs like IntelliJ actually show you the types inline automatically, so you can always see the return type easily.
Interestingly the source code base does not seem to have any tests. I guess it speaks of the strength of Scala's type system or the rigorousness of the developer.
I tried Scala a long time ago but I didn't like it, it felt every student of Martin Odersky's CS course at the EPFL got to design their own language feature, and they often
picked the same one but came up with different implementations.
As a few-hours-a-week Lichess user I knew they had to have been yoloing a bit in prod because sometimes they’ll restart servers several times within an hour interval, presumably as they deploy to prod, try to fix them, and redeploy.
But I’m honestly shocked it works as well as it does without tests. I mean I’m not a paying customer and I much prefer Lichess to the other chess website so I guess I have little say…
Does anybody know if Lichess would accept code contributions to add tests?
Scala certainly doesn't remove the needs for tests. But it reduces it a lot. I would say that it reduces the required number of tests to something between 1% to 10%, depending on the domain. The more "calculations"/"math" the domain contains, the more tests are necessary. But for the "plumbing" which usually makes up for the biggest amount of code, it does wonders.
Scala 3 is also quite different compared to the early times. Unfortunately the reputation is still a bit outdated.
Like many seasoned Scala developers I was skeptical at first but it's actually pretty enjoyable despite a few inconsistencies. Jumping between Scala 2 and 3 codebases isn't much of a problem. Granted, it makes the life of people working on tooling significantly more complicated but I'm not one of them.
The decision was directly motivated by Odersky's experience teaching Scala to students for 15+ years, it wasn't made for the sake of fueling drama in the community.
In Scala, eg 2.13 is a major version. Scala 3 is like a super-major version and still it's backwards compatible with 2.13: the new syntax is fully optional.
Which languages don't make any changes to syntax over major versions?
I'm not against syntax changes, but significant indentation seems like an insane choice to make. That's been a thorn in the side of Python for decades. Even if some people do like the look of it, it doesn't seem worth the tradeoffs.
I believe that significant indentation is only a problem for people whose text editor Tab binding doesn't handle it for them? At least, I've used Python for 15 years and can't think of a problem it's caused me -- what problems are you aware of? It's similar to the people who complain about lisp but are pairing up parens manually. I think the solution in both cases is "don't do that" -- there are perfectly good text editors that do these things for you.
I've always just added a git hook that runs the auto-formatter before they commit when I've coded on teams. Much more "strict" than even python is when it comes to coding standards.
> When everything compiled, I shipped it. And to everyone's surprise, apart from a few bugs I had created while rewriting thousands of lines of code... it worked. It just did. No explosions, no obscure bugs, no memory leak, no performance degradation. That was rather unexpected.
...followed by
> Then we saw the JVM CPU usage rise to alarming heights, with unusual patterns. And no obvious culprit in the thread dumps...
> it was just the JVM that needed some tuning.
Details: https://lichess.org/@/thibault/blog/lichess-on-scala3-help-n...