According to their normalized "global" results, something interesting i see:
1. Pascal, surprisingly, the most memory efficient of all. I should take a look at the implementation they used.
2. Rust a good alternative to C which leads in "energy efficiency" and speed.
3. Common Lisp most energy-efficient and fastest and smallest memory footprint of all the dynamic programming languages in the list -- like Python, Ruby, Lua, Perl, and even Racket (which fares pretty well)
4. PHP, JRuby, Ruby, Typescript, Perl, Python, being massively slow than the fastest languages, for example Ruby being 59 times slower than Rust or C.
I agree that these languages (in item 4) are "acceptably fast" for many applications, but we can't say they are "close to the speed" of the fastest languages, even more if we consider the fastest dynamic+interactive languages like Lisp (Python being 21 times slower than Lisp.)
5. I wonder which implementation of Lua they used. Lua can be pretty fast, one of the fastest dynamic languages out there.
"Pascal is a "hidden" gem in the area of languages"
I agree. Programmers won't look at it because they perceive it to be old and out-of-date. But the language hasn't stood still. It's a fast, low-memory language. FreePascal with the Lazarus IDE is one of the best cross-platform development toolkits for building native desktop apps.
Sadly a lot of programmers can never see beyond the verbose (but readable) syntax.
Every time I look at the syntax, it feels really verbose. IIRC some dialects require all caps or initial caps on keywords; at any rate that's how most examples are shown, and it's rather off-putting. Superficial, I know, but I'm probably not the only one.
I don't know about other implementations but FreePascal is completely case insensitive. Identifiers are actually treated the same if the only difference is case. I don't particularly like this, but I've never seen it actually be an issue
When I started programming in the early 90's, I remember Pascal being a serious contender, with C vs Pascal compilers being a pretty evenly matched contest.
Basically, C's semantics assume too little - everything is a primitive type or a pointer dressed up to look like something else. Pascal has a stronger notion of types and this allows for additional automatic optimization where they're relevant - memory layout just happens to be one such area.
I miss the ability to pair structures as "antagonistic" - aka if one exists the other is not, thus allowing for unions of various datatypes who guarantee that if they exist- the shared memory they reside in is theirs and theirs alone.
I have noticed a significant speedup using LuaJit, similar to PyPy compared to normal Python. I'm guessing if LuaJit & PyPy were used, the performance for Lua & Python would be similar to JS in these results.
> 5. I wonder which implementation of Lua they used. Lua can be pretty fast, one of the fastest dynamic languages out there.
Apparently 5.3, but I am very skeptical of these results. Also the way the data is organised is very weird. One should not compare a JIT compiled VM (JavaScript) with interpreters (Lua, Python, Ruby...). An honest approach would have used LuaJIT, PyPy etc. or separated the languages into "families" better. Not to mention the choice of algorithms they used. Some of the algorithms in the benchmark game have many flaws.
As it was already stated in this thread, some of the Python ones were multithreaded but the ones in Lua weren't. Why? In a different benchmark I've seen algorithms using C wrappers for Python, but not in Lua. While I understand wrapping C and multithreading reflect the industry use in Python, so they do in Lua. Benchmarks are useless if you are not consistent in the implementations of the things you are comparing with. It disgusts me, to be honest, when I see benchmarks comparing JIT compilers with vanilla Lua just to say "hey we're faster", as LuaJIT does not even exist. It looks like pure dishonesty. In the case of this paper in particular, as they do not represent any language, it strikes me as lack of research / understanding.
No mystery: someone has made the effort to contribute all those multi-process Python programs; no one has made the effort to contribute all those multi-process Lua programs.
Oh yes, that is true. But the person making the benchmarks should be aware of that and either look for better matching algorithms or implement them themselves.
If your intention is not to make a good comparison, then sure. I've seen other papers on language implementation and benchmarks, and researchers were way more careful than this, individually inspecting the algorithms they were using.
I happened to spot a problem on this one on the Lua part and also on some other parts concerning how they classify things, because it happens to be things I know more. I'm not going out of my way to look for problems in the rest of the paper because I'm not a reviewer and I have better things to do.
But if I was the one way conducting this academic research, then I would want the results not to be bogus. The difference here is this is not a blog post about some bullshit comparison someone is making. If that was the case, then alright, if public contributed algorithms failed to represent exactly what was being looked for, why not go for whatever is out there. I just expect a more rigorous procedure from academics.
> Are they likely to be expert in all those programming languages?
No, but again, I take paper results very seriously. Misleading results are bad for science, period. Here is an example of more rigorous research: https://arxiv.org/pdf/1602.00602.pdf
> If Lua wasn't included would you complain ?
Maybe I would, who knows. I did also mention in another comment it would be nice if Julia was there. It depends how relevant they are for the research being conducted. As a language used very often in microcontrollers, Lua is very relevant for energy efficiency research. I don't know anything about Forth, but I wouldn't just dismiss people saying it should be there with "then contribute forth algorithms to the benchmarks game yourself".
"disgusts me" & "pure dishonesty" don't seem like a serious response.
> … an example of more rigorous research…
Which again uses programs that were contributed to the benchmarks game.
> … wouldn't just dismiss people…
I wouldn't just dismiss them: I'd tell them that others have every right to present what seems important to them, and exclude what seems less important -- without being accused of dishonesty.
Because they're "different families" of language implementation. A Just In Time compiler is a compiler. It looks like an interpreter, but it compiles the code to machine code and runs it.
And obviously the performance of programs using those language implementations on the same computer under the same workload may be measured and compared.
Why should "different families" of language implementation not be compared?
Firstly, the paper separates languages into compiled languages and interpreted languages to begin with. So I was expecting it would follow this pattern correctly.
Secondly, why would you compare the performance of C and Ruby, for example? Ruby is bound to be slower for design reasons. Doing it to say "Look, C is fast!" doesn't mean anything. There are different reasons one would compare C and Rust, or C and Ruby. The way the results are displayed and what it tries to convey is very important.
If you're not making a baseline, or being very clear, etc. then it's just misleading to have a benchmark portraying Javascript as faster than Ruby, Python and Lua. Either you put those into their correct classifications clearly, or you compare it with LuaJIT, PyPy, Julia etc.
The paper, both shows tables that include all the results (ordered by Energy consumed) and shows separate charts for what the authors classify as "either a compiled, interpreted, or virtual-machine language".
What is "an interpreted language" ?
"Although we refer to Lua as an interpreted language, Lua always precompiles source code to an intermediate form before running it. … The presence of a compilation phase may sound out of place in an interpreted language like Lua. However, the distinguishing feature of interpreted languages is not that they are not compiled, but that any eventual compiler is part of the language runtime and that, therefore, it is possible (and easy) to execute code generated on the fly."
p57 "Programming in Lua" (2003)
> … why would you compare the performance of C and Ruby, for example? Ruby is bound to be slower…
Except when the C program is measured to be 50x slower than the Ruby program --
Lua is not compiled to machine code. It compiles to an intermediate bytecode, which is then interpreted. So you could say it is a VM. Also bear in mind, LuaJIT is not Lua. It's a different implementation.
I guess you misinterpreted my comment. I didn't imply there are no reasons to compare C and Ruby. I said the reason is important and the way you portray that, and the way you portray your results, changes things.
Do you think Roberto Ierusalimschy is confused about that ?
Again, here's what the creator of Lua says -- "… the distinguishing feature of interpreted languages is not that they are not compiled, but that any eventual compiler is part of the language runtime…"
EDIT:
Is that not what you mean by "interpreted language" ?
> Do you think Roberto Ierusalimschy is confused about that ?
No, of course not. I think you are. You are clearly misinterpreting his words. I will repeat, (PUC)Lua is not compiled to machine code, but to an intermediate bytecode form. Those are completely different things, and what I said does not come in conflict at all with the quote you took from PiL, so I don't know what you are confronting me about.
> Lua is not compiled to machine code, but to an intermediate bytecode form.
Correct.
Now look at what Roberto Ierusalimschy means by "interpreted language": "… the distinguishing feature of interpreted languages is not that they are not compiled, but that any eventual compiler is part of the language runtime and that, therefore, it is possible (and easy) to execute code generated on the fly."
SBCL is one lisp that I intend to learn more about. It's apparently the lisp of choice for XMaxima, which is a program that I used a lot in high school.
>5. I wonder which implementation of Lua they used. Lua can be pretty fast, one of the fastest dynamic languages out there.
I looked at the binary trees benchmark and while the Python version used multiprocessing to parallelize the code, the Lua version was not parallelized at all..
.. not that this is entirely unrealistic. Python comes with MP in the standard library while I am not aware of any "industrial strength" solution for Lua, whether MP or thread-based.
Idk if it is industrial strong, but there is lanes and probably few more listed at lua-users. Do you have any experience to share on these?
The choice of 5.3 is pretty strange, as Lua is known to split in two at 5.2, because LuaJIT was fixed to 5.1 variant with compat backports and is to my opinion an "industrial" Lua standard. 5.x are actually three different (but not absolutely superior to each other) languages with shared 5-like foundation and app-level code compatibility, so speaking about Lua in general is pointless in a sense.
Where are you getting that Pascal is the most memory efficient (edit: duh, under the "memory efficiency" table, mistakingly read this as "efficient")? It's firmly in the middle of the pack in every result, in their global result table it's slower and less energy efficient than Java (!). It's only advantage is that it's smaller than anything else.
Don't mean to assume you are wrong, but what are you referring to?
> 5. I wonder which implementation of Lua they used.
There is a table with the versions they used¹, but they don't mention the implementation – in can be inferred for some: e.g. for Pascal they used the Free Pascal Compiler.
To the point where a lot of popular libraries (like Love2D) use LuaJIT by default. IIRC, Lapis (Lua-based web framework) can be easily configured for it as well.
I'd be interested in seeing how it fared; Lua really is a nice language.
> 4. PHP, JRuby, Ruby, Typescript, Perl, Python, being massively slow than the fastest languages, for example Ruby being 59 times slower than Rust or C.
Take a look at some of the TypeScript code and compare it to the Javascript versions. At least some are way different, i.e. the TypeScript uses "modern" JS features and the JS version... looks about as much like C as it can.
Typescript is not its own language in terms of having a distinct runtime from Javascript so it has no performance difference and does not make sense to appear in this comparison.
I can see comparing different target versions of Javascript against one another (es2015 vs es2017 for instance), but it all depends on the VM chosen as well.
Exactly. The difference at least some of these tests is measuring between the two is idiomatic modern JS (promise, map, careful variable scoping) with fast & loose old-school c-like JavaScript (for loops, global vars galore). Typescript has little to do with it.
Typescript can use only of these, so it does make sense to have it on the list. Clearly it doesn't fall right next to JavaScript in the results. It is a benchmark of language implementations not runtimes. This is why JRuby, C# and F# are on the list, one uses the JVM and the latter two both use the CLR. Different languages.
1. Pascal, surprisingly, the most memory efficient of all. I should take a look at the implementation they used.
2. Rust a good alternative to C which leads in "energy efficiency" and speed.
3. Common Lisp most energy-efficient and fastest and smallest memory footprint of all the dynamic programming languages in the list -- like Python, Ruby, Lua, Perl, and even Racket (which fares pretty well)
4. PHP, JRuby, Ruby, Typescript, Perl, Python, being massively slow than the fastest languages, for example Ruby being 59 times slower than Rust or C.
I agree that these languages (in item 4) are "acceptably fast" for many applications, but we can't say they are "close to the speed" of the fastest languages, even more if we consider the fastest dynamic+interactive languages like Lisp (Python being 21 times slower than Lisp.)
5. I wonder which implementation of Lua they used. Lua can be pretty fast, one of the fastest dynamic languages out there.