Remember when it was discovered that arc welders could produce Carbon 60, and there was a huge run on arc welders? All of a sudden there was a whole new class of customer beyond the usual customer base of people who do welding: people who do materials science research.
I would hazard a guess to say that 99% of production Python is people doing the equivalent of welding, but there is also this 1% who want the GIL to be gone so they can repurpose Python as a tool for a whole new class of problem solving.
It will be an exciting future for them.
For us welders, we’ll carry on gluing stuff together blissfully unaware of the GIL. Async and non-blocking IO are great. I don’t think I’ve ever needed compute-concurrency, not in Python anyway.
You may think you don’t need it but I suspect if you program larger than trivial python applications there comes a point where you do but you don’t even think about how the GIL is restricting you. Think about load time for example of something that does a bunch of work to initialise. Without the GIL it would be relatively trivial to speed up things that are independent, while with GIL you are usually out of luck because you can have either shared memory (e.g. previously loaded state) or concurrency but not both at the same time. From experience, trying to serialise and use multiprocessing is often eating up all the potential gains.
I would love to parallelize a plugin script in Cura, the 3D print slicer. It does a bunch of embarrassingly parallel calculations, and could be made at least 16x faster for me. Because it's a plugin, though, it isn't pickle-able and multiprocessing doesn't work. I managed to make it work in a branch, but only on OSes that can fork processes. On windows, the plugin spawns multiple GUIs because importing the cura package apparently has the side effect of launching it...
If there wasn't the GIL, I could just create a thread pool and be done, and Cura could continue to be a delightful mess. :-)
20-odd years ago, my team solved some performance issues in our desktop app by splitting a couple of tasks into their own threads on a computer with a single CPU. Being able to write threaded code is really useful.
Some programming tasks are so trivial that processing speed is completely uninteresting.
Some are so compute heavy that massive amounts of time is put into tuning them (sort algorithms, fast-fourier transforms, etc).
Most fall somewhere on the spectrum between those extremes. If you can speed up your program by a factor of 20 by adding threads, Python can cover a bit more of that in-between spectrum.
I'm not sure I fully buy this argument. I think programming languages are chosen for projects based on a number of different factors, speed being one of them, but developer productivity being another (and there are certainly many more). Given that there isn't usually going to be one language that is the best choice for every single factor, it seems pretty natural that some teams might pick Python due to productivity being more important but still have some need for better performance, and other teams might not be able to compromise on performance and have to resort to something like Java or Go or C++ but still would be able to iterate faster with something higher level. It's definitely not a given that there are enough potential projects that would get enough benefit from removing the GIL to make it worth it, but it seems silly to claim that anyone would would get any possible benefit from performance would never choose Python for other reasons.
There are enough choices that offer Python's productivity alongside JIT/AOT options, better supported from the community than PyPy.
As for the C, C++, Fortran libraries with Python bindings, any language with FFI can call into them as well.
I would say, Python's adoption while lacking performance is what is now building pressure to avoid specific Python communities to leave Python and migrate into one of those ecosystems in search of a better mix of productivity/performance, without being forced to use two languages.
Ooh, a Clojure .NET implementation? This is news to me. From a quick look at the website, it looks really good... but how is it in practice? One of the major advantages of F Sharp's .NET integration is that it's developed by Microsoft, as of course they pretty much created the ecosystem from scratch.
But it is being continuously kept up to date by Cognitect, along with active development on a clojure-clr-next version. Maybe it has a bigger population of non public users.
Also: here's another implementation of Clojure on CLR, "Morgan And Grand Iron Clojure Compiler": https://github.com/nasser/magic (see also http://nas.sr/magic/) of which there's a great cmpiler implementation talk (at Clojure/North) somewhere on Youtube.
Performance is not binary. If you can be productive with Python and get closer to your performance goal, you get the best of both worlds. A lot of people who aren't primarily programmers don't have time to write everything in C++ (or keep up with development in that area). A more performant Python is a huge win for everybody.
Just like language choice isn't binary, there are plenty of options with Python like productivity, with much better performance, it isn't Python vs C++.
This is a tire fire of a thread, it's clear there's lots of confusion about the tradeoffs.
This isn't a case of "x" or "y". There is literally nothing valuable about the GIL, it's an ugly hack of a vestigial appendage. Perhaps the reason I'm familiar with it is because the lack of elegant MP threading in Python perturbed me for years, until I was introduced to Golang.
Python devs generally don't want to use Java, JavaScript, etc. And the Go ecosystem is good but not as rich as Pythons.
Anyhow, take care pjmlp. Until our paths cross again I wish you all the best!
a few years back I was curious so I took a self-balancing robot that I had written in C (it did PID in real time to set motors in response to the current angle).
I ported it to Python, with a totally naive simple single threaded loop. It worked perfectly. 25 updates a second, forever. No C code, except the interpreter doing its thing, and some GPIO code.
More than 5x can be achieved with the GIL included (and has been promised, by the new Microsoft Python-speed-up initiative and that Python dev who made a similar proposal).
> I don’t think I’ve ever needed compute-concurrency, not in Python anyway.
I have. Some very inexperienced “senior engineers” at my last gig thought it would be fine to build an analytics platform in Python because “Pandas will make it fast”. Unfortunately even modest datasets would cause timeouts and even seize the event loop, while a naive Go prototype would finish in a few hundred milliseconds.
Here's the text of a patent describing the process. [0] Googling "fullerene" will get you more useful hits than C-60, so, e.g. "generating fullerene from arc welding."
I believe the original paper on Buckminsterfullerine aka Bucky balls aka C60 described the approach. Basically do violent stuff with carbon and stick it in a mass spectrometer. The violent carbon bit is arc welding, because the arc welding tips are made of graphite.
Wikipedia informs me that the first papers used a laser (https://www.nature.com/articles/318162a0 - "graphite has been vaporized by laser irradiation, producing a remarkably stable cluster consisting of 60 carbon atoms") (1985) and the arc welding approach was a few year later. The paper "Solid C60: a new form of carbon" (1990) paper describes using benzene to dissolve the soot to extract the C60 (https://heyokatc.com/pdfs/MISC/Solid_C60-_a_new_form_of_carb...).
I would hazard a guess to say that 99% of production Python is people doing the equivalent of welding, but there is also this 1% who want the GIL to be gone so they can repurpose Python as a tool for a whole new class of problem solving.
It will be an exciting future for them.
For us welders, we’ll carry on gluing stuff together blissfully unaware of the GIL. Async and non-blocking IO are great. I don’t think I’ve ever needed compute-concurrency, not in Python anyway.