> As I understand it, the article talks about switching from python to rust in a non-trivial database service that is required to be fast and robust. I can't imagine python to do well in this regard, especially when C/C++ extensions are required to enhance the performance.
The article specifically talked about following the conventional Python advice to use C/C++ for the fast parts and Python for the "glue". This is precisely where Python people say Python shines.
> I just don't agree those points necessarily imply "avoiding Python for new projects".
You should avoid Python for new projects because you never know at the outset whether you will run into a performance bottleneck that can't be resolved by "just rewrite in C/numpy/etc" and there are other languages that are better than Python at both its strengths and weaknesses. The odds of painting yourself into a corner with Python are very, very high, and there are other languages that are better than Python at just about everything (the only area where Python still dominates is data science exploration libraries). Even if you don't paint yourself into a corner, you'll be dealing with lots of surprising papercuts: if you want reproducible builds, you'll have to wade through an ecosystem of package managers each with their own pitfalls including long build times (e.g., I saw pipenv take 30-45 minutes just to `pipenv lock` on a small-medium sized project); developing with docker basically doesn't work on mac (and windows?) because the VM uses all available CPU marshaling filesystem events over the guest/host boundary; your developers accidentally pulled in an API SDK that makes sync I/O calls (or else does some CPU-intensive task) which seizes up the event loop bringing the process down (and the failure cascades to remaining instances bringing the whole app down); and then there's the "regular" dynamic type system errors like forgetting to "await" things or missing/incorrect type documentation.
> Even in the case of the original article, was it, in hindsight, a bad decision to start with python? I'd argue not necessarily. I'd argue not necessarily. Python is still great for quickly hacking together something that works, not necessarily fast or maintainable, but if you're under time pressure or you're still trying to feel out the market fit for your product, it's not a great idea to waste brain cycles to deal with borrow checks for example.
My point isn't that Python is bad for literally every project--I actually think this project was probably a best-case scenario for Python (they didn't describe many of the issues I described in my previous paragraph) and they knew ahead of time where there performance issues would be so they were able to start with an architecture that was amenable to "write the slow parts in C/C++". My point is that there are languages that are better at fast iteration than Python (namely Go or TypeScript) while also excelling at Python's weaknesses (performance, tooling, distribution, type system, package management, etc etc etc). I think it's incredibly reckless to start a new commercial project in Python in almost all cases (with a possible exception for data science, but even then I would look hard at everything else first).
> Python's weaknesses only show when you need to scale, both on the performance and on the lines of code / number of collaborators metric, which is a problem you have only when the project succeeds.
This might be true for certain Django CRUD apps, but it's patently untrue in the general case. I've seen Python work fine for a prototype data science app, but fall over when real customer data was introduced. Similarly, I'm currently an SRE attached to a Python project which falls over regularly because of some event loop issue or another (often someone pulls in an SDK that makes sync calls under the hood, or else some CPU-intensive task blocks the event loop). I see stuff like this regularly, and it never happens in our Go apps.
> As other sibling comments point out, python is still great for small scripts and weekend projects.
Granted. Do what you want in your free time, but it's risky to use Python for new commercial apps.
> Perhaps you do mean 'avoiding Python for new projects that are expected to grow non-trivially', which is fine advice.
Yes, this is what I meant. I didn't mean "hobby projects".
> I'm just a bit tired of all those over-reaching blanket statements people make regarding software design
I don't usually make blanket statements, but Python really is so bad that this is a blanket statement I'm pretty comfortable with. I've scarcely ever heard of someone getting burned for picking Go over Python (there was some Python2 veteran who wrote an angry blog post once about Go back in the early days, but never since). Python just refuses to adequately progress because it wants to maintain compatibility--that's fine, but as professional engineers we should understand that the rest of the ecosystem is moving beyond where Python is willing/able to go, and it's no longer an appropriate tool for the overwhelming majority of new projects.
> I think it's incredibly reckless to start a new commercial project in Python in almost all cases...
> I don't usually make blanket statements, but Python really is so bad that this is a blanket statement I'm pretty comfortable with.
Speaking as someone who agrees with you that something static is very likely better than python for a large project, how do you square this with the existence of a lot of Python projects, even "very commercial", that seem to go on just fine?
It's a fine theory and something I instinctively support, but I don't see measurable evidence in support of it.
Just fine, you say? I had dozens of colleagues being in medium to large Python projects and every single one of them was terrified of every small change in the projects because inevitably something fails in an unexpected way.
These guys and girls are heroes. They are like the plumbers and electricians working at 3:00 AM so your water and electricity never stop while you're awake.
"Just fine" is likely only your external observation. I'd give them medals and help them retire at 35 for all the horrors they have endured. It's "just fine" because they tirelessly made sure of it.
Things were just fine despite Python. Not because of Python.
Plus, none of us has statistics that prove a theory beyond any doubt. Forget about that; this scarcely (if ever) existed in the programming world and likely will not exist anytime soon. We should apply our experience and the intuition that grows out of it.
Finally: a pile of anecdotal evidence can be treated as an objective evidence with, say, 50-60% probability. Insisting that the real world must comply with academically pure practices is not going to get us anywhere (and is misguided).
Survivorship bias. You don't hear about the failures. Moreover, many of those things (e.g., lots of Python at Google) was written at a time when there weren't better options, so you weren't struggling to make Python work while your competitor is writing Go or TypeScript.
Those projects were started when python was the least bad choice and grew out quickly enough that they could hire legions of developers to work around pythons problems.
> This might be true for certain Django CRUD apps, but it's patently untrue in the general case. I've seen Python work fine for a prototype data science app, but fall over when real customer data was introduced.
The cognitive dissonance embedded in this statement is rather striking, considering Django itself is written in Python and is known for its ruggedness and code quality. Django manages all of this without even taking advantage of somewhat newer features in Python like typing.
The article specifically talked about following the conventional Python advice to use C/C++ for the fast parts and Python for the "glue". This is precisely where Python people say Python shines.
> I just don't agree those points necessarily imply "avoiding Python for new projects".
You should avoid Python for new projects because you never know at the outset whether you will run into a performance bottleneck that can't be resolved by "just rewrite in C/numpy/etc" and there are other languages that are better than Python at both its strengths and weaknesses. The odds of painting yourself into a corner with Python are very, very high, and there are other languages that are better than Python at just about everything (the only area where Python still dominates is data science exploration libraries). Even if you don't paint yourself into a corner, you'll be dealing with lots of surprising papercuts: if you want reproducible builds, you'll have to wade through an ecosystem of package managers each with their own pitfalls including long build times (e.g., I saw pipenv take 30-45 minutes just to `pipenv lock` on a small-medium sized project); developing with docker basically doesn't work on mac (and windows?) because the VM uses all available CPU marshaling filesystem events over the guest/host boundary; your developers accidentally pulled in an API SDK that makes sync I/O calls (or else does some CPU-intensive task) which seizes up the event loop bringing the process down (and the failure cascades to remaining instances bringing the whole app down); and then there's the "regular" dynamic type system errors like forgetting to "await" things or missing/incorrect type documentation.
> Even in the case of the original article, was it, in hindsight, a bad decision to start with python? I'd argue not necessarily. I'd argue not necessarily. Python is still great for quickly hacking together something that works, not necessarily fast or maintainable, but if you're under time pressure or you're still trying to feel out the market fit for your product, it's not a great idea to waste brain cycles to deal with borrow checks for example.
My point isn't that Python is bad for literally every project--I actually think this project was probably a best-case scenario for Python (they didn't describe many of the issues I described in my previous paragraph) and they knew ahead of time where there performance issues would be so they were able to start with an architecture that was amenable to "write the slow parts in C/C++". My point is that there are languages that are better at fast iteration than Python (namely Go or TypeScript) while also excelling at Python's weaknesses (performance, tooling, distribution, type system, package management, etc etc etc). I think it's incredibly reckless to start a new commercial project in Python in almost all cases (with a possible exception for data science, but even then I would look hard at everything else first).
> Python's weaknesses only show when you need to scale, both on the performance and on the lines of code / number of collaborators metric, which is a problem you have only when the project succeeds.
This might be true for certain Django CRUD apps, but it's patently untrue in the general case. I've seen Python work fine for a prototype data science app, but fall over when real customer data was introduced. Similarly, I'm currently an SRE attached to a Python project which falls over regularly because of some event loop issue or another (often someone pulls in an SDK that makes sync calls under the hood, or else some CPU-intensive task blocks the event loop). I see stuff like this regularly, and it never happens in our Go apps.
> As other sibling comments point out, python is still great for small scripts and weekend projects.
Granted. Do what you want in your free time, but it's risky to use Python for new commercial apps.
> Perhaps you do mean 'avoiding Python for new projects that are expected to grow non-trivially', which is fine advice.
Yes, this is what I meant. I didn't mean "hobby projects".
> I'm just a bit tired of all those over-reaching blanket statements people make regarding software design
I don't usually make blanket statements, but Python really is so bad that this is a blanket statement I'm pretty comfortable with. I've scarcely ever heard of someone getting burned for picking Go over Python (there was some Python2 veteran who wrote an angry blog post once about Go back in the early days, but never since). Python just refuses to adequately progress because it wants to maintain compatibility--that's fine, but as professional engineers we should understand that the rest of the ecosystem is moving beyond where Python is willing/able to go, and it's no longer an appropriate tool for the overwhelming majority of new projects.