I'm brushing up with Python for a new job, and boy what a ride. Not because of the language itself but the tooling around packages. I'm coming from Go and TS/JS and while these two ecosystems have their own pros and cons, at least they are more or less straightforward to get onboarded (there are 1 or 2 tools you need to know about). In Python there are dozens of tools/concepts related to packaging: pip, easy_install, setuptools, setup.py, pypy, poetry, uv, venv, virtualenv, pipenv, wheels, ...
There's even an entire website dedicated to this topic: https://packaging.python.org
Don't understand how a private company like Astral is leading here. Why is that hard for the Python community to come up with a single tool to rule them all? (I know https://xkcd.com/927/). Like, you could even copy what Go or Node are doing, and make it Python-aware; no shame on that. Instead we have these who-knows-how-long-they-will-last tools every now and then.
They should remove the "There should be one-- and preferably only one --obvious way to do it." from the Python Zen.
> In Python there are dozens of tools/concepts related to packaging: pip, easy_install, setuptools, setup.py, pypy, poetry, uv, venv, virtualenv, pipenv, wheels,
Some of those are package tools, some are dependency managers, some are runtime environments, some are package formats...
Some are obsolete at this point, and others by necessity cover different portions of programming language technologies.
I guess what I'm saying is, for the average software engineer, there's not too many more choices in Python for programming facilities than in Javascript.
It's not an easy task, and when there's already lots of established practices, habits, and opinions, it becomes even more difficult to get around the various pain points. There's been many attempts: pip (the standard) is slow, lacks dependency resolution, and struggles with reproducible builds. Conda is heavy, slow to solve environments, and mixes Python with non-Python dependencies, which makes understanding some setups very complicated. Poetry improves dependency management but is sluggish and adds unnecessary complexity for simple scripts/projects. Pipenv makes things simpler, but also has the same issue of slow resolution and inconsistent lock files. Those are the ones I've used over the years at least.
uv addressed these flaws with speed, solid dependency resolution, and a simple interface that builds on what people are already used to. It unifies virtual environment and package management, supports reproducible builds, and integrates easily with modern workflows.
I don't know what guides you're reading but I haven't touched easy_install in at least a decade. It's successor, pip, had effectively replaced all use cases for it by around 2010.
I actually think the main problem is that they aren't even willing to admit that there is a problem. It's the classic "our program is great; if users have issues it's because they are using it wrong / didn't read the manual / are stupid".
Go and look up why you can't run scripts with `python3 foo.py` on Windows. It's like a one-line fix and they've come up with all sorts of naysaying reasons rather than just adding python3.exe (which Microsoft did years ago in their Python packages).
You don't need to know most of those things. Until last year I used setup.py and pip exclusively for twenty years, with a venv for each job at work. Wheels are simply prebuilt .zips. That's about an hour of learning more or less.
Now we have pyproject.toml and uv to learn. This is another hour or so of learning, but well worth it.
Astral is stepping up because no one else did. Guido never cared about packaging and that's why it has been the wild west until now.
I don't know; I was looking at TS tutorials the other day and there seemed to be at least half a dozen "bundlers" with different tutorails suggesting different ones to use. It took me a while to figure out I could just directly invoke "tsc" to generate javascript from typescript.
Go only "got this right" after a decade of kicking and screaming about how it wasn't necessary. Dealing with the vendor and go.mod transitions was incredibly painful. There wasn't even a mechanism for hiding packages from everyone in the world from importing until Go 1.4!
I'm still not convinced that the built-in vendor support added in Go 1.5 wasn't intentionally made incompatible with the community-developed solutions out of some weird kind of spite. Why didn't they just use "./vendor/src" like all of the existing tools were using? (Remember that Go was written by plan9 folks, so making a "src" symlink didn't work. In fact, the Go compiler dislikes symlinks in most places -- pre-modules the unholy things you had to do with GOPATH were awful.)
Unless the thing you're building is a python library you want other people to install, you don't need to understand all that much of it.
You just need to understand how to setup a venv, and for each thing you want to depend on how to install it in the venv. Put those bits in a shell script and you can clone the project, run the script and then have a working env
It will break sometimes because python and a lot of python packages have zero respect for backward compatibility, but that's python for you.
It's ugly and horrible, but you don't have to relearn the latest python packaging "standard" every other year
I work with Python, Node and Go and I don't think any of them have great package systems. Go has an amazing module isolation system and boy do I wish hiding functions within a module/package was as easy in Python as it is in Go. What saves Go is the standard library which makes it possible to write almost everything without needing external dependencies. You've worked with JavaScript and I really don't see how Python is different. I'd argue that Deno and JSR is the only "sane" approach to packages and security, but it's hardly leading and NPM is owned by Microsoft so it's not like you have a great "open source" platform there either. On top of that you have the "fun" parts of ESM vs CommonJS.
Anyway, if you're familiar with Node then I think you can view pip and venv as the npm of Python. Things like Poetry are Yarn, made as replacements because pip sort of sucks. UV on the other hand is a drop-in replacement for pip and venv (and other things) similar to how pnpm is basically npm. I can't answer your question on why there isn't a "good" tool to rule them all, but the single tool has been pip since 2014, and since UV is a drop-in, it's very easy to use UV in development and pip in production.
I think it's reasonable to worry about what happens when Astral needs to make money for their investors, but that's the beauty of UV compared to a lot of other Python tools. It's extremely easy to replace because it's essentially just smarter pip. I do hope Astral succeeds with their pyx, private registries and so on by the way.
> Why is that hard for the Python community to come up with a single tool to rule them all?
Whatever I would say at this point about PyPA would be so uncharitable that dang would descend on me with the holy hammer of banishment, but you can get my drift. I just don't trust them to come out with good tooling. The plethora they have produced so far is quite telling.
That said, pip covers 99% of my needs when I need to do anything with Python. There are ecosystems that have it way worse, so I count my blessings. But apparently, since Poetry and uv exist, my 99% are not many other people's 99%.
If I wanted to package my Python stuff, though, I'm getting confused. Is it now setup.py or pyproject.toml? Or maybe both? What if I need to support an older Python version as seen in some old-but-still-supported Linux distributions?
> They should remove the "There should be one-- and preferably only one --obvious way to do it." from the Python Zen.
Granted, tooling is different from the language itself. Although PyPA could benefit from a decade having a BDFL.
> If I wanted to package my Python stuff, though, I'm getting confused. Is it now setup.py or pyproject.toml? Or maybe both? What if I need to support an older Python version as seen in some old-but-still-supported Linux distributions?
Your Python version is irrelevant, as long as your tools and code both run under that version. The current ecosystem standard is to move in lock-step with the Python versions that the core Python team supports. If you want to offer extended support, you should expect to require more know-how, regardless. (I'm happy to receive emails about this kind of thing; I use this username, on the Proton email service.)
Nowadays, you should really always use at least pyproject.toml.
If your distribution will include code in non-Python languages and you choose to use Setuptools to build your package, you will also need a setup.py. But your use of setup.py will be limited to just the part that explains how to compile your non-Python code; don't use it to describe project metadata, or to orchestrate testing, or to implement your own project management commands, or any of the other advanced stuff people used to do when Setuptools was the only game in town.
In general, create pyproject.toml first, and then figure out if you need anything else in addition. Keeping your metadata in pyproject.toml is the sane, modern way, and if we could just get everyone on board, tools like pip could be considerably simpler. Please read https://blog.ganssle.io/articles/2021/10/setup-py-deprecated... for details about modern use of Setuptools.
Regardless of your project, I strongly recommend considering alternatives to Setuptools. It was never designed for its current role and has been stuck maintaining tons of legacy cruft. If your project is pure Python, Flit is my current recommendation as long as you can live with its opinionated choices (in particular, you must have a single top-level package name in your distribution). For projects that need to access a C compiler for a little bit, consider Hatch. If you're making the next Numpy, keep in mind that they switched over to Meson. (I also have thrown my hat in this ring, although I really need to get back to that project...)
If you use any of those alternatives, you may have some tool-specific configuration that you do in pyproject.toml, but you may also have to include arbitrary code analogous to setup.py to orchestrate the build process. There's only so far you can get with a config file; real-world project builds get ferociously complex.
> Why is that hard for the Python community to come up with a single tool to rule them all?
Design by committee is hard. Especially building consensus around reconsidering what at this point should be considered bad design. Python has a few decades of that. It took decades for the to even consider starting the process of removing the GIL, which is a great example of "it's clearly broken but lets not fix it". Packaging is the same.
These guys are laser focused on just fixing things the right way and that's what it takes,
There seems to be a lot of movement around some of these topics. The Gil is being removed, python's performance issues are meling away with recent releases, and things like uv are happening. I like it.
> It took decades for the to even consider starting the process of removing the GIL,
This is not true. Discussions about removing the GIL have been ongoing for decades. There were some abortive attempts over the years but it is regarded as quite the technical challenge, and has implications for compatibility with code written in C.
> which is a great example of "it's clearly broken but lets not fix it".
I appreciate everything they’ve done but the group which maintains Pip and the package index is categorically incapable of shipping anything at a good velocity.
It’s entirely volunteer based so I don’t blame them, but the reality is that it’s holding back the ecosystem.
I suspect it’s also a misalignment of interests. No one there really invests in improving UX.
> the group which maintains Pip and the package index is categorically incapable of shipping anything at a good velocity.
> It’s entirely volunteer based so I don’t blame them
It's not just that they're volunteers; it's the legacy codebase they're stuck with, and the use cases that people will expect them to continue supporting.
> I suspect it’s also a misalignment of interests. No one there really invests in improving UX.
"Invest" is the operative word here. When I read discussions in the community around tools like pip, a common theme is that the developers don't consider themselves competent to redesign the UX, and there is no money from anywhere to hire someone who would be. The PSF operates on an annual budget on the order of $4 million, and a big chunk of that is taken up by PyCon, supporting programs like PyLadies, generic marketing efforts, etc. Meanwhile, total bandwidth use at PyPI has crossed into the exabyte range (it was ~600 petabytes in 2023 and growing rapidly). They would be completely screwed without Fastly's incredible in-kind donation.
Indeed, they broke a few features in the last few years and made the excuse "we can't support them, we're volunteers." Well, how about stop breaking things that worked for a decade? That would take less effort.
They had time to force "--break-system-packages" on us though, something no one asked for.
> how about stop breaking things that worked for a decade?
They aren't doing this.
> They had time to force "--break-system-packages" on us though, something no one asked for.
The maintainers of several Linux distros asked for it very explicitly, and cooperated to design the feature. The rationale is extensively documented in the proposal (https://peps.python.org/pep-0668/). This is especially important for distros where the system package manager is itself implemented in Python, since corrupting the system Python environment could produce a state that is effectively unrecoverable (at least without detailed Python-specific know-how).
Was being a facetious, sure someone asked for it, but it was pretty dumb. This has never "corrupted" anything, is rare (not happened to me in last 15 years), and simply fixed when knowledgeable.
Not everyone can simply fix it, so a better solution would be to isolate the system python, allow more than one installed, etc.
Distros already do this to some extent.
> there are dozens of tools/concepts related to packaging ... Why is that hard for the Python community to come up with a single tool to rule them all?
They have: uv
> Don't understand how a private company like Astral is leading here
> Why is that hard for the Python community to come up with a single tool to rule them all? (I know https://xkcd.com/927/).
because they're obsessed with fixing non-issues (switching out pgp signing for something you can only get from microsoft, sorry, "trusted providers", arguing about mission statements, etc.)
whilst ignoring the multiple elephants in the room (namespacing, crap slow packaging tool that has to download everything because the index sucks, mix of 4 badly documented tools to build anything, index that operates on filenames, etc.)
> Don't understand how a private company like Astral is leading here. Why is that hard for the Python community to come up with a single tool to rule them all? (I know https://xkcd.com/927/). Like, you could even copy what Go or Node are doing, and make it Python-aware; no shame on that. Instead we have these who-knows-how-long-they-will-last tools every now and then.
Python packaging is (largely) solving problems that Go and Node packaging are not even trying to address.
Specifically simultaneous distribution of precompiled binaries for many different OS and hardware configurations and built-on-demand source distribution of non-Python software to be used as dependencies with as little (ideally none) host setup by the user all installable under a single name/version everywhere.
imagine a world without: failed to build native gem extension
Not the person you're replying to, so I don't know if this is what he had in mind, but with Python packages you can distribute more than just Python. Some packages contain C/C++/Fortran/Rust/others? source code that pip will try to automatically build upon install. Of course you can't expect everyone to have a dev environment set up, so packages can also contain pre-compiled binary for any combination of windows/mac/linux + amd/arm + glibc/musl + CPython/pypy (did I miss any?).
I don't know much about go, and I've only scratched the surface with node, but as far as node goes I think it just distributes JS? So that would be one answer to what Python packaging is trying to solve that node isn't trying to address.
> any combination of windows/mac/linux + amd/arm + glibc/musl + CPython/pypy (did I miss any?).
From a standards perspective, it is a combination of a Python version/implementation, a "platform" and an "ABI". (After all, the glibc/musl distinction doesn't make sense on Windows.)
Aside from CPython/pypy, the system recognizes IronPython (a C# implementation) and Jython (a Java implementation) under the version "tag"; of course these implementations may have their own independent versioning with only a rough correspondence to CPython releases.
The ABI tag largely corresponds to the implementation and version tag, but for example for CPython builds it also indicates whether Python was built in debug or release mode, and from 3.13 onward whether the GIL is enabled.
The platform tag covers Mac, Windows, several generic glibc Linux standards (called "manylinux" and designed to smooth over minor differences between distros), and now also some generic musl Linux standards (called "musllinux"). Basic CPU information (arm vs intel, 32- vs 64-bit etc.) is also jammed in here.
> Some packages contain C/C++/Fortran/Rust/others? source code that pip will try to automatically build upon install.
And in the TS/JS world we have React.Native that has a flexible pluggable model that allows creating XCode projects with autodiscovered dependencies in C, C++, Swift and other languages.
It's also flexible enough to allow third-party products like Sentry to integrate into the build process to upload debug symbols to the Sentry servers on release builds.
So no, Python is really not unique in its requirements.
Come on, React Native is a thing but as a % of JS engineers, the number that will regularly stray into compiled packages is much smaller than with Python. Basically every time you pick up a Maths or ML project which is a huge part of Python's ecosystem, you'll be using C or Fortran code underneath. That library ecosystem in that are is just non-existent in JS, and where it does people are typically using compile-to-WASM anyway.
Pypi doesn't have any issue distributing binaries either these days? The issue is (a) diversity of CPU/GPU microarchitectures inc. CUDA (b) diversity of OS-es.
I've never had to compile a native Node library on a POWER9 GPU cluster, but I have had to compile Python ones.
And the JS world has ESM modules, CommonJS, various polyfills for compatibility, and other quirks that need to be accommodated by the package manager. With native code compilation affected by these settings.
> I don't know much about go, and I've only scratched the surface with node, but as far as node goes I think it just distributes JS?
As a sibling comment posted, Node packages have no issue distributing non-JS assets. Since the very beginning, the "package.json" file had support for the "arch" and "postinstall" fields to distribute precompiled binaries or compile on install, for example using node-gyp. If you split your precompiled binaries and use optional dependencies then only the right package for your machine is downloaded.
In all package managers, distributing native code may be uncommon but it's always supported well enough (Node/npm/yarn, JVM/Gradle, PHP/Composer, Ruby/gem, Rust/Cargo, etc). What's unique about Python is how messy all the packaging handling is compared to other languages.
It's almost all largely because of compiled packages.
What happens in Go/TS when you want to use a C library underneath that's dynamically linked? Oh wait, there's no mechanism in it's package manager to support this, so it's assumed it's installed already, or in Go, C code is bundled instead. This is the problem all the Python packaging stuff above has tried to solve.
> Why is that hard for the Python community to come up with a single tool to rule them all?
0. A lot of those "tools and concepts" are actually completely irrelevant or redundant. easy_install has for all practical purposes been dead for many years. virtualenv was the original third party project that formed the basis for the standard library venv, which has been separately maintained for people who want particular additional features; it doesn't count as a separate concept. The setup.py file is a configuration file for Setuptools that also happens to be Python code. You only need to understand it if you use Setuptools, and the entire point is that you can use other things now (specifically because configuring metadata with Python code is a terrible idea that we tolerated for far too long). Wheels are just the distribution format and you don't need to know anything about how they're structured as an end user or as a developer of ordinary Python code — only as someone who makes packaging tools. And "pypy" is an alternate implementation of Python — maybe you meant PyPI? But that's just the place that hosts your packages; no relevant "concept" there.
Imagine if I wanted to make the same argument about JavaScript and I said that it's complicated because you have to understand ".tar.gz (I think, from previous discussion here? I can't even find documentation for how the package is actually stored as a package on disk), Node.js, NPM, TypeScript, www.npmjs.com, package.json..." That's basically what you're doing here.
But even besides that, you don't have to know about all the competing alternatives. If you know how to use pip, and your only goal is to install packages, you can completely ignore all the other tools that install packages (including poetry and uv). You only have any reason to care about pipenv if you want to use pip and care about the specific things that pipenv does and haven't chosen a different way to address the problem. Many pip users won't have to care about it.
1. A lot of people actively do not want it that way. The Unix philosophy actually does have some upsides, and there are tons of Python users out there who have zero interest in participating in an "ecosystem" where they share their code publicly even on GitHub, never mind PyPI — so no matter what you say should be the files that give project metadata or what they should contain or how they should be formatted, you aren't going to get any buy-in. But beyond that, different people have different needs and a tool that tries to make everyone happy is going to require tons of irrelevant cruft for almost everyone.
2. Reverse compatibility. The Python world — both the packaging system and the language itself — has been trying to get people to do things in better, saner ways for many years now; but people will scream bloody murder if their ancient stuff breaks in any way, even when they are advised years in advance of future plans to drop support. Keep in mind here that Python is more than twice as old as Go.
3. Things are simple for Go/JS/TS users because they normally only have to worry about that one programming language. Python packages (especially the best-known, "serious" ones used for heavyweight tasks) very commonly must interface with code written in many other programming languages (C and C++ are very common, but you can also find Rust, Fortran and many more; and Numpy must work with both C and Fortran), and there are many different ways to interface (and nothing that Python could possibly do at a language level to prevent that): by using an explicit FFI, by dlopen() etc. hooks, by shelling out to a subprocess, and more. And users expect that they can just install the Python package and have all of that stuff just work. Often that means that compiled-language code has to be rebuilt locally; and the install tools are expected to be able to download and set up a build system, build code in an isolated environment, etc. etc. All of this is way beyond the expectations placed on something like NPM.
4. The competition is deliberate. Standards — the clearest example being the PEPs 517/518/621 that define the pyproject.toml schema — were created specifically to enable both competition and interoperation. Uv is gaining market share because a lot of people like its paradigms. Imagine if, when people in the Python community first started thinking about the problems and limitations of tools from the early days, decided to try to come up with all the paradigms themselves. Imagine if they got them wrong, and then set it in stone for everyone else. When you imagine this, keep in mind that projects like pip and setuptools date to the 2000s. People were simply not thinking about open-source ecosystems in the same way in that era.
> They should remove the "There should be one-- and preferably only one --obvious way to do it." from the Python Zen.
First, define "it". The task is orders of magnitude greater than you might naively imagine. I know, because I've been an active participant in the surrounding discussion for a couple of years, aside from working on developing my own tooling.
Don't understand how a private company like Astral is leading here. Why is that hard for the Python community to come up with a single tool to rule them all? (I know https://xkcd.com/927/). Like, you could even copy what Go or Node are doing, and make it Python-aware; no shame on that. Instead we have these who-knows-how-long-they-will-last tools every now and then.
They should remove the "There should be one-- and preferably only one --obvious way to do it." from the Python Zen.