I really want to like Alpine, but we (Fly.io) have seen so many DNS issues with customer images that we’re now recommending Ubuntu or Debian slim. The extra ~50mb is a worthwhile trade off to avoid hard to debug musl-libc issues.
Yep. I had an (immutable) JVM ECS service that worked fine for a year, then started failing DNS in just one of several AWS AZs and for just one of various host names--one with something like 24 A records, a few but not an outrageous number. Occasionally forking a process to run 'dig' on the same name made it work for a little while.
AWS support's only advice was "don't use Alpine"; annoyingly, switching the containers to a Debian base cured it, even though this would appear to make absolutely no sense with respect to it failing in just one AZ.
Failing in one AZ probably means they changed something with DNS servers for that region. We had a similar issue recently when we rolled out a new DNS server that returned longer SOA records which broke python's mysql driver in only some regions during the rollout. Debugging nightmare fuel.
Yes, my theory was also some DNS variance in the AZ, but AWS stolidly refused to supply any information to that effect. Adding 'dig +trace' to the equation only deepened the mystery.
> Keep in mind that today due to the great popularity of Docker, Alpine Linux is one of the most deployed operating systems currently in use, because within every other operating system that uses docker, the docker image it uses is almost always Alpine Linux.
Isn't this a little overstated? I'm willing to accept it has a majority, especially once you get into custom images, but e.g. docker's library images are usually debian based, with a -slim or -alpine variant available, and of developer provided images on my network, debian + derived images have a majority, while Alpine's statement leads you to expect some kind of 90% share.
Docker images used to be entirely Debian-based but have slowly become almost entirely Alpine based. A lot of Debian users have become Alpine users, and I've been mulling over if I should join as well (since I've used Debian for 20+ years but are friends with Alpine devs and users).
I'm finding it hard to reconcile "Some of the software has secondary alpine images available" with "Almost all software uses alpine images"
For professional use, our company mandates all images used are built off a common base image, which is Ubuntu based (my previous employer was similar, but used a Red Hat based image).
We use alpine based images extensively where possible. Most popular language ecosystems and tools have an alpine option. Alpine as a base image is nice because it's nearly blank, but easy to customize unlike distroless. Recently one tool did drop Alpine, most based on it's reduced performance, and that was envoyproxy.
Same currently. Unless I'm just modifying a simple trusted image (which does happen often enough) I just start from Ubuntu during development and only attempt to move the image to Alpine if I'm really trying to improve resource usage down the line.
Alpine uses a different C standard library called musl compared to most well known distros that use glibc. You can run into issues if the application makes use of functions in glibc that musl has not implemented. Usually you run into them in applications that are written in C, such as in some Python modules that utilise C under the hood or in databases etc.
"I can't find $SOFTWARE in $PACKAGE_MANAGER now I have to compile it from source and my life sucks" has happened with me way more times with Alpine than Debian.
Because you cannot just jump in your container to debug. There are ephemeral containers for Kubernetes to add an image on top of a running container, for troubleshooting, but I haven't found something like that for Docker or Podman. You would need some kind of patchwork script where you untar an image and copy it inside the running container I guess.
Instead of using podman directly, and potentially messing with X11 permissions, you could use x11docker (despite the name, it podman support). It attempts to safely share X11 into a container.
Tried using x11docker in Alpine, didn't work for the Electron apps I tried with it. Tried for several days; finally got the app running in Docker without x11docker.
I got hazed by an upperclassmen with gentoo around that time and it's ruined me ever since. It's just not Linux if I can't edit a text file and rebuild the world.
Doesn't Alpine have a noticeable performance penalty due to being based on musl libc instead of glibc?
This seems to be born out in multiple people's micro benchmarks though how much of an issue it is for larger applications probably depends on the workload.
I went from an Ubuntu to an Alpine desktop and it was like bolting on a turbo. You know that rush when the turbo spools up and then it just plants you in your seat and you rocket away? Like this wasn't just "I think it's faster", I was like laughing and shouting it was so much faster. It doesn't seem quite as fast now; I feel like I must have added on enough extra software and crap now that it's close to where it was before. But the feeling when bringing up the desktop for the first time was visceral.
I've tried alpine's gcc before and it was about 50% slower to compile a bunch of packages. That said, if you compile llvm with MinSize/-Os on glibc distro's it runs 50% slower too, so it's unclear if slowdown is due to musl libc or rather that they optimize for size instead of performance.
Compiling is CPU-bound and doesn't really bottleneck on anything that libc is involved with other than the allocator, so unless musl's allocator is comically slower than glib's, chances are it's just -Os.
This is pretty much -Os.
Void Linux provides -O2 builds of GCC for both musl and glibc with a negligible performance difference. I have not benchmarked it lately, but it is clearly below 10%.
This release features the new "mallocng" malloc implementation, replacing musl's original dlmalloc-like allocator that suffered from fundamental design problems.
Anecdotal but I work on a performance oriented application and there is no such measured performance penalty from musl. For that particular use-case, like all benchmark caveats. Older musl, perhaps. Musl in other domains that bottleneck on something less performant, perhaps. Ymmv, measure twice cut once. :)
A nice feature of Alpine Linux is that the package repository is a relatively straightforwardly-structured git repository containing build scripts for each package:
It's worth reading the contributor guide[1] before signing up and providing any fixes and improvements - and from there, hopefully it's a relatively familiar workflow for many developers.
I use it mostly as a build image in order to produce completely static binaries which isn't possible with most other distros because the glibc maintainers at some point in the distant past decided to impose their opinion on the matter upon you by making it impossible to statically link against glibc.
Just burned out static toolchains that make me static binaries for all architectures gcc supports. Much like musl.cc but they suggest building your own and I do.
I use these toolchains on debian (/ anywhere a non-ancient linux kernel runs) to make static binaries, you can too!
When I looked into this, I decided against musl-cross-make because all the tools in there are way out of date. Due to its use of custom patches, you can't easily update it yourself; you must use their specific outdated versions. They're a full major version behind on gcc (it was two major versions behind at the time), and they're two minor versions behind on musl itself. Binutils is five minor versions behind. It was much easier to just fire up an Alpine build container with the latest versions of everything, and I'm actually capable of keeping it up to date. The Alpine folks seem to have more manpower at their disposal for keeping the packages up to date with upstream.
Not an apologist by any means and all that makes sense. I've found it easy and not-a-problem to bump those versions in the config file and made gcc 11.2.0 compilers with everything up-to-date this way quite recently.
There's also this fork of MCM that's a fair amount more active, as it backs musl.cc. I probably should have mentioned it
Regardless, your argument that a distro taking care of something for you is exactly what they're there for. Makes sense to me.
Mostly surfacing that there's multiple ways to handle that sort of thing. I find cross compiling simplest with these and version bumping wasn't a big deal at all compared to the complexities of other ways of cross compiling.
I don't think I had seen that fork before, thanks for the link. They appear to be much more up-to-date. There are still issues using Alpine as a development container. I ultimately resorted to a separate Ubuntu container for development, building against glibc. It was the only way to get VSCode's C++ and debugger extensions to work on aarch64. I don't love the workaround; using Ubuntu for everything would be nicer. I only use Alpine for test and release builds.
It seems though that glibc was inspired by musl to move away from many small shared libraries to a single one, namely the dynamic linker executable also being lib{c,m,...}.
I really like statically linking everything. I think the trade off space is a lot different now though. But I still agree they shouldn’t have done that
Absolutely beautiful little OS - I love that it manages to balance well between being light, secure and rather versatile for server environments. I run it as a WireGuard & Docker server on a low-end VPS, and it just works without any problems at all thus far.
Its wiki does a fine job at being a pretty good resource for documentation, too, and both ArchWiki & Gentoo Wiki work well as fallbacks.
Using musl instead of glibc leads to enough surprises that I've migrated my alpine containers over to Debian-slim based ones. If you have a bunch of containers built on the same base, the storage of Alpine vs Debian becomes less substantial, and you avoid a lot of weird bugs and a surprisingly large performance hit (at least for python)
Started using Alpine as a curiosity for embedded/high security development a while ago, probably 2011-2012. I've of course used it a fair bit in $day_job containerization of services. Nowadays personally I find myself using it for the bare metal OS on servers running minimal services: things like VM hosts, purpose-built fileservers, etc.
It's a fine distro, you can typically go a good bit of time between updates without worrying about fragging your install, which is especially important to me on personal projects, which are often more like pets than livestock :P
Question, we're a long time Windows shop and just recently have been spinning up some linux containers (redis, rabbitmq). The decision fell to me and I went with Ubuntu. (I'm confident in our networking team that everything is firewalled well and not externally accessible so these are local LAN only.)
Should I instead be using Alpine? If someone is isn't distro aficionado and is utilizing "mainstream" docker images, what's the recommendation?
> just recently have been spinning up some linux containers (redis, rabbitmq)
Not sure if you're trying to make your own containers or not, but stick to the stock Docker Hub containers as much as possible. If there's an "official" Redis container, use that, regardless of the base image. Since it's supported, other people will be fixing bugs and upgrading things so you don't have to.
> If someone is isn't distro aficionado and is utilizing "mainstream" docker images, what's the recommendation?
For non-container stuff, CentOS is the "enterprisey" distro that's safest in case you one day have to support an app that only supports RHEL (and it's the base for Amazon Linux, in case you use AWS).
For container stuff, it's a toss-up between Ubuntu and Debian; I'd use Ubuntu for developer-oriented stuff, and Debian for anything else. You can of course use Alpine for containers, but there are some maintenance/development costs that come as well, so I only recommend Alpine if you really need to save space.
Ubuntu will likely be more compatible and easier to work with. The version you get in docker is very minimal as well, although maybe not quite as tiny as alpine.
The maturity of your organisation’s use of containers is quite important: Alpine is a fantastic platform to build very lightweight images, but there’s organisations where people want containers that have everything. I’d say, as your usage of and experience with containers grows, aim to build more lightweight images and eventually you’ll get to the point where Alpine is the logical next step, rather than being the right choice because people on HN love it.
Distros seem to be having a hard time updating glibc, and there are known vulnerabilities with the pinned version of most of them. Musl/Alpine doesn't have that problem, because it doesn't use glibc.
Ubuntu comes with snaps. If you don't use snaps, they are consuming a not insignificant portion of boot time. Use Debian if you need something glib-based. If you must use Ubuntu, make sure you are using slim: you don't need manpages etc. in a production container.
Overall boot time is affected by image size, because your cluster has to download that image from $SERVER. Alpine is smaller than most, scratch is even better (if you can pull it off). Building containers with Nix can result in stupidly small images.
Its not clear-cut, but hopefully that helps you make a choice.
I think Ubuntu server is a fine choice for your needs. I tend to use it on small VPSes as it's easy to setup and get running, relatively up to date and in wide usage.
I recently deployed a small fleet of Dell thin clients around my house to run zwavejs2mqtt[1] via docker. I decided to use Alpine because the machines have 2GB of ram and 8GB of eMMC storage and I figured Alpine would minimize resource usage. They're working great so far!
I'm typing this from an Alpine laptop. It still has a ways to go in terms of being as user-friendly as Slackware, but it's getting there. Some trouble at the moment is figuring out all the packages you need to install (and how to configure them) to get an actual working desktop as you'd expect it. There are also bugs in things like Bluetooth support and some other things. But overall it's quite usable, and very fast. You end up using Flatpak for any proprietary apps once dealing with glibc compatibility problems becomes too annoying. Some apps on Flatpak don't really work, like Slack (maybe it's possible to get it to work, but after a few days I gave up). Electron apps in general are a huge pain. I've gotten 1Password to work in Docker after over a week of futzing around. You really learn how much of a nightmare the Linux desktop today is. You also learn how much of the desktop (and non-desktop) apps actually require systemd to run at all, and how people have had to come up with fake systemd components. You definitely learn a lot about Linux!
>due to the great popularity of Docker, Alpine Linux is one of the most deployed operating systems currently in use
Wow, that's impressive! However, when I check on Distrowatch it's only in the 30s. Is there a better statistic? Something more representative of actual amounts of installs on anything at all?
It’s used to build pretty much every Docker image around us. Works extremely well as WSL. Super lightweight, performant, and the number of packages is growing strong.
Pair it with xcfe (or gnome or kde) and you have a super desktop, no more worries about RAM or background processes eating up CPU (looking at you Windows).
Yes, there’s a Blender package! and many other good stuff.
Alpine made Linux fresh again, and yes I have 32GB of RAM and a fast CPU and SSD, using Alpine will make sure it stays fast
It needs much less than that to run, of course, it’s brilliantly focused.
I'm working on a personal project where musl is the foundation of my project. So I'm interested in learning about all the downsides, corner cases, and footguns of using musl. Can you please share some pointers (search phrases, mailing list/forum posts, etc.) for me to read more.
I saw a thread the other day on docker command pull alpine:latest is not good. Im pretty sure like every enterprise docker container is running this command. Anyone see anyone change this or bring this up at work?
They’re probably referring to the “latest” tag. The “latest” tag is a terrible idea to be anywhere near anything that matters but I’d be very surprised if any enterprise is depending on a latest tag.
Digest pinning is what should be used in any system that matters.
While the difference between some distros is almost academic, other distros have very strong principles that lead them to having very different forms and functions. Alpine falls into that category - if one were to not 'worry' about distros as you seem to be suggesting and downloaded Alpine without reading up on it, you would likely be put out by what you find.
Alpine is a distro with a purpose, and that is worth 'worrying' about.
You say that but there's other reasons to use linux (on the desktop!), and serverless itself is an old concept in which I never used to care which flavour of unix/linux my host was running my php or cgi scripts on.
eg https://www.linkedin.com/pulse/musl-libc-alpines-greatest-we...