Shout out to https://github.com/samuelkarp/runj which aims to provide OCI compatible runtime for FreeBSD jails. We are working on a Jails based sandbox implementation for running OCaml builds on FreeBSD, https://github.com/ocurrent/obuilder. If runj gets more support then it would be a good alternative to the various iocage/ezjail/etc solutions or the raw Jails implementation we have right now.
Not necessarily, Kubernetes also needs a ton of other things which are Linux kernel specific, like cgroups.
If you're just looking for an orchestrator capable of running FreeBSD jails, take a look at Nomad (Disclaimer: I work at HashiCorp, opinions my own, etc.)
Thank you for the suggestion, I'm interested in checking it out.
How does Nomad handle scaling? I'm thinking of a new project, I'd like to have the equivalent of HPA that easily scales to zero on custom metrics and a cluster autoscaler that can easily add heterogenous nodes as necessary. Even better if it can work with MacOS VMs.
Nomad supports all sorts of architectures and OSes, including M-series CPUs and MacOS. It can also scale to a lot, 2 million containers being tested. [0]
In terms of autoscaling there's the Nomad Autoscaler[1][2] that is super flexible, can get metrics from a bunch of sources, do custom queries, and perform actions (everything is via plugins, so even if your current stack isn't already supported, it's easy to add) like add extra nodes or add more instances of jobs.
The issue I have with Jails, over something like docker, is that the kernel ABI changes between major releases so jails need to migrate with base systems upgrades. It also means a container registry would need to store container versions per release.
I like the isolation for my own bespoke home network, but does it work well in production set up’s?
There are compat libraries that are available, so if you are running 13.x (the most current major release) but want to run binaries originally compiled under 12.x, you install:
And for anyone that isn't intimately familiar with FreeBSD's release history or anything, I feel like it's worth pointing out that the 4.x major release dates from over 23 years ago. You can run binaries from before people were inhabiting the ISS and/or right around the time AOL was big enough to purchase Time Warner.
Kernels are generally backwards compatible and major versions are explicitly supported for five (5) years. Over in Linux land you'll see builds for the different libcs and it seems to work just fine.
The difference is that the container libc version and kernel version don't have to correspond to each other in any way on Linux (unless your kernel is like 20 years older, in which case it wouldn't support Docker anyway).
The end result is that you're matching the executable to the libc variant instead of to the kernel. In practice it's not much of a difference, if at all. If you need to use older binaries, install the compatibility shims (or use a base jail based on the older version of FreeBSD). In terms of a support matrix there are two major versions of FreeBSD that are given active support, which is less than over in Linux land.
All in all of all the concerns I'd have about using jails in a production environment, dealing with different major versions is not on that list.
In FreeBSD the userland and kernel move in unison, and with the backwards compatibility there's nothing stopping you from running a jail based on a previous version of FreeBSD. The compatX packages are only there to provide the older libraries if you want to run an older application on a newer jail/install.
As was pointed out, you can run binaries from a FreeBSD version (4) that's probably older than most posters here. Still not seeing what the big fuss is about.
the hangup is because half the point is to be able to take a random userland and run it on your current system without changing it. As soon as your userland bitrots because the kernel ABI changed, you've lost a lot of the benefit.
And to be clear, you can not take a complete userland from 4.x. You have to reinstall your application on a newer userland with compat shims when you change the kernel.
Ironically enough, the Linux syscall compat layer in FreeBSD may give you more stability here.
You have to reinstall your application on a newer userland with compat
shims when you change the kernel.
No, you don't. You can absolutely take a jail based on a previous version of FreeBSD and run it on a newer version. I've not tried going as far back as 4.x, but I've done jail images of 9 and 10 on 11 and 12 hosts as part of a CI system.
The shims you're talking about are not shims at all, just packages of the older libraries that a program built for an older version of FreeBSD would expect. You can go ahead and look at the package manifests if you don't believe me. They're literally just a handful of older shared libraries.
Honestly, I can't remember a time when the syscall table changed in a backwards incompatible manner. The compat packages are only needed if you're trying to run an older program with a newer userland. If you're running an older jail in a newer freebsd host, you can run your program as-is because those libraries have already been installed.
> If the kernel version differs from the one that the system utilities have been built with, for example, a kernel built from -CURRENT sources is installed on a -RELEASE system, many system status commands like ps(1) and vmstat(8) will not work. To fix this, recompile and install a world built with the same version of the source tree as the kernel. It is never a good idea to use a different version of the kernel than the rest of the operating system.
Just stop already, you don't know what you're talking about. ps works just fine. The jail itself picks up the version of the host because it's calling the newer kernel, but otherwise works as one would expect. In fact it's probably worth noting that between 10 and 12 the libc major version wasn't even bumped. For all the bellyaching you're missing that each major version is supported for five years, and that unlike RHEL you'll get up-to-date ports with that (instead of the half-assed cobbled together ancient versions that linger with the RHEL model) so there's less incentive to upgrade until you're ready. That's a lot of hand waving over a non fucking issue.
If you were making a more earnest argument you'd probably have looked at the current version of the documentation and noted that it doesn't have the warning you quoted.
Note that FreeBSD 10.0 was released in January 2014, over nine years ago. 10.4, which is the oldest base jail image I have kicking around, was released in 2017. So the userland-kernel interfaces you're wringing your hands over have remained stable for nearly a decade across three or four different major releases.
Of all the concerns I'd have about running FreeBSD and/or jails in a prod environment, dealing with binary compatibility across different major versions is not even on my radar and not even remotely an advantage for docker.
# uname -rv
12.4-RELEASE-p1 FreeBSD 12.4-RELEASE-p1 GENERIC
# ldd /bin/ps
/bin/ps:
libm.so.5 => /lib/libm.so.5 (0x800254000)
libkvm.so.7 => /lib/libkvm.so.7 (0x80028b000)
libjail.so.1 => /lib/libjail.so.1 (0x80029e000)
libxo.so.0 => /lib/libxo.so.0 (0x8002a6000)
libc.so.7 => /lib/libc.so.7 (0x8002c5000)
libelf.so.2 => /lib/libelf.so.2 (0x8006b9000)
libutil.so.9 => /lib/libutil.so.9 (0x8006d4000)
# md5sum /bin/ps
ad8b5c8966c71e31cdc9603967860fa7 /bin/ps
# objdump -p /lib/libc.so.7 | tail -12
Version definitions:
1 0x01 0x0865f4e7 libc.so.7
2 0x00 0x077a28b0 FBSD_1.0
3 0x00 0x077a28b1 FBSD_1.1
4 0x00 0x077a28b2 FBSD_1.2
5 0x00 0x077a28b3 FBSD_1.3
6 0x00 0x077a28b4 FBSD_1.4
7 0x00 0x077a28b5 FBSD_1.5
8 0x00 0x077a28b6 FBSD_1.6
9 0x00 0x0f1efaa0 FBSDprivate_1.0
# iocage create -r 10.4-RELEASE
e950694e-6182-410d-bc70-1a8f6e340516 successfully created!
# iocage start e950694e-6182-410d-bc70-1a8f6e340516
* Starting e950694e-6182-410d-bc70-1a8f6e340516
+ Started OK
+ Using devfs_ruleset: 1007 (iocage generated default)
+ Using IP options: ip4.saddrsel=1 ip4=new ip6.saddrsel=1 ip6=new
+ Starting services OK
+ Executing poststart OK
# iocage console e950694e-6182-410d-bc70-1a8f6e340516
FreeBSD 12.4-RELEASE-p1 GENERIC
Welcome to FreeBSD!
Release Notes, Errata: https://www.FreeBSD.org/releases/
Security Advisories: https://www.FreeBSD.org/security/
FreeBSD Handbook: https://www.FreeBSD.org/handbook/
FreeBSD FAQ: https://www.FreeBSD.org/faq/
Questions List: https://lists.FreeBSD.org/mailman/listinfo/freebsd-questions/
FreeBSD Forums: https://forums.FreeBSD.org/
Documents installed with the system are in the /usr/local/share/doc/freebsd/
directory, or can be installed later with: pkg install en-freebsd-doc
For other languages, replace "en" with a language code like de or fr.
Show the version of FreeBSD installed: freebsd-version ; uname -a
Please include that output and any error messages when posting questions.
Introduction to manual pages: man man
FreeBSD directory layout: man hier
Edit /etc/motd to change this login announcement.
# ldd /bin/ps
/bin/ps:
libm.so.5 => /lib/libm.so.5 (0x800827000)
libkvm.so.6 => /lib/libkvm.so.6 (0x800a50000)
libjail.so.1 => /lib/libjail.so.1 (0x800c59000)
libc.so.7 => /lib/libc.so.7 (0x800e5e000)
# objdump -p /lib/libc.so.7 | tail -16
Version definitions:
1 0x01 0x0865f4e7 libc.so.7
2 0x00 0x077a28b0 FBSD_1.0
3 0x00 0x077a28b1 FBSD_1.1
FBSD_1.0
4 0x00 0x077a28b2 FBSD_1.2
FBSD_1.1
5 0x00 0x077a28b3 FBSD_1.3
FBSD_1.2
6 0x00 0x077a28b4 FBSD_1.4
FBSD_1.3
7 0x00 0x077a28b5 FBSD_1.5
FBSD_1.4
8 0x00 0x0f1efaa0 FBSDprivate_1.0
FBSD_1.5
# md5 /bin/ps
MD5 (/bin/ps) = 8a9c364705d29beb98503415428067cc
# ps
PID TT STAT TIME COMMAND
80454 39 IJ 0:00.01 login [pam] (login)
80455 39 SJ 0:00.03 -csh (csh)
80593 39 R+J 0:00.00 ps
You are 100% factually right and I shared the same level of frustration as it seems you had with GP. I just wanted to 1) say thanks for sharing knowledge!, but also 2) suggest taking a deep breath or a walk before replying to folks like GP. I’m often guilty of it myself, but trying to get better.
The point at which it becomes clear (e.g. digging up outdated documentation) that the argument is not being made in earnest is the point at which I stop taking the poster seriously. My comment was meant more for anyone else who might stumble on this thread than as a reply to OP.
That said, I wouldn't look to run FreeBSD in a production environment without a good reason and 13 looks to be where I'll jump ship to Debian for homelab stuff. The big thing holding me back before was ZFS, but that's essentially a non-issue now.
They don’t need to migrate, but it’s usually a good idea to keep up to date. It’s indeed somewhat of a thing if you have a lot of systems and you have uptime requirements. Plan accordingly and it’ll be fine.
Vnet jails are awesome. Totally changed the way I do networking and application hosting. Makes it pretty trivial to do things like set up a jail that can only talk to the internet through a VPN.
An extra NIC is fine if you've got slots, but you can also do an if_bridge in the main host, add the real nic and half of an if_epair; then move the other half to the vnet jail.
Potentially a non-vnet jail with the ipv4/ipv6 options set may also be an option. I used to use jails with those settings to squeeze multiple environments into a single host at Yahoo.
I use only if_epair and then pf to NAT it to the external world. I’ve only had issues with if_bridge, where I would either be able to make connections between jail and the external world, but not between host and jail or vice versa (between host and jail fine, but not between jail and external world). And it would act really flaky, like getting one ping response, followed by silence. pf is a lot simpler and more reliable.
IME, the base-system jail stuff is super easy to understand, use, and customize. When you start using "easy" tools on top of that, it rapidly becomes more complicated and opaque for non-trivial stuff.
It's awesome that you can do everything you would want with a jail with only 3 config files: rc.conf (for networking and start-up), pf.conf (for firewalling/routing), and jail.conf (for specifying jail params like mount points, network if setup, etc.).
VNET jails are great, especially now that they're stable, but I'd strongly recommend using a higher level tool. I've settled on iocage, but there are a few other tools out there that seem to have gained traction. Nobody sets up cgroups by hand, right?
Depends on your workloads and needs. It's not as mature as k8s and there are rough edges but if you're into the Hashicorp ecosystem and can run your workloads in FreeBSD jails it's pretty slick - think golang & rust apps, web, gaming servers, etc
I have a soft spot for FreeBSD. I hope someday we’ll have a lightweight container solution that works across the BSDs (macOS included), Linux and maybe even Windows. POSIX is all you need.
I made a very serious attempt at using this awhile back on Arch and it high-key sucks. Like, it's not really meant as a general purpose container runtime solution, it's meant for developing systemd and related software. The ergonomics, particularly around networking are quite bad and it literally requires that you're all in on systemd-networkd and resolved to get any networking at all.
The OS X kernel is a mashup of things; it's been too long since I looked, but IIRC the tcp code comes from just before FreeBSD added syncache (released in FreeBSD 4.5), so that should be after the release of jails in 4.0. But, Apple would have needed to integrate jails throughout the NeXT/Mach part of the kernel too.
And, you'd be missing out on features like VNET that came later (FreeBSD 8), unless Apple reintegrated with upstream, which is pretty rare for them. (They did update the FreeBSD userland, once)
I might be entirely off-base with this, but I think it probably has something to do with the roots of macOS' (or really, NeXTSTEP's) BSD component (BSD 4.3) pre-dating FreeBSD 4.0 (the first version that introduced jails) by 14 years.
It does seem like a feature worth porting, though.
They have that limitation, but it doesn't manifest in quite a simple way. If the container's glibc and all other binaries require Linux 6.3, you're not running it on a 5.18 host.
FreeBSD is developed as a whole unit. It makes it easier to reason that a 13.2 userland doesn't run on a 12.4 kernel.
> If the container's glibc and all other binaries require Linux 6.3, you're not running it on a 5.18 host.
FWIW, it would take quite a long time for glibc itself to start requiring Linux 6.3 on its current targets. Since glibc 2.24 (Aug. 2016), the minimum supported kernel has been at most Linux 3.2 (Jan. 2012), on every target that has existed that long. And that's the smallest that the gap between the glibc and Linux releases has ever gotten (except for new targets). Extrapolating, we'd have to wait for glibc 2.47 (Feb. 2028) to require Linux 6.3 (Apr. 2023), if that gap were to be repeated, something which I don't find particularly likely.
That is to say, with the exception of new targets, you'll rarely see a userland that absolutely requires the latest Linux kernel, even possibly if it suggests that kernel for the best support.
It's the "all other binaries" part that will catch you. You've completely overlooked that part.
For example: Per its own README, systemd requires Linux 3.15 at absolute minimum, recommends Linux 4.15 for baseline functionality to at least work, and specifies Linux 5.7 for full BPF functionality.
Depends what you mean by "Linux". You can possibly use a newer userland, but containers share the kernel with the host. That's kind of the whole point of containers: they're just a set of processes that are grouped together and configured with certain restrictions.
Unless that container uses kernel semantics that changed between kernel versions, or a feature that was added in the new version. Which invariably happens. See the RHEL 6/7 upgrade and the changes to syscall.
