> NVMe vendors are also looking into supporting offloading functionality to BPF for storage devices with a new framework called eXpress Resubmission Path (XRP), though this effort is currently stalled due to BPF not being standardized

Neat potential application!

Worth pointing out AMD is fulfilling the first most obvious listed goal in this article, offloading network processing, with Nanotube, a eBPF XDP -> fpga suite, based around LLVM & Vites-HLS. https://www.phoronix.com/news/AMD-Xilinx-Nanotube-Compiler

Associating the desire of NVMe vendors to allow users to ship down eBPF programs to run on the device and XRP is a major mistake in the article. XRP has nothing to do with what the NVMe vendors want to do, and XRP is a pure kernel solution that doesn't need any participation from NVMe vendors. I think it's unclear whether XRP even has real value - it certainly may, but I believe the benchmarking in the paper was deeply flawed[1].

1. https://github.com/xrp-project/BPF-KV/issues/3

I'm not closely tied to what the NVMe vendors want, so you could be right, but I very much doubt you are given that Christoph didn't flag this when he reviewed the article.

Edit: And to be clear, from my understanding of XRP, the device itself calls back into a BPF function in the NVMe driver. That requires some notion of standardization. It's not exactly offloading directly to the storage device, but the storage device still relies on some standardized behavior in the BPF program, such as divide by zero, what instructions are supported in the ISA, etc.

I am very closely tied to what the NVMe vendors want, having written the first internal draft of the proposal to the standards body (since that draft many smart people have taken the pen and done a lot of great work).

XRP is unrelated to offloading eBPF to NVMe devices.

Sure, whatever. "Offloading" was perhaps a poor choice of wording, but it is related to the standardization efforts. The NVMe vendors don't want to be calling out to BPF programs in the driver if the runtime semantics are not standardized.

XRP is a regular BPF hook in Linux and requires no additional standardization. The device never "calls out to BPF programs in the driver" - it generates a normal completion interrupt and Linux runs a BPF hook in the completion path. This is no different than other kernel BPF hooks elsewhere and doesn't provide any additional reason or need to standardize BPF.

The article misstated that XRP was a framework used for offloading BPF programs to NVMe devices. That's not correct, and XRP is not one of the emerging use cases for BPF that is driving standardization.

XRP initially looks to be focused on moving work upstream into the kernel not devices.

But absolutely, once you start shipping ebpf into the kernel, people do quickly start asking, "how can we hardware accelerate that?". Having standards would be helpful.

> the device itself calls back into a BPF function in the NVMe driver

This statement contradicts itself. A driver (a “kernel module” in Linux lingo) runs in the kernel, and sure, that driver can call out to BPF or whatever else it desires, but then that isn’t the device doing so, that’s your computer (running your Linux kernel, in turn executing the aforementioned kernel module / driver) doing so.

Restated from another perspective, drivers don’t run on devices. Something may run on devices too, but that’s different, and we’d call that something like “firmware”.

Edit: the intended takeaway being that device manufacturers/designers should have little to gain from BPF being standardized (unless that BPF is being executed on the device, as this now implies an API contract between device and host) — a driver can always declare that the semantics is whatever the Linux kernel does and call it a day.

This has been a long time coming… eBPF is the de facto tracing, networking and security stack on basically every platform that mankind cares about right now. Great to see the standardization kicking off now.

eBPF Foundation is the right body for this standard, as that’s where experts (from users and vendors) come together. I have had the privilege of working with the first engineers who designed and upstreamed eBPF. I have no doubt these geniuses can put a solid standard together.

What is the licensing situation with eBPF now?

Last time I looked (a long long time ago), BPF was ok, because of the "B", but eBPF was linux-only in such a way that loading eBPF program into kernel made your userspace program a derived work of linux kernel and forced GPL-2 on the userspace.

Has this changed? Is there a good writeup about eBPF wrt licensing?

edit: quoting current manpage on struct bpf_attr member "license":

    license is a license string, which must be GPL compatible to call helper functions marked gpl_only.  (The licensing rules are the same as for kernel modules, so that
    also dual licenses, such as "Dual BSD/GPL", may be used.)

So what is gpl_only now? Used to be the whole of eBPF.

  > eBPF was linux-only in such a way that loading eBPF program into
  > kernel made your userspace program a derived work of linux kernel
  > and forced GPL-2 on the userspace.

Copyright doesn't work like that:

1. If some particular eBPF program is a derived work of the kernel, then whether it's loaded or not doesn't matter. Distributing that program (such as embedded into an ELF binary) would still require being accompanied by its source code (/ a written offer, etc).

2. If some eBPF program is not a derived work, then loading it into the kernel doesn't cause the userspace program to become a derived work. Programs distributed as interpreted bytecode are not automatically derived works of their interpreter.

3. Even if a program depends on GPL'd code in such a way as to be a derived work when compiled, its own license only needs to be compatible with the GPL, it doesn't need to be GPL'd itself. For example, it's legal to write and distribute a Linux kernel module licensed under various permissive licenses (MIT, 3-clause BSD, etc).

> If some particular eBPF program is a derived work of the kernel, then whether it's loaded or not doesn't matter.

IANAL, but as eBPF programs are distributed as bytecode and only JIT-compiled and linked when they are loaded into the kernel, I think there's an argument to be made that it doesn't become a derivative work until it's loaded.

No such argument is needed (or possible). If you've written an eBPF program that is a derived work of the Linux kernel (for example by copy-pasting Linux code into it), then it's a derived work whether or not it's ever loaded.

Conversely, if your eBPF program is not a derived work of the Linux kernel (because you hand-coded its bytecode in a hex editor via reference to a clean-room eBPF spec) then loading it won't change that.

If you wrote it in C and compiled it with Clang, then whether the compiled eBPF bytecode is a derived work would require (1) looking at the source code and (2) inspecting the compiled output to see if anything unexpected got pulled in via #include.

That is not at all what is implied by the "same as kernel modules" text in the bpf manpage. What the manpage says is, essentially, any reference to gpl_only stuff is similar to using EXPORT_SYMBOL_GPL functions in modules, that "cannot be used without the user being a derived work of kernel" according to GKH.

Now, if the userspace program that loads this eBPF is fundamentally coupled to that eBPF program, the argument carries, no?

  > And, wanting to use some eBPF produced by clang, that person comes
  > to this "license" member in the manpage. What to do? Call lawyer?
  > Call police? Call reddit?

They could try reading, which is effective in this situation given that the quoted portion of the manpage is a concise and easily-understood description of the rules.

> BPF programs are written in C, and compiled into BPF bytecode. Like other bytecode instruction sets, BPF programs are platform-independent and just-in-time (JIT) compiled.

That sounds similar to what Inferno was doing. Though I'm unaware whether Inferno did it in kernel space, but PhantomOS definitely did - their argument was that, since you can't run native code directly, only through the bytecode, memory safety can be achieved by simply not giving a program any reference to a memory address it's not supposed to access. Apparently, this approach achieves memory protection without using an MMU.

Would be cool to see if Linux eventually goes down that path.

Inferno uses C (Plan 9 C dialect) for the kernel and Limbo for the complete userspace, with exception of some C libraries like Tk for the GUI.

DisVM runs in userspace.

https://github.com/inferno-os

Java did this in the 90s. Not sure why we need yet another bytecode format.

Bytecode format itself is not interesting (PhantomOS runs Java bytecode, Inferno was a competitor to Java in the 90s), but the way it is executed.

Java runs bytecode in a VM as a userspace process, eBPF runs it in kernel space. That in itself is interesting.

Does running in kernel space justify an entirely new bytecode?

Justify? I don't know. Require? I also don't know, but intuition tells me that it does, at least in the case of Linux.

It's about time that BPF is being standardized.

It increasingly being used not only in the conventional field of network and security accelerations, now it's also being used for other uses cases for example 5G Open RAN [1]. For more comprehensive review of latest eBPF based applications please check it here [2].

[1] Taking 5G RAN Analytics and Control to a New Level:

https://www.microsoft.com/en-us/research/publication/taking-...

[2] Extended Berkeley Packet Filter: An Application Perspective:

https://ieeexplore.ieee.org/abstract/document/9968265

Is there a full userspace version of Linux eBPF now? Last time I looked the userspace implementations were all very partial and in particular didn't include the verifier.

I don't want an independent userspace implementation, I want the verifier the kernel uses. I'm was looking into this and I'm not sure why it can't be pulled out. I guess depending on what you're plugging the eBPF into the environment (functions, etc) may be tightly coupled to the kernel, but surely the loop analysis and stuff could?

It basically forces you to program on a live system. No CI, no static analysis, no editor help, etc.

UBPF https://github.com/iovisor/ubpf/ and rbpf https://github.com/qmonnet/rbpf are userspace bpf interpreters.

Via https://github.com/zoidbergwill/awesome-ebpf#user-space-ebpf

Not sure if there's a full user space runtime for BPF progs, but there are lots of user space tools which are useful for working with BPF progs. For example, you can use veristat (https://github.com/libbpf/veristat) to easily see whether a BPF prog would be verified, and collect statistics, etc.