Good article, thanks for sharing. I've been working on one part of this problem space for quite a while too. I want ability to directly drill down into latency reasons and underlying application component threads' wall-clock time, instead of having to correlate various systemwide utilization metrics and try to manually connect the dots.
I'm using eBPF-based dimensional data analysis, starting from bottom up (every system is a bunch of threads, including distributed systems) and move up from there. This doesn't replace existing distributed tracing approaches for end to end request view, but gives you deep observability all the way down to each service's underlying threads' wall-clock time (where blocked, sleeping and why, etc).
At this year's P99CONF I will launch the first GA release of my (open source) 0x.tools xcapture eBPF collectors, with a reference implementation of a TUI tool, showing dimensional performance modeling on these new thread sampling signals (xtop).
This wasn't on Linux, but on one of the old-school commercial Unixes - a customer had memory leaks in some of their daemon processes. They couldn't fix them for some reason.
So they invested in additional swap space, let the processes slowly grow, swap out leaked stuff and restart them all over the weekend...
I wrote a chat server back in the 2000s which would gradually use more and more memory over a period of months. After extensive debugging, I couldn't find any memory leak and concluded the problem was likely inside glibc or caused by memory fragmentation. Solution was to have a cron job that ran every 3 months and rebooted the machine.
Interesting, works for me (3 different browsers on the same mac).
Oracle's blogs use some (JS based?) dynamic retrieval of the content, it's not all in "index.html" apparently, perhaps you have an ad blocker or something which prevents the dynamic load.
Indeed. Previously, when computing latency histograms from sampled events, I've abused DuckDB SQL cross join/cartesian products to generate zero values for "empty buckets" within a single SQL statement [1]. But makes the SQL unnecessarily complex & slower too, so I quickly moved that empty bucket rendering functionality to the frontend...
Feature author here. The new FILL uses linear interpolation (and extrapolation at the ends). There were a bunch of other algorithms requested (e.g., cubic splines), but the syntax was not clear.
I think the real solution is to add support for non-aggregate window functions in the catalogue. Right now, they are all hard-coded, but if they were extensible, there are a number applications in addition to filling that could be supported.
Asciinema is one of the best products/tools I’ve ever used. I mean the auth/upload CLI flow is so straight to the point and easy to use, despite having to cross a few layers (yep I know other CLIs have something similar, it’s just that Asciinema has never “gotten in my way” when doing something with it)
I have since converted the entire codebase to use modern eBPF functionality (task iterators, task-local storage maps, sleepable iterators capable of reading user-space memory of other processes, etc), with libbpf-based userspace frontend.
I'm using eBPF-based dimensional data analysis, starting from bottom up (every system is a bunch of threads, including distributed systems) and move up from there. This doesn't replace existing distributed tracing approaches for end to end request view, but gives you deep observability all the way down to each service's underlying threads' wall-clock time (where blocked, sleeping and why, etc).
At this year's P99CONF I will launch the first GA release of my (open source) 0x.tools xcapture eBPF collectors, with a reference implementation of a TUI tool, showing dimensional performance modeling on these new thread sampling signals (xtop).
A couple of 1-minute asciicasts of xtop are here: https://tanelpoder.com/posts/xcapture-xtop-beta/