Was anybody else disappointed that it's still just gdb for the debugger? I've thought for a long time that something better must be out there, especially a debugger that better handles multiple threads in an application, but every time I need to debug something I'm back in the 80s.
What would be better? How to improve the handling of multiple threads?
Every time I have tried a different debugger, I have always ended up coming running back to GDB. My only gripes with it are basically gripes with DWARF.
I still use dbx from Sun Studio. The now-oracle connection taints it, but it was from good old Sun. It was always better than gdb for threaded debugging.
Haven't checked if it is still available but I have it installed from long ago.
I don't get what it's missing other than a GUI. I can view stacktraces, view memory in various formats, create new commands on the fly or in .gdbinit... what are people missing?
I tried to love ddd for quite a few months. It's definitely a GUI, of the same era as Xfig. That's about the nicest thing I can say about it; I prefer the CLI.
What about lldb? Has anybody had extensive experience with both to say anything about the actual debugging experience? Does it offer anything more than just "infrastructure"?
Pernosco is certainly 2021... Debugging as a service...
Did anyone try it? It looks like it a web frontend for rr, but the idea seems alien to me, especially the pricing part. A subscription for a number of submissions per month?! If there is anything you can't really plan, it is bugs, one month, you may need only one or two, the next it will be 50 in a day.
It’s more than merely a front–end for rr, because with rr you are still inspecting a single moment in time in your program. You can rewind to past moments any time you want, but that fundamental limitation remains. With Pernosco you are really querying a database that contains all the information about all the states your program was ever in. It’s a superpower that you can purchase.
I think the price is generally worth it, if you have at least a couple bugs to use it on every month. If you have more than 5 per month but don’t want to go up a tier, I believe you can pay “a la carte” just for the extras. If you have enough people, and you can afford the hardware to run it, I really recommend the on–premises installation.
I see a lot of sudo in there, for debugging a program presumably not running as root. Is it possible with this tooling to debug core dumps as a normal user?
Systemd's core dump utility stuffs the files in /var/lib/systemd/coredump with 640 permissions and ownership of root:root regardless of who owned the process.
It might be paranoia on someone's part, but I agree that it causes a lot of unnecessary privilege escalation for people who want to actually fix the problem.
> Systemd's core dump utility stuffs the files in /var/lib/systemd/coredump with 640 permissions and ownership of root:root regardless of who owned the process.
This isn't true for +ACL builds of systemd, look at the output of `getfacl /var/lib/systemd/coredump/$foo` and you'll see.
It looks like coredumpctl might be suid? Or that the service is, and coredumpctl gets access through the service - so there's a certain elevation going on either way?
ls -l will note the presence of ACLs with a "+" next to the permissions. I typically `ls -l` or just have that aliased to ll if it isn't already… (but YMMV)
sudo should not be required if the coredumped process was running as your user, but this will only be true if systemd was built with +ACL (look at `systemctl --version` output).
Visual studios support for minidumps is top notch. You can load it, there’s a symbol server if you’re dealing with a binary not built locally, and then you can fire up the parallel stacks to visually see what all the threads are doing.
Coredumps and command line analysis with gdb by comparison leave much to be desired.
Linux was created in 1991, and its use is growing more than ever. When something was created doesn't really say anything about modern day usage.
GDB is still in wide use today and shows no sign of becoming history. If anything, its easy scripting features continues to prompt new tools based on GDB to pop up.
One day we may get a vim or even emacs mode for gdb, who knows? Until then, I will continue to "enjoy" interacting with the data of the program under debugger in the ed-style, like a True Programmer™.
The stories have it that back in the days some people could read actual core dumps with the tips of their fingers à la Braille, but surely that's just an urban myth.
Splitting debug symbols into separate packages is expected. Splitting them into a separate repository is a bit surprising but no big deal.
The problem with NixOS is that it doesn’t ship debug symbols at all.
Nix does at least make it easy to rebuild a package with debug symbols enabled, but:
- Some packages take a very long time to rebuild.
- IIUC, the standard builds are done without debug info enabled at all, rather than building with it enabled and then deleting it. Usually, whether debug info is enabled shouldn’t affect any other aspects of code generation, so rebuilding with debug info should produce a symbol-rich binary that’s compatible with existing core dumps for the original binary. But this is not always the case.
- If the package you want debug info for is a library, Nix will want to rebuild not just the library itself but everything that (transitively) depends on it. This is really a more general problem with Nix: it has zero notion of ABI compatibility. But always building with debug info would work around the problem.
One feature of Nix is to be able to rebuild something the same way on different systems. The article suggests that adding the debug symbols is a simple flag in the build expression.
Based on that, you shouldn't need the debug symbols in advance, though ymmv, when trying to debug an issue in container A on dev machine B.
Coredumps are a crucial source of information when investigating crashes and containerization doesn't invalidate any of that. Sure application restarts could minimize the impact of issues, but investigating them still is and should be standard practice. Furthermore, just because something isn't easy to do with k8s doesn't mean that it should be laughed at or scorned upon. I guess it's things like this that's causing articles to pop up that are somewhat skeptical of k8s.
Last time I tried to get coredumps working in a container (maybe four years ago) I got nowhere (it wasn't for a C/C++ application configuring everything to generate a core dump and then store it somewhere retrievable was too much).
K8s can generate coredumps if you configure it to, and they can actually still be useful in debugging certain classes of problems! However, at the multi-thousand person company I work at, there is like one guy concerned with core dumps.
This brings up a question I have had, is it not possible to put the debug info in a separate file that gdb could load instead of in the binary itself? If you could do this, you could then easily have xyz-dbg packages that only contained these files, or even an distro repository of these files that the debugger could download as needed.
This is exactly what most distributions do these days.
The debug symbol packages are usually called -dbgsym (Ubuntu, Debian, ...) or -debuginfo (openSUSE, Fedora).
This has been done automatically for almost a decade in most distros' packaging systems. GDB usually tells you what are the names of the debug packages you need to install for whatever you are trying to debug.
The "distro repository of files that the debugger can download as needed" is called a debuginfod URL. E.g., for Debian : https://debuginfod.debian.net/ , openSUSE : https://debuginfod.opensuse.org/ , etc.
This is a more recent development and distros are just starting to turn it on by default. But if enabled you'll basically see GDB start downloading symbol files automatically ala Windows when you run it. It even downloads the source code as it goes.
Personally I think the latter is a security risk (since it sends to a server the hash/build-id of everything you try to debug), but I can't argue it's not convenient.
I wanted to ask you if the "-gsplit-dwarf" is a 100% complete alternative to strip the symbols as done previously. I ask you because some modern projects[1] still prefer to compile with symbols and strip afterward.
[1] https://github.com/dotnet/runtime/blob/main/eng/native/funct...
If you are in control of your build, ideally you want to embed build IDs and use a symbol server. Here is an older article from Bruce Dawson on this (unfortunately way underdeveloped) area of Linux:
