It's all fun and games until you're stuck for a hour downloading 600MB of updated packages over a metered LTE. The same is with RAM usage: 512MB was enough for a phone back in 2014, now a smart TV with 2GB is barely capable of multitasking. Sure, binary sizes don't matter in most contexts. But when they do, it's a PITA.
Sure, but we're talking about an application written for a cloud/hosted environment in a datacenter somewhere. nicking at the size of a statically linked binary meant for production grade environments with fast computers and fat pipes feels overly pedantic no? Especially when we're talking about a mere 100MB
What you're kind of missing is that the S5 was a flagship phone. Generally, one has to save for more than a month to afford a purchase like that. The idea of working an extra month so that some FAANG prick meets their KPI by cutting corners on optimization doesn't even look like feudalism. It looks like idiocracy. Paying the lip service of fat shaming code bloat is the cost-effective option by comparison :)
Don't FANNG people obsess over bloat because they're trying to reach billions of customers? It might not seem that way since their pages are bigger but I'd be surprised if they were happy to leave 10s of millions of customers on the table.
They're just poster children for the particular brand of disdain $100k+/year "tech workers" bear for their users: they make enough for the shiniest of toys, so they're too far above spending their valuable time to make their software run smooth on our $100 crap phones. Nevermind that each Fb client update likely produces hundreds of tons of toxic trash called gadgets. Sure, sometimes they do optimizations. Generally, though, both Fb and Google keep exploring the physical limits to code bloat. Remember that one time that Fb hit the JVM class count limit?
FAANG are the worst offenders. Didn't facebook employ ungodly hacks to unload/load parts of the android app to navigate around the 65k method limit of dex? Have you looked at the js monstrosity of the Google hardware shop website?
You misunderstand the point. You're comparing a 1080p phone to a 4k television when texture memory is what will take up the vast majority of ram. Code footprint is pretty irrelevant.
Still the TV does fine with 2GB. Doesn't seem fair to complain.
I wasn't speaking of a 4k TV, but still, this doesn't check out. A single 2160p framebuffer is 8MPix, or 32MiB. Not counting the original FB size, the extra 1.5GiB are enough for 48 whole framebuffers. You don't need that much image data all at once, the number is ridiculous. No, I believe it's just that the code became that much less efficient.
Think of each app and all the texture content that needs to be loaded. App textures get 4x as big, all things being equal. You see a 4x change in ram across those devices.
I can guarantee 100% my Prometheus instance will never be running on metered LTE. If such a situation arises then my operational metrics are the least of my concern.
This one does not even make sense - 100 megs for a binary for centralized metrics? Who would even notice next to the OS and metrics storage.
By design you should not install prometheus on every server you monitor - it's designed to scrape metrics
Its a database, webui with support for email, webhooks, slack, pagerduty, aws api and many others. 100megs does not sound like a lot for all Pormetheus provides
Sure, lots, go look for studies on estimation of defects based on LOC and project size/complexity (they go back to the 1970s). But you don't need to look, the principles are simple.
Unless an application is filled with JPEGs or uncompressed arbitrary data files, its size reflects lines of code (machine code, interpreted code, etc). Bigger the app, the more lines of code.
Every line of code has a non-zero bug probability. Every new line of code increases probability. More lines of code, higher probability. Bugs include security bugs; higher probability of bugs, higher probability of security bugs.
CPU cache is finite. Only so many lines of code can be cached or optimized. Larger size takes up more room in memory, which when combined with lot of other gigantic apps, means less memory for heap space, disk cache, etc. Larger size also takes up more room on disk, which adds up when you don't delete old builds on disk and loop over a build process. Since larger size means more lines of code, that means longer compile times, which means longer wait every time you change a line and need to recompile, copy an artifact somewhere, retest.
More lines of code means more code executed. If you have 10 lines of code in a function, and you add 100 lines to it, the compiler doesn't just optimize away all 100 new lines, it's going to add more machine code and code paths. Unless you only ever add new code paths, some of that new code will extend existing code paths or add instructions, and that means more CPU cycles to complete execution. (Same concept for interpreted code)
More lines of code means more code paths. More code paths increases complexity. The more code paths, the longer and more difficult testing gets to the point you can't even develop enough tests to cover all the code paths, so it's impossible to even find all the bugs. More complexity leads to difficulty in humans understanding and working with the codebase, and difficulty in understanding leads to slower and more error-prone development.
Larger means more network bandwidth, meaning file transfers take longer, increasing speed to iterate and producing worse UX. If people download your app every 10 minutes in their CI/CD pipeline, larger size means more network bandwidth used. "Free" CDNs have limits; the larger a project gets, the more file size affects network performance, reliability, and cost. If you pay for bandwidth, a 100MB file costs 100x more than a 1MB file.
The more apps you use that are big, the more every one of these effects increase. One big app you might not notice. 100 big apps lead to noticeable slowness, bugs, less memory, less disk space.
It's kind of ironic reading this comment given that at the time you posted it, I was screaming at gcc's stupid code generator for wasting bytes recreating constants that were already there in that very register! That code needs to fit in a couple hundred bytes..
And half an hour ago I was (once again) checking out hosting providers and lamenting the fact that most don't seem to offer support for loading custom ISOs so I could install a 30 megabyte distro and make the most out of the cheap plans that only offer something like 10 gigabytes of storage. Half of it is wasted after you install one of the these obese mainstream distros.
I got a VPS from Hetzner last year but they decided to block my home IP. After reading some anecdotes on the internet, I had to conclude they're exactly the kind of company I want to avoid (large, opaque, they employ weird algorithms/heuristics to flat out reject customers or suddenly take down their servers, no warning, you can't get an explanation, you're just fucked, just like when Google decides to arbitrarily block you; I've been there).
IMO the point of a hosting provider is supposed to be that you can have some peace of mind and not worry about your shit breaking (that's still a worry as I continue to host everything at home). Instead with providers like this, you worry about them breaking your shit.
Author doesn't even say why they object to the size. Are they aware that file-backed executables are paged on demand and only the active parts of the program will be resident?
Granted these days everyone is used to applications consuming massive amounts of drive space. But perhaps they're using legacy hardware for a home lab, or a IoT device with limited disk space.
From a security stand point, reduced application code decreases risk. It was service discovery code he removed, what if it reached out to discover services on application start up, that's a potential attack vector.
Does it actually reduce the risk? Sure if you audit, its easier to identify the risks, but a windows 98 program is going to be full of vulnerabilities while being small. Being small doesn't remove the vulnerabilities
> From a security stand point, reduced application code decreases risk. It was service discovery code he removed, what if it reached out to discover services on application start up, that's a potential attack vector.
Agreed. I've see a similar pattern with certain open source libraries.
The first example I think of is the spf13/viper [1] library, used to load configuration into go applications. Viper is equipped with code for reading config from various file formats, environment variables, as well as remote config sources such as etcd, consul. If you introduce the viper library as a dependency of your application to merely read config from environment variables and YAML files in the local filesystem, then your go application suddenly gains a bunch of transitive dependencies on modules related to remote config loading for various species of remote config provider. It's not uncommon for these kind of remote config loading dependencies to have security vulnerabilities.
As well as the potential increased attack surface if a bunch of unnecessary code to load application configuration from all manner of remote config providers ends up in your application binary [2], if you work in an environment that monitors for vulnerabilities in open source dependencies, if you depend on an open source library that drags in dozens of transitive dependencies you don't really need, it adds a fair bit of additional overhead re: detecting, investigating and patching the potential vulnerabilities.
I guess there's arguably a "Hickean" simple-vs-easy tradeoff in how such libraries are designed. The "easy" design, that makes it quick for developers to get started and achieve immediate success with a config loading library, is to include code to load config from all popular supported config sources into the default configuration of the library, reducing the amount of steps a new user has to do to get the library to work for their use case. A less easy but arguably "simpler" design might be to only include a common config-provider interface in the core module and push all config-provider-specific client/adaptor code into separate modules, and force the user to think about which config sources they want to read from and then manually add and integrate the dependencies for the corresponding modules that contain the additional code they want.
edit: there has indeed been some discussion about the proliferation of dependencies, and what to do about them, in viper's issue tracker [3] [4]
[1] https://github.com/spf13/viper
[2] this may or may not actually happen, depending on which function calls you actually use and what the compiler figures out. If your application doesn't call any remote-config-provider library functions then you shouldn't expect to find any in your resulting application binary, even if the dependency is there at the coarser-grain module dependency level
[3] https://github.com/spf13/viper/issues/887
[4] https://github.com/spf13/viper/issues/707
Image pull size for a container is likely the concern. It could shave a few seconds off a regularly-run integration test. If it's run via on-demand build agents, then there's no image cache.
I was estimating off the speed it takes to pull images to my local computer where the limiting factor appears to be something other than my internet connection so either the image extraction process or a docker hub throttle.
That only helps if the code is well segregated by usage. Looking at the ELF symbol table for prometheus-2.33.0-rc.1.linux-amd64, it's not clear to me this is the case. Not sure how it's ordered. Lexical import order? Anyhow, without profiling how could the compiler know how to order things optimally?
I think this is one of those cases where, in the absence of profiling or some other hack (e.g. ensuring all routines within a library are cleanly segregated across page boundaries within the static binary and the I/O scheduler doesn't foil your intent), dynamic linking would prove superior, at least for such large amounts of code.
sorry not to make it obvious in the article, I'm planning to run it in small iot pi based device locally. So having something small and fast is preferable, however the runtime performance is a more important thing I haven't touch.
Prometheus is rather efficient, but it's focus is a little different than yours. Its designed to for large scale collection of metrics, scraped from many remote endpoints
You can run it locally but the "prometheus" way for iot env would be a central prometheus server that scrapes the iot devices running a prometheus exporter, which tend to be very light weight
Prometheus works fine as-is on Pi devices. You'll spend most of your memory on ingestion buffering. I did the same tests as you did a while back, it only saves like 25-50MiB of memory IIRC.
The only thing you really need to worry about on a Pi is that the default kernels are still 32-bit, and are set to 2GiB kernel boundary. So you'll be limited to how much TSDB storage can be mmap'd unless you switch to a 64-bit kernel.
You may want to consider agent mode on your IoT device, and stream the data to an external server/service.
I'm curious -- is it the binary size that's a problem, or the resident size in memory? Demand paging should help, although you'd be stuck with carrying the enlarged binary.
ps RSS measurement is pretty good for a start (although note in a forked process, shared copy-on-write pages are both reported as RSS for both the parent and child process).
top reports this as RES.
IIRC, debugging information is in a separate part of the process, so it's not loaded until it's used. Does that make it free? Probably not quite, but the kernel can ignore it until the process (presumably via its debugger) looks at it.
Auditd_2.8-amd64.deb is 194kb on debian, rsyslog_8.32-amd64 is 411kb, and they both support centralized auditing and log collection from multiple hosts.
This doesn't seem a fair comparison. Prometheus is statically linked like all Go applications, and those packages are not. You can debate the merits of that, but if you compare a "only rsyslog" server vs a "only prometheus" server the 2 will be much closer in size.
Even journalctl is only 90kb, and the entire combined systemd package is 4mb (but that's including all the documentation and a dozen other different binaries)
I work in a lot of situations with hard or soft resource limits where I actually do need to count bytes and/or CPU cycles, so it's bizarre to see anyone shrugging about distributing tens of megabytes of fat for literally no reason.
One thing Microsoft got right a long time ago was separating out debug symbols into their own file by default. I think that's still awkward on Linux.
At least for .deb and .rpm packages, the default build process automatically extracts debug symbols into separate packages. Eg the process of building package `foo` also produces `foo-dbgsym` and `foo-debuginfo` packages respectively that contain debug symbols for every involved binary and library, while `foo` contains the stripped files.
So anyone who wants to debug a coredump / live process just installs the corresponding -dbgsym / -debuginfo package and now gdb has all the debug info it needs.
Distros have also started incorporating debuginfod into their repos so that gdb can download symbols automatically. So you don't even have to hunt for the right debuginfo package.
