Author doesn't even say why they object to the size. Are they aware that file-backed executables are paged on demand and only the active parts of the program will be resident?
Granted these days everyone is used to applications consuming massive amounts of drive space. But perhaps they're using legacy hardware for a home lab, or a IoT device with limited disk space.
From a security stand point, reduced application code decreases risk. It was service discovery code he removed, what if it reached out to discover services on application start up, that's a potential attack vector.
Does it actually reduce the risk? Sure if you audit, its easier to identify the risks, but a windows 98 program is going to be full of vulnerabilities while being small. Being small doesn't remove the vulnerabilities
> From a security stand point, reduced application code decreases risk. It was service discovery code he removed, what if it reached out to discover services on application start up, that's a potential attack vector.
Agreed. I've see a similar pattern with certain open source libraries.
The first example I think of is the spf13/viper [1] library, used to load configuration into go applications. Viper is equipped with code for reading config from various file formats, environment variables, as well as remote config sources such as etcd, consul. If you introduce the viper library as a dependency of your application to merely read config from environment variables and YAML files in the local filesystem, then your go application suddenly gains a bunch of transitive dependencies on modules related to remote config loading for various species of remote config provider. It's not uncommon for these kind of remote config loading dependencies to have security vulnerabilities.
As well as the potential increased attack surface if a bunch of unnecessary code to load application configuration from all manner of remote config providers ends up in your application binary [2], if you work in an environment that monitors for vulnerabilities in open source dependencies, if you depend on an open source library that drags in dozens of transitive dependencies you don't really need, it adds a fair bit of additional overhead re: detecting, investigating and patching the potential vulnerabilities.
I guess there's arguably a "Hickean" simple-vs-easy tradeoff in how such libraries are designed. The "easy" design, that makes it quick for developers to get started and achieve immediate success with a config loading library, is to include code to load config from all popular supported config sources into the default configuration of the library, reducing the amount of steps a new user has to do to get the library to work for their use case. A less easy but arguably "simpler" design might be to only include a common config-provider interface in the core module and push all config-provider-specific client/adaptor code into separate modules, and force the user to think about which config sources they want to read from and then manually add and integrate the dependencies for the corresponding modules that contain the additional code they want.
edit: there has indeed been some discussion about the proliferation of dependencies, and what to do about them, in viper's issue tracker [3] [4]
[1] https://github.com/spf13/viper
[2] this may or may not actually happen, depending on which function calls you actually use and what the compiler figures out. If your application doesn't call any remote-config-provider library functions then you shouldn't expect to find any in your resulting application binary, even if the dependency is there at the coarser-grain module dependency level
[3] https://github.com/spf13/viper/issues/887
[4] https://github.com/spf13/viper/issues/707
Image pull size for a container is likely the concern. It could shave a few seconds off a regularly-run integration test. If it's run via on-demand build agents, then there's no image cache.
I was estimating off the speed it takes to pull images to my local computer where the limiting factor appears to be something other than my internet connection so either the image extraction process or a docker hub throttle.
That only helps if the code is well segregated by usage. Looking at the ELF symbol table for prometheus-2.33.0-rc.1.linux-amd64, it's not clear to me this is the case. Not sure how it's ordered. Lexical import order? Anyhow, without profiling how could the compiler know how to order things optimally?
I think this is one of those cases where, in the absence of profiling or some other hack (e.g. ensuring all routines within a library are cleanly segregated across page boundaries within the static binary and the I/O scheduler doesn't foil your intent), dynamic linking would prove superior, at least for such large amounts of code.
sorry not to make it obvious in the article, I'm planning to run it in small iot pi based device locally. So having something small and fast is preferable, however the runtime performance is a more important thing I haven't touch.
Prometheus is rather efficient, but it's focus is a little different than yours. Its designed to for large scale collection of metrics, scraped from many remote endpoints
You can run it locally but the "prometheus" way for iot env would be a central prometheus server that scrapes the iot devices running a prometheus exporter, which tend to be very light weight
Prometheus works fine as-is on Pi devices. You'll spend most of your memory on ingestion buffering. I did the same tests as you did a while back, it only saves like 25-50MiB of memory IIRC.
The only thing you really need to worry about on a Pi is that the default kernels are still 32-bit, and are set to 2GiB kernel boundary. So you'll be limited to how much TSDB storage can be mmap'd unless you switch to a 64-bit kernel.
You may want to consider agent mode on your IoT device, and stream the data to an external server/service.
I'm curious -- is it the binary size that's a problem, or the resident size in memory? Demand paging should help, although you'd be stuck with carrying the enlarged binary.
ps RSS measurement is pretty good for a start (although note in a forked process, shared copy-on-write pages are both reported as RSS for both the parent and child process).
top reports this as RES.
IIRC, debugging information is in a separate part of the process, so it's not loaded until it's used. Does that make it free? Probably not quite, but the kernel can ignore it until the process (presumably via its debugger) looks at it.
