Good point! Log levels are pretty useless most of the time.
I would add that there can be value in having 2 log levels: verbose and non-verbose. It is helpful if you can selectively switch on verbose logging by user or by API endpoint.
In one application which I maintain, when verbose logging is switched on for a particular user, TCP/UDP socket objects are automatically wrapped and packet captures are logged, only for packets sent/received while servicing that particular user's requests. This has been a lifesaver when debugging things like weird, transient authentication problems stemming from upstream providers.
> It is helpful if you can selectively switch on verbose logging by user or by API endpoint.
We currently use two log levels:
- When 'debug = true', debug logs are printed immediately (like a DEBUG log level)
- When 'debug = false', debug logs go into a buffer: if the request-handler succeeds, its debug buffer gets discarded. If it catches an exception, the buffer gets printed.
This avoids the main problem of log levels, which is having to guess up-front which level we might want (and inevitably get it wrong, and have to try re-creating a problem with more verbose logging!)
Nice! It sounds like you would do well to capture the debug output if a request handler takes an unusually long time to return (not just if an uncaught exception occurs).
This completely overlooks many things such as logs used as metrics or even just using them to reason about the state of your application. What if your application isn't throwing errors but some thing is still broke Or you're shipping bad data?
IMO, the entire point of logs is to be able to ask questions of and reason about the current state of your application. If you're only logging errors that you can't recover from you may as well just throw and exception and restart.
> If you're only logging errors that you can't recover from you may as well just throw and exception and restart.
Unironically tho this is a good decision if you design for it from the beginning crash-only software style. Most of what I log is INFO and WARN level because unexpected combinations of business-level state are where the real subtle nasty bugs are. Nil reference or whatever can just crash who cares.
It absolutely matters. Reading messages from a message queue? Have users submitting some subtly bad edge case you've never organically seen? Just a crash-restart isn't saving you, and worse because you have a discipline of who cares, you probably have inadequate code helpers to diagnose the scenario that triggered your flaw. If you're business is dead simple, whatever but there's too much code in the world to be pumped into such a limited way of thinking.
You have to output a lot of logs before you fill up even a single large consumer-grade hard drive, especially given logs are typically compressed when rotated.
It's usually only when you involve ELK or something like that your logs start to get big. Which in turn is typically necessitated by over-complicated distributed software design.
If you're at the scale where this actually matters and you're serving millions or requests per second from a worldwide user-base, then affording storage for the logs really shouldn't be a problem anymore (idk, with the possible exception of Twitter)
> You have to output a lot of logs before you fill up even a single large consumer-grade hard drive, especially given logs are typically compressed when rotated.
This is a good point - a RAID array of a few HDDs/SSDs scales surprisingly far and is cheaper than many of the cloud services out there, though whether you can or can't use either approach probably relies on compliance requirements and such.
I will definitely add that logs can compress really well - to the point where it's been close to a year since I added Logrotate to a project that didn't have it before, for a pretty basic setup, and I haven't had the need to even look at how many archives are currently retained, given that the disk usage has changed very slightly. And that's for multiple systems that filled up the available storage in months previously.
Of course, my personal gripe is that most of the logging solutions out there are rather complex - something like Graylog feels like one of the simpler self-hostable options while still being fully featured, but in my experience anything that runs ES is really resource hungry. Sometimes it feels like MariaDB/PostgreSQL would be good enough for most of the simpler low logging volume setups out there - if you don't want to manage logs as files, want to ship them somewhere, but don't want the receiving system to be too complex either.
Except you know when you.actually want to do something valuable with all those logs. You _should_ be creating logs (signals) to be valuable in some way (diagnostics, alerting, canaries statistics), etc. If you're just dumping logs into opaque blobs that are never looked at them sure write them to blobs to your heart's content and have fun hunting and pecking for reasons you're users are already screaming at you. That strategy is fine, but the limitations are clear. It's reactive.
Is it audit logs for security or due to some regulatory requirement? Then huge blobs are fine. Desirable, even.
Transaction logs for machine-loading so you're able to replay an application's state at any given moment in time? Yeah probably gonna end up with huge blobs again.
And what are you going to do when you need a human to read sixteen trillion bytes of compressed logs streamed off a single SATA disk?
Once you face the fact that the performance of a single SATA disk means you can't search the logs in any quick time, and nobody can possibly read that much log data, so nobody will use it, you start to see it as a hoarding disorder not a useful tool.
It's not unheard of to need to retain years even decades worth of logs due to regulatory compliance. Nobody is reading them, they just need to exist. In that scenario you'll probably keep the current year or so fresh on a mechanical drive and past years on tape.
As someone currently working on the employee side of a remote.com setup, it’s been great.
They have a much better understanding of my local laws, tax systems etc than the US-based HR and Payroll departments at companies I’ve worked for in the past.
Remote isn't a hiring platform, it's a service companies can use to manage HR compliance. You get hired by the company, and your employment goes through Remote for compliance with local laws.
