I don't recommend Splunk because I don't want someone recommending we run Splunk onsite ever again.
Running your own log infrastructure is the absolute worst. As in, we had to put a staff devops engineer on just that for 2 months the last time the company I was at needed an upgrade.
Correct, it's not uncommon to have one engineer full time on an internal Splunk infrastructure at a mediumish org. Not everything has to be Cloud. This is what people are paid to do.
Running your own log infrastructure is the absolute worst. As in, we had to put a staff devops engineer on just that for 2 months the last time the company I was at needed an upgrade.