You can use the Prometheus TSDB package, it's easy to import and use. Plus it's designed around mmap, so the kernel deals with the caching.

You don't have to use a whole Prometheus server to make something like this work.