This is an interesting write-up although I'd question the take home at the end: "It really doesn’t take much to find 10x wins."
Sometimes there's a really obvious problem that you can solve that will get you that kind of 10x performance boost. Often though you have to work your ass off for it or, even having found a problem where fixing it might even double your performance, the fix might be complex, touching multiple parts of the system.
In my experience, whilst you can often find quick wins, and sometimes these will be enough, there will always be cases where the analysis is much harder, or where more complex fixes are required.
(Disclaimer: I am a performance consultant, focussed mainly on SQL Server and .NET. My background is that years and years ago I lead the team that built Red Gate's ANTS Performance Profiler and ANTS Memory Profiler, and also worked on a number of their SQL Server tools.)
This. I would call the optimization described on the article as "low-hanging fruit": if you don't have the right DB indices or you are not batching your operations, you can find that kind of improvement easily, mostly just because your code was not following the "fast path".
Still very useful article even if just to show how to use profiling tools, which you will need to even for much-higher-hanging fruit.
> I'd question the take home at the end: "It really doesn’t take much to find 10x wins."
Hmm. I guess it depends on what you mean with "much". In terms of how capable our hardware is and how comparatively slow our software, yes, it really doesn't take much, there are typically orders of magnitude to be had. Sometimes that means ditching assumptions, however, and that can be hard. For example, I've often had pretty easy 10x, 100x and 1000x+ wins by removing SQL databases and associated ORMs.
The aim of the product is to display usage trends to it's customer, in real time.
My understanding as a non CS, self taught programmer is that such workloads are better served by NoSql DB s. Yes, this comes with the lack of acid compliance, but spotting trends in real-time (apparently) don't require acid compliance.
Please correct me if wrong as I'm getting down voted, which is rather amusing
Nosql is very big. Compare redis + cassandra + elasticsearch and you'll see. You have to look at: data-structures the db uses, query patterns, data-velocity, data-shape, etc etc. Basically how the db really works in a low level, to understand what is it good for.
Current case they: have a lot of data, need a lot of compression (or low overhead as possible), need as much query-selectivity as possible (reading as few rows as possible for each query) that's why they have a unique case of selective indexes.
Bonus: mongodb agg sucks. Seach online for comparison to pg or es agg.
