And the vendors are happy to tell you to throw more cores at it instead of grapple with the core issues.

One of the biggest problems with spark imo is that people think that the magic cloud is going to make it go fast, when it actually means that the REPL is just muuuuch slower to iterate on.