I'm wondering if anybody lives in the hypothetical perfect world scenario the author writes about. I'm at one of the larger tech companies and it's inconceivable that something like this could exist (though the churn here is extremely high - a mature shop with longstanding membership could implement the hypothetical in some form). Everything sounds nice when dreaming it up in one's head, but discounts the reality of things. You can only lead a horse to water so many times before recognizing that they just will not drink the water themselves - some people actively refuse to implement solutions, no matter how convenient the building process might be. And then the more you burden them with things like SLAs, performance, etc., the more of shit show things become.
There are some forward-moving, solid "soft skill" analysts/data scientists that can make this happen. But by and large they shouldn't all be held to this standard. Maybe my standards/expectations have been soured too much and I'm too pessimistic, but as a whole they're just not cut out for this kind of stuff. Which is fine - being a "doer" is easy to begin with, and over time the more that you're able to automate as a data engineer, the more trivially easy ETL/everything else becomes.
I've experienced that hypothetical perfect world, it comprised teams of data scientists, developers, and system engineers working on making progress towards multiple distinct 'business insights' and operating within a big data ecosystem. People's job roles were tightly defined but their participation in tasks was largely determined through self-selection and reputation.
I think you may be experiencing a local minimum, so to speak. Data scientist programmers who "simply refuse" to think about basic productionization criteria like SLAs shouldn't remain employed.
They shouldn't, but they do because understanding a little bit of mathematics is at a premium these days and it's very much in vogue to crap all over computer science majors even though we had the same kind of mathematics curriculum as they at the university.
Really? I had one sort of similar because I took as many cross listed math electives as possible and avoided any software type courses, but even then I learned a lot less analysis, algebra, geometry than a good math student would learn
There are some forward-moving, solid "soft skill" analysts/data scientists that can make this happen. But by and large they shouldn't all be held to this standard. Maybe my standards/expectations have been soured too much and I'm too pessimistic, but as a whole they're just not cut out for this kind of stuff. Which is fine - being a "doer" is easy to begin with, and over time the more that you're able to automate as a data engineer, the more trivially easy ETL/everything else becomes.