Airflow and tools like it are probably the biggest reason for the shift. Another issue is the need to integrate different technologies requires having the skills of a software engineer.
When the landscape for tech in DE was oracle, mysql and Cognos, DE's didn't need to know about OOP or consensus algorithms. Because the landscape now includes hadoop, redshift, kafka, spark, airflow, notebooks, TiDB and lord knows what else, DE's need to have most of the skills of a software engineer to be successful.
When the landscape for tech in DE was oracle, mysql and Cognos, DE's didn't need to know about OOP or consensus algorithms. Because the landscape now includes hadoop, redshift, kafka, spark, airflow, notebooks, TiDB and lord knows what else, DE's need to have most of the skills of a software engineer to be successful.