I never understood why we limit ourselves to sending SQL to databases. Instead of sending SQL we should be sending actual logical plans to the database. These logical plans could then be optimized by the database and executed. This would solve two issues I have with SQL:
1. Without CTEs SQL isn't that expressive and with CTEs queries can be hard to understand. Logical query plans on the other hand are rather easy to understand and can express more queries than SQL can.
2. SQL is hard to modify programmatically. To most software, SQL is simply a string without any inherent meaning. A logical query plan on the hand is a tree structure that can be introspected and modified.
That's basically the same thing you want, except that you want it implemented on dataframes while I want it implemented for database systems.
1. Without CTEs SQL isn't that expressive and with CTEs queries can be hard to understand. Logical query plans on the other hand are rather easy to understand and can express more queries than SQL can.
2. SQL is hard to modify programmatically. To most software, SQL is simply a string without any inherent meaning. A logical query plan on the hand is a tree structure that can be introspected and modified.
That's basically the same thing you want, except that you want it implemented on dataframes while I want it implemented for database systems.