> The declarative concept is highly tied to the trend of moving away from data pipelines and embracing data products –
Of course an Airbyte article would say this, because they are selling these tools, but my experience has been the opposite. People buy these tools because they claim to make it easier for non-software people to build pipelines. But the problem is that these tools seem to end up being far more complicated and less reliable than pipelines built in code.
There's a reason that this domain is saturated with so. many. tools. None of them do a great job. And when a company invariably hits the limits of one, they start shopping for a replacement, which will have it's own set of limitations. Lather-rinse-repeat.
I built a solid career over the past 8 or so years of replacing these "no code" pipeline tools with code once companies hit the ceilings of these tools. You can get surprisingly far in the data world with Airflow + a large scale database, but all of the major cloud providers have great tool offerings in this space. Plus, for platforms that these tools don't interface with, you're going to have to write code anyway.
Oh, declarative doesn't necessarily mean no-code. Airbyte data integration connectors are built with an SDK in Python, Java, and a low-code SDK that was just released...
You can then build custom connectors on top of these and many users actually need to modify an existing connector, but would rather start from a template than from scratch.
> I built a solid career over the past 8 or so years of replacing these "no code" pipeline tools with code once companies hit the ceilings of these tools.
I'm sure you earn a nice living doing this, but surely this is not a convincing argument against using off-the-shelf data products. It will always come down to the cost (including ongoing maintenance) for the business. Bespoke in-house software is always the most flexible route, but rarely the cheaper one.
Of course an Airbyte article would say this, because they are selling these tools, but my experience has been the opposite. People buy these tools because they claim to make it easier for non-software people to build pipelines. But the problem is that these tools seem to end up being far more complicated and less reliable than pipelines built in code.
There's a reason that this domain is saturated with so. many. tools. None of them do a great job. And when a company invariably hits the limits of one, they start shopping for a replacement, which will have it's own set of limitations. Lather-rinse-repeat.
I built a solid career over the past 8 or so years of replacing these "no code" pipeline tools with code once companies hit the ceilings of these tools. You can get surprisingly far in the data world with Airflow + a large scale database, but all of the major cloud providers have great tool offerings in this space. Plus, for platforms that these tools don't interface with, you're going to have to write code anyway.