From the Data Exchange podcast, we present recent conversations with creators of popular open source data and machine learning orchestration frameworks. Modernizing an organization’s data infrastructure is increasingly difficult without an orchestrator. At a high-level, these are tools that enable developers to write, schedule, monitor, and manage pipelines. In the early stages of gathering and unlocking data assets, bespoke orchestrators may be sufficient. But as teams build and deploy more data and AI products and services, dataflow automation tools become essential.
Jeremiah Lowin is co-founder and CEO of Prefect, the company behind the popular open source data and workflow orchestration system of the same name. We discussed the major design changes in Prefect 2.0.
Haytham Abuelfutuh is co-founder and CTO of Union, a startup founded by the team behind Flyte, a workflow automation platform originated by Lyft.
[Image: Connected stairways by Ben Lorica; original photos from Unsplash, sourced on Infogram.]