From the Data Exchange podcast, we present recent conversations with creators of popular open source data and machine learning orchestration frameworks. Modernizing an organization’s data infrastructure is increasingly difficult without an orchestrator. At a high-level, these are tools that enable developers to write, schedule, monitor, and manage pipelines. In the early stages of gathering and unlocking data assets, bespoke orchestrators may be sufficient. But as teams build and deploy
more data and AI products and services, dataflow automation tools become essential.
Dataflow Automation
Jeremiah Lowin is co-founder and CEO of Prefect, the company behind the popular open source data and workflow orchestration system of the same name. We discussed the major design changes in Prefect 2.0.Software-Defined Assets
Nick Schrock is co-creator of GraphQL, and founder and Elementl, the startup behind Dagster, a popular open source, data orchestration platform. We discussed Dagster’s introduction of software-defined assets.Orchestrating Machine Learning Applications
Haytham Abuelfutuh is co-founder and CTO of Union, a startup founded by the team behind Flyte, a workflow automation platform originated by Lyft.Orchestration and Pipelines for Data Scientists
Rick Lamers is co-Founder and CEO at Orchest, the startup behind an an elegant development tool built on top of Argo Workflows, a widely used framework for orchestrating parallel jobs on Kubernetes.Learn More
Download
[Image: Connected stairways by Ben Lorica; original photos from Unsplash, sourced on Infogram.]