technologyDec 2, 202245:43pending
Build More Reliable Machine Learning Systems With The Dagster Orchestration Engine
About this episode
Summary
Building a machine learning model one time can be done in an ad-hoc manner, but if you ever want to update it and serve it in production you need a way of repeating a complex sequence of operations. Dagster is an orchestration engine that understands the data that it is manipulating so that you can move beyond coarse task-based representations of your dependencies. In this episode Sandy Ryza explains how his background in machine learning has informed his work on the Dagster project and the foundational principles that it is built on to allow for collaboration across data engineering and machine learning concerns.
Interview
Parting Question
Building a machine learning model one time can be done in an ad-hoc manner, but if you ever want to update it and serve it in production you need a way of repeating a complex sequence of operations. Dagster is an orchestration engine that understands the data that it is manipulating so that you can move beyond coarse task-based representations of your dependencies. In this episode Sandy Ryza explains how his background in machine learning has informed his work on the Dagster project and the foundational principles that it is built on to allow for collaboration across data engineering and machine learning concerns.
Interview
- Introduction
- How did you get involved in machine learning?
- Can you start by sharing a definition of "orchestration" in the context of machine learning projects?
- What is your assessment of the state of the orchestration ecosystem as it pertains to ML?
- modeling cycles and managing experiment iterations in the execution graph
- how to balance flexibility with repeatability
- What are the most interesting, innovative, or unexpected ways that you have seen orchestration implemented/applied for machine learning?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on orchestration of ML workflows?
- When is Dagster the wrong choice?
- What do you have planned for the future of ML support in Dagster?
Parting Question
- From your perspective, what is the biggest barrier to adoption of machine learning today?
- Thank you for listening! Don't forget to check out our other shows. The Data Engineering Podcast covers the latest on modern data management. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you've learned something or tried out a project from the show then tell us about it! Email [email protected]) with your story.
- To help other people find the show please leave a review on iTunes and tell your friends and co-workers
- Dagster
- Cloudera
- Hadoop
- Apache Spark
- Peter Norvig
- Josh Wills
- REPL == Read Eval Print Loop
- RStudio
- Memoization
- MLFlow
- Kedro
- Metaflow
- Kubeflow
- dbt
- Airbyte
Get every episode summarized
Each time AI Engineering Podcast publishes, we email you a written briefing from the transcript — the topics, who appeared, and any specific claims, with the ad reads skipped.
Email me new episodesFree for 3 shows. No card needed.
Hosts & guests
No transcript yet
This episode has not been transcribed. Request it and it moves to the front of the queue.
More episodes
More from AI Engineering Podcast
Kubernetes, Compliance, and Control: The Operational Backbone of AI Sovereignty
AI Engineering Podcast
Feb 25, 20261:01:16pending
From Blind Spots to Observability: Operationalizing LLM Apps with OpenLit
AI Engineering Podcast
Feb 15, 202650:36pending
Taming Voice Complexity with Dynamic Ensembles at Modulate
AI Engineering Podcast
Feb 8, 202659:25pending
GPU Clouds, Aggregators, and the New Economics of AI Compute
AI Engineering Podcast
Jan 27, 202646:02pending