Specialized AI for Data Engineers: Inside Astronomer’s Otto

Summary 
In this episode Yetunde Dada discusses Otto, Astronomer’s AI agent for Airflow, and the broader challenge of making agentic tooling actually useful for data engineers. She explored why generic coding assistants often fall short in data workflows, how Otto adds the missing context around Airflow, Astro, upgrades, and troubleshooting, and why Astronomer focused first on high-leverage use cases such as DAG authoring, investigation of pipeline failures, version migrations, and legacy scheduler modernization. She also discussed the practical realities of introducing agents into engineering teams: model choice, security boundaries, vendor lock-in concerns, validation of generated code, and the need for agents to fit into existing workflows rather than forcing users into new ones. Overall, this conversation offers a detailed look at how specialized AI agents can support data engineers today, and where Astronomer is headed next with a vision for self-healing pipelines that keep humans in control while automating more of the operational burden. 


Announcements 

• Hello and welcome to the Data Engineering Podcast, the show about modern data management
• Your host is Tobias Macey and today I'm interviewing Yetunde Dada about Otto, Astronomer's expert Airflow agent

Interview
 

• Introduction
• How did you get involved in the area of data management?
• Can you describe what Otto is and the story behind it?
• What are the core problems that you are trying to solve with Otto and for whom?
• What was your process for identifying the scope of activities that Otto should be incorporated into?
• Orchestration engines are a rich source of information. What are the aspects of Airflow that lend themselves to extending with this agentic context?
• What are the other supporting systems that are necessary to enable Otto to work effectively, especially in mixed orchestration environments? (e.g. metadata platforms)
• One of the explicit capabilities that you invested in is code review for Airflow DAGs. What are the pain points that you are trying to solve with a specialized review agent?
• Can you describe the architecture of the Otto system and how you're managing the complex task of context curation?
• In a production context accuracy and latency are both critical, and often in tension with each other. How do you monitor and optimize for each of those objectives?
  • What are the options for tuning Otto's behavior to bias more toward one direction or another?
• What are some examples of the type of work that Otto can help automate?
• How is it measurably different from a generic coding agent that has MCP connections to something like an Open Metadata or DataHub for platform and data context, Airflow documentation, etc.?
• There are numerous general purpose and specialized agent systems available. What are some of the ways that Otto can work collaboratively with those other products?
• What are the most interesting, innovative, or unexpected ways that you have seen Otto used?
• What are the most interesting, unexpected, or challenging lessons that you have learned while working on Otto?
• What do you have planned for the future of Otto?

Contact Info
 

• LinkedIn

Parting Question
 

• From your perspective, what is the biggest gap in the tooling or technology for data management today?

Links
 

• Astronomer
• Otto
• Announcement Post
• Astronomer Cosmo dbt automation
• Hadoop
• Spark
• Otto Automatic Pipeline Failure Investigation
• Otto Code Review
• Astro CLI
• Astro IDE
• Airflow MCP
• Kedro
• Quantum Black
• Django
• React
• Airflow Providers
• Pi Framework
• Agent Skills
• AGENTS.md

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA