Applying Machine Learning To The Problem Of Bad Data At Anomalo

Summary
All data systems are subject to the "garbage in, garbage out" problem. For machine learning applications bad data can lead to unreliable models and unpredictable results. Anomalo is a product designed to alert on bad data by applying machine learning models to various storage and processing systems. In this episode Jeremy Stanley discusses the various challenges that are involved in building useful and reliable machine learning models with unreliable data and the interesting problems that they are solving in the process.
Announcements

• Hello and welcome to the Machine Learning Podcast, the podcast about machine learning and how to bring it from idea to delivery.
• Your host is Tobias Macey and today I'm interviewing Jeremy Stanley about his work at Anomalo, applying ML to the problem of data quality monitoring

Interview

• Introduction
• How did you get involved in machine learning?
• Can you describe what Anomalo is and the story behind it?
• What are some of the ML approaches that you are using to address challenges with data quality/observability?
• What are some of the difficulties posed by your application of ML technologies on data sets that you don't control? 
  • How does the scale and quality of data that you are working with influence/constrain the algorithmic approaches that you are using to build and train your models?
• How have you implemented the infrastructure and workflows that you are using to support your ML applications?
• What are some of the ways that you are addressing data quality challenges in your own platform? 
  • What are the opportunities that you have for dogfooding your product?
• What are the most interesting, innovative, or unexpected ways that you have seen Anomalo used?
• What are the most interesting, unexpected, or challenging lessons that you have learned while working on Anomalo?
• When is Anomalo the wrong choice?
• What do you have planned for the future of Anomalo?

Contact Info

• @jeremystan on Twitter
• LinkedIn

Parting Question

• From your perspective, what is the biggest barrier to adoption of machine learning today?

Closing Announcements

• Thank you for listening! Don't forget to check out our other shows. The Data Engineering Podcast covers the latest on modern data management. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used.
• Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
• If you've learned something or tried out a project from the show then tell us about it! Email [email protected]) with your story.
• To help other people find the show please leave a review on iTunes and tell your friends and co-workers

Links

• Anomalo
  • Data Engineering Podcast Episode
• Partial Differential Equations
• Neural Network
• Neural Networks For Pattern Recognition by Christopher M. Bishop (affiliate link)
• Gradient Boosted Decision Trees
• Shapley Values
• Sentry
• dbt
• Altair

The intro and outro music is from Hitman's Lovesong feat. Paola Graziano by The Freak Fandango Orchestra/CC BY-SA 3.0