How Shopify Built A Machine Learning Platform That Encourages Experimentation

Summary
Shopify uses machine learning to power multiple features in their platform. In order to reduce the amount of effort required to develop and deploy models they have invested in building an opinionated platform for their engineers. They have gone through multiple iterations of the platform and their most recent version is called Merlin. In this episode Isaac Vidas shares the use cases that they are optimizing for, how it integrates into the rest of their data platform, and how they have designed it to let machine learning engineers experiment freely and safely.
Announcements

• Hello and welcome to the Machine Learning Podcast, the podcast about machine learning and how to bring it from idea to delivery.
• Your host is Tobias Macey and today I'm interviewing Isaac Vidas about his work on the ML platform used by Shopify

Interview

• Introduction
• How did you get involved in machine learning?
• Can you describe what Shopify is and some of the ways that you are using ML at Shopify? 
  • What are the challenges that you have encountered as an organization in applying ML to your business needs?
• Can you describe how you have designed your current technical platform for supporting ML workloads? 
  • Who are the target personas for this platform?
  • What does the workflow look like for a given data scientist/ML engineer/etc.?
• What are the capabilities that you are trying to optimize for in your current platform? 
  • What are some of the previous iterations of ML infrastructure and process that you have built?
  • What are the most useful lessons that you gathered from those previous experiences that informed your current approach?
• How have the capabilities of the Merlin platform influenced the ways that ML is viewed and applied across Shopify?
• What are the most interesting, innovative, or unexpected ways that you have seen Merlin used?
• What are the most interesting, unexpected, or challenging lessons that you have learned while working on Merlin?
• When is Merlin the wrong choice?
• What do you have planned for the future of Merlin?

Contact Info

• @kazuaros on Twitter
• LinkedIn
• kazuar on GitHub

Parting Question

• From your perspective, what is the biggest barrier to adoption of machine learning today?

Closing Announcements

• Thank you for listening! Don't forget to check out our other shows. The Data Engineering Podcast covers the latest on modern data management. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used.
• Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
• If you've learned something or tried out a project from the show then tell us about it! Email [email protected]) with your story.
• To help other people find the show please leave a review on iTunes and tell your friends and co-workers

Links

• Shopify
• Shopify Merlin
• Vertex AI
• scikit-learn
• XGBoost
• Ray
  • Podcast.__init__ Episode
• PySpark
• GPT-3
• ChatGPT
• Google AI
• PyTorch
  • Podcast.__init__ Episode
• Dask
• Modin
  • Podcast.__init__ Episode
• Flink
  • Data Engineering Podcast Episode
• Feast Feature Store
• Kubernetes

The intro and outro music is from Hitman's Lovesong feat. Paola Graziano by The Freak Fandango Orchestra/CC BY-SA 3.0