Ep 2: Databricks CTO Matei Zaharia on scaling and orchestrating large language models

Patrick and Jacob sit down with Matei Zaharia, Co-Founder and CTO at Databricks and Professor at Stanford. They discuss how companies are training and serving models in production with Databricks, where LLMs fall short for search and how to improve them, the state of the art AI research at Stanford, and how the size and cost of models is likely to change with technological advances in the coming years.

 

(0:00) - Introduction

(2:04) - Founding story of Databricks

(6:03) - PhD classmates using early version of spark for Netflix competition

(6:55) - Building applications with MLFlow

(9:55) - LLMs and ChatGPT

(12:05) - Working with and fine-tuning foundation models

(13:00) - Prompt engineering here to stay or temporary?

(15:12) - Matei’s research at Stanford. The Demonstrate-Search-Predict framework (DSP)

(17:42) - How LLMs will be combined with classic information retrieval systems for world-class search

(19:38) - LLMs writing programs to orchestrate LLMs

(20:36) - Using LLMs in Databricks cloud product

(24:21) - Scaling LLM training and serving

(27:29) - How much will cost to train LLMs go down in coming years?

(29:22) - How many parameters is too many?

(31:14) - Open source vs closed source?

(35:19) - Stanford AI research - Snorkel, ColBERT, and More

(38:58) - Matei getting a $50 amazon gift card for weeks of work

(43:23) - Quick-fire round

 

With your co-hosts:

@jasoncwarner

- Former CTO GitHub, VP Eng Heroku & Canonical

 

@ericabrescia

- Former COO Github, Founder Bitnami (acq’d by VMWare)

 

@patrickachase

- Partner at Redpoint, Former ML Engineer LinkedIn

 

@jacobeffron

- Partner at Redpoint, Former PM Flatiron Health

Jacob is an AI investor at Redpoint Ventures. He's led Redpoint's investments in companies like Abridge, Physical Intelligence & Legora. Follow Jacob on Twitter (@jacobeffron).

On Unsupervised Learning we probe the sharpest minds in AI in search for the truth about what's real today, what will be real in the future and what it all means for businesses and the world. If you're a builder, researcher or investor navigating the AI world, this podcast will help you deconstruct and understand the most important breakthroughs and see a clearer picture of reality. Subscribe to this show to stay up to date on our latest episodes.