Building the Foundation Model Ops Platform — with Raza Habib of Humanloop

Want to help define the AI Engineer stack? >500 folks have weighed in on the top tools, communities and builders for the first State of AI Engineering survey! Please fill it out (and help us reach 1000!)

The AI Engineer Summit schedule is now live! We are running two Summits and judging two Hackathons this Oct. As usual, see our Discord and community page for all events.

A rite of passage for every AI Engineer is shipping a quick and easy demo, and then having to cobble together a bunch of solutions for prompt sharing and versioning, running prompt evals and monitoring, storing data and finetuning as their AI apps go from playground to production. This happens to be Humanloop’s exact pitch.

full show notes: https://latent.space/p/humanloop

Timestamps

* [00:01:21] Introducing Raza

* [00:10:52] Humanloop Origins

* [00:19:25] What is HumanLoop?

* [00:20:57] Who is the Buyer of PromptOps?

* [00:22:21] HumanLoop Features

* [00:22:49] The Three Stages of Prompt Evals

* [00:24:34] The Three Types of Human Feedback

* [00:27:21] UI vs BI for AI

* [00:28:26] LangSmith vs HumanLoop comparisons

* [00:31:46] The TAM of PromptOps

* [00:32:58] How to Be Early

* [00:34:41] 6 Orders of Magnitude

* [00:36:09] Becoming an Enterprise Ready AI Infra Startup

* [00:40:41] Killer Usecases of AI

* [00:43:56] HumanLoop's new Free Tier and Pricing

* [00:45:20] Addressing Graduation Risk

* [00:48:11] On Company Building

* [00:49:58] On Opinionatedness

* [00:51:09] HumanLoop Hiring

* [00:52:42] How HumanLoop thinks about PMF

* [00:55:16] Market: LMOps vs MLOps

* [00:57:01] Impact of Multimodal Models

* [00:57:58] Prompt Engineering vs AI Engineering

* [01:00:11] LLM Cascades and Probabilistic AI Languages

* [01:02:02] Prompt Injection and Prompt Security

* [01:03:24] Finetuning vs HumanLoop

* [01:04:43] Open Standards in LLM Tooling

* [01:06:05] Did GPT4 Get Dumber?

* [01:07:29] Europe's AI Scene

* [01:09:31] Just move to SF (in The Arena)

* [01:12:23] Lightning Round - Acceleration

* [01:13:48] Continual Learning

* [01:15:02] DeepMind Gato Explanation

* [01:17:40] Motivations from Academia to Startup

* [01:19:52] Lightning Round - The Takeaway

This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe