The Art of Database Selection and Evolution

Summary
In this episode of the Data Engineering Podcast Sam Kleinman talks about the pivotal role of databases in software engineering. Sam shares his journey into the world of data and discusses the complexities of database selection, highlighting the trade-offs between different database architectures and how these choices affect system design, query performance, and the need for ETL processes. He emphasizes the importance of understanding specific requirements to choose the right database engine and warns against over-engineering solutions that can lead to increased complexity. Sam also touches on the tendency of engineers to move logic to the application layer due to skepticism about database longevity and advises teams to leverage database capabilities instead. Finally, he identifies a significant gap in data management tooling: the lack of easy-to-use testing tools for database interactions, highlighting the need for better testing paradigms to ensure reliability and reduce bugs in data-driven applications.


Announcements

• Hello and welcome to the Data Engineering Podcast, the show about modern data management
• It’s 2024, why are we still doing data migrations by hand? Teams spend months—sometimes years—manually converting queries and validating data, burning resources and crushing morale. Datafold's AI-powered Migration Agent brings migrations into the modern era. Their unique combination of AI code translation and automated data validation has helped companies complete migrations up to 10 times faster than manual approaches. And they're so confident in their solution, they'll actually guarantee your timeline in writing. Ready to turn your year-long migration into weeks? Visit dataengineeringpodcast.com/datafold today to learn how Datafold can automate your migration and ensure source to target parity. 
• Your host is Tobias Macey and today I'm interviewing Sam Kleinman about database tradeoffs across operating environments and axes of scale

Interview

• Introduction
• How did you get involved in the area of data management?
• The database engine you use has a substantial impact on how you architect your overall system. When starting a greenfield project, what do you see as the most important factor to consider when selecting a database?
• points of friction introduced by database capabilities
• embedded databases (e.g. SQLite, DuckDB, LanceDB), when to use and when do they become a bottleneck
• single-node database engines (e.g. Postgres, MySQL), when are they legitimately a problem
• distributed databases (e.g. CockroachDB, PlanetScale, MongoDB)
• polyglot storage vs. general-purpose/multimodal databases
• federated queries, benefits and limitations 
  • ease of integration vs. variability of performance and access control

Contact Info

• LinkedIn
• GitHub

Parting Question

• From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements

• Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
• Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
• If you've learned something or tried out a project from the show then tell us about it! Email [email protected] with your story.

Links

• MongoDB
• Neon
  • Podcast Episode
• GlareDB
• NoSQL
• S3 Conditional Write
• Event driven architecture
• CockroachDB
• Couchbase
• Cassandra

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA