Overcoming Redis Limitations: The Dragonfly DB Approach

Summary
In this episode of the Data Engineering Podcast Roman Gershman, CTO and founder of Dragonfly DB, explores the development and impact of high-speed in-memory databases. Roman shares his experience creating a more efficient alternative to Redis, focusing on performance gains, scalability, and cost efficiency, while addressing limitations such as high throughput and low latency scenarios. He explains how Dragonfly DB solves operational complexities for users and delves into its technical aspects, including maintaining compatibility with Redis while innovating on memory efficiency. Roman discusses the importance of cost efficiency and operational simplicity in driving adoption and shares insights on the broader ecosystem of in-memory data stores, future directions like SSD tiering and vector search capabilities, and the lessons learned from building a new database engine.


Announcements

• Hello and welcome to the Data Engineering Podcast, the show about modern data management
• Data migrations are brutal. They drag on for months—sometimes years—burning through resources and crushing team morale. Datafold's AI-powered Migration Agent changes all that. Their unique combination of AI code translation and automated data validation has helped companies complete migrations up to 10 times faster than manual approaches. And they're so confident in their solution, they'll actually guarantee your timeline in writing. Ready to turn your year-long migration into weeks? Visit dataengineeringpodcast.com/datafold today for the details.
• Your host is Tobias Macey and today I'm interviewing Roman Gershman about building a high-speed in-memory database and the impact of the performance gains on data applications

Interview

• Introduction
• How did you get involved in the area of data management?
• Can you describe what DragonflyDB is and the story behind it?
• What is the core problem/use case that is solved by making a "faster Redis"?
• The other major player in the high performance key/value database space is Aerospike. What are the heuristics that an engineer should use to determine whether to use that vs. Dragonfly/Redis?
• Common use cases for Redis involve application caches and queueing (e.g. Celery/RQ). What are some of the other applications that you have seen Redis/Dragonfly used for, particularly in data engineering use cases?
• There is a piece of tribal wisdom that it takes 10 years for a database to iron out all of the kinks. At the same time, there have been substantial investments in commoditizing the underlying components of database engines. Can you describe how you approached the implementation of DragonflyDB to arive at a functional and reliable implementation?
• What are the architectural elements that contribute to the performance and scalability benefits of Dragonfly?
  • How have the design and goals of the system changed since you first started working on it?
• For teams who migrate from Redis to Dragonfly, beyond the cost savings what are some of the ways that it changes the ways that they think about their overall system design?
• What are the most interesting, innovative, or unexpected ways that you have seen Dragonfly used?
• What are the most interesting, unexpected, or challenging lessons that you have learned while working on DragonflyDB?
• When is DragonflyDB the wrong choice?
• What do you have planned for the future of DragonflyDB?

Contact Info

• GitHub
• LinkedIn

Parting Question

• From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements

• Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
• Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
• If you've learned something or tried out a project from the show then tell us about it! Email [email protected] with your story.

Links

• DragonflyDB
• Redis
• Elasticache
• ValKey
• Aerospike
• Laravel
• Sidekiq
• Celery
• Seastar Framework
• Shared-Nothing Architecture
• io_uring
• midi-redis
• Dunning-Kruger Effect
• Rust

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA