Dramatiq: Distributed Task Queue For Python 3 with Bogdan Popa

Summary

A majority of projects will eventually need some way of managing periodic or long-running tasks outside of the context of the main application. This is where a distributed task queue becomes useful. For many in the Python community the standard option is Celery, though there are other projects to choose from. This week Bogdan Popa explains why he was dissatisfied with the current landscape of task queues and the features that he decided to focus on while building Dramatiq, a new, opinionated distributed task queue for Python 3. He also describes how it is designed, how you can start using it, and what he has planned for the future.

Preface

• Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great.
• I would like to thank everyone who supports us on Patreon. Your contributions help to make the show sustainable.
• When you’re ready to launch your next project you’ll need somewhere to deploy it. Check out Linode at podastinit.com/linode and get a $20 credit to try out their fast and reliable Linux virtual servers for running your awesome app. And now you can deliver your work to your users even faster with the newly upgraded 200 GBit network in all of their datacenters.
• If you’re tired of cobbling together your deployment pipeline then it’s time to try out GoCD, the open source continuous delivery platform built by the people at ThoughtWorks who wrote the book about it. With GoCD you get complete visibility into the life-cycle of your software from one location. To download it now go to podcatinit.com/gocd. Professional support and enterprise plugins are available for added piece of mind.
• Visit the site to subscribe to the show, sign up for the newsletter, and read the show notes. And if you have any questions, comments, or suggestions I would love to hear them. You can reach me on Twitter at @Podcast__init__ or email [email protected])
• To help other people find the show please leave a review on iTunes, or Google Play Music, tell your friends and co-workers, and share it on social media.
• Your host as usual is Tobias Macey and today I’m interviewing Bogdan Popa about Dramatiq, a distributed task processing library for Python with a focus on simplicity, reliability and performance

Interview

• Introductions
• How did you get introduced to Python?
• What is Dramatiq and what was your motivation for creating it?
• How does Dramatiq compare to other task queues in Python such as Celery or RQ?
• How is Dramatiq implemented and how has the internal architecture evolved?
• What have been some of the most difficult aspects of building Dramatiq?
• What are some of the features that you are most proud of?
• For someone who is interested in integrating Dramatiq into an application, can you describe the steps involved and the API?
• Do you provide any form of migration path or compatibility layer for people who are currently using Celery or RQ?
• Can you describe the licensing structure for the project and your reasoning?
  • How did you determine the price point for commercial licenses?
  • Have you been successful in selling licenses for commercial use?



• What are some of the features that you have planned for future releases?

Keep In Touch

• Project Website
• Personal Website
• Bogdanp on GitHub
• @Bogdanp on Twitter

Picks

• Tobias
  • The Anybodies by N.E. Bode



• Bogdan
  • Pipenv

Links

• Dramatiq
• LeadPages
• Lisp
• Celery
• RQ
• Billiard
• Kombu
• Google App Engine
• GAE Task Queue
• RabbitMQ
• APScheduler
• Redis
• Memcached
• LRU (Least Recently Used)
• Middleware
• Gevent
• Pika
• SQS (Amazon Simple Queue Service)
• Google Cloud PubSub
• Django
• API*
• Bundler
• Cargo

The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA