Fast, Flexible, and Incremental Task Automation With doit

Summary

Every software project needs a tool for managing the repetitive tasks that are involved in building, running, and deploying the code. Frustrated with the limitations of tools like Make, Scons, and others Eduardo Schettino created doit to handle task automation in his own work and released it as open source. In this episode he shares the story behind the project, how it is implemented under the hood, and how you can start using it in your own projects to save you time and effort.

Announcements

• Hello and welcome to Podcast.__init__, the podcast about Python’s role in data and science.
• When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With the launch of their managed Kubernetes platform it’s easy to get started with the next generation of deployment and scaling, powered by the battle tested Linode platform, including simple pricing, node balancers, 40Gbit networking, dedicated CPU and GPU instances, and worldwide data centers. Go to pythonpodcast.com/linode and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
• Your host as usual is Tobias Macey and today I’m interviewing Eduardo Schettino about Doit, a flexible and low overhead task automation tool

Interview

• Introductions
• How did you get introduced to Python?
• Can you describe what doit is and the story behind it?
• What are the main goals and use cases of doit?
• Can you describe how you approached the implementation of Doit?
  • How has the design changed or evolved since you first began working on it?
• The realm of task automation tools for developers is an exceedingly crowded one, with each tool prioritizing certain use cases. How would you characterize the position of doit in the current ecosystem?
  • How does it compare to e.g. Click, Invoke, Typer, etc.?
• What is your guiding philosophy for when and how to add new features?
  • You have been running the project for ~13 years now. How has the evolution of the Python language and ecosystem influenced your approach to the development and maintenance of doit?
• What is the workflow for getting started with doit and integrating it into your development process?
• For every project there are some tasks that are identical and some that are bespoke for that application. What are the options for maintaining a standard set of tasks across repositories and composing them with per-project activites?
• What are some of the useful patterns that you and the community have established for designing tasks and execution graphs?
• How do you use doit in your own work?
• What are the most interesting, innovative, or unexpected ways that you have seen doit used?
• What are the most interesting, unexpected, or challenging lessons that you have learned while working on doit?
• When is doit the wrong choice?
• What do you have planned for the future of doit?

Keep In Touch

• LinkedIn
• schettino72 on GitHub

Picks

• Tobias
  • The Matrix series
• Eduardo
  • John Pilger

Closing Announcements

• Thank you for listening! Don’t forget to check out our other show, the Data Engineering Podcast for the latest on modern data management.
• Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
• If you’ve learned something or tried out a project from the show then tell us about it! Email [email protected]) with your story.
• To help other people find the show please leave a review on iTunes and tell your friends and co-workers

Links

• doit
• Zope
• Twisted
• Django
• Pyflakes
• scons
• Make
• Nikola
  • Podcast Episode
• Nose
• Pytest
  • Podcast Episode
• Click
• Typer
• Invoke
• Puppet
• Ansible
• Chef
• Sphinx
• Snakemake
• Airflow
• Luigi
• pytest-incremental
• import-deps
• dbm
• MetalK8s

The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA