Dependency Management Improvements In Pip's Resolver

Summary

Dependency management in Python has taken a long and winding path, which has led to the current dominance of Pip. One of the remaining shortcomings is the lack of a robust mechanism for resolving the package and version constraints that are necessary to produce a working system. Thankfully, the Python Software Foundation has funded an effort to upgrade the dependency resolution algorithm and user experience of Pip. In this episode the engineers working on these improvements, Pradyun Gedam, Tzu-Ping Chung, and Paul Moore, discuss the history of Pip, the challenges of dependency management in Python, and the benefits that surrounding projects will gain from a more robust resolution algorithm. This is an exciting development for the Python ecosystem, so listen now and then provide feedback on how the new resolver is working for you.

Announcements

• Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great.
• When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With 200 Gbit/s private networking, node balancers, a 40 Gbit/s public network, fast object storage, and a brand new managed Kubernetes platform, all controlled by a convenient API you’ve got everything you need to scale up. And for your tasks that need fast computation, such as training machine learning models, they’ve got dedicated CPU and GPU instances. Go to pythonpodcast.com/linode to get a $20 credit and launch a new server in under a minute. And don’t forget to thank them for their continued support of this show!
• You listen to this show because you love Python and want to keep your skills up to date, and machine learning is finding its way into every aspect of software engineering. Springboard has partnered with us to help you take the next step in your career by offering a scholarship to their Machine Learning Engineering career track program. In this online, project-based course every student is paired with a Machine Learning expert who provides unlimited 1:1 mentorship support throughout the program via video conferences. You’ll build up your portfolio of machine learning projects and gain hands-on experience in writing machine learning algorithms, deploying models into production, and managing the lifecycle of a deep learning prototype. Springboard offers a job guarantee, meaning that you don’t have to pay for the program until you get a job in the space. Podcast.__init__ is exclusively offering listeners 20 scholarships of $500 to eligible applicants. It only takes 10 minutes and there’s no obligation. Go to pythonpodcast.com/springboard and apply today! Make sure to use the code AISPRINGBOARD when you enroll.
• Your host as usual is Tobias Macey and today I’m interviewing Tzu-ping Chung, Pradyun Gedam, and Paul Moore about their work to improve the dependency resolution capabilities of Pip and its user experience

Interview

• Introductions
• How did you get introduced to Python?
• Can you start by describing the focus of the work that you are doing?
  • What is the scope of the work, and what is the established criteria for when it is considered complete?
• What is your history with working on the Pip source code and what interests you most about this project?
• What are the main sources or manifestations of technical debt that exist in Pip as of today?
  • How does it currently handle dependency resolution?
• What are some of the workarounds that developers have had to resort to in the absence of a robust dependency resolver in Pip?
• How is the new dependency resolver implemented?
  • How has your initial design evolved or shifted as you have gotten further along in its implementation?
• What are the pieces of information that the resolver will rely on for determining which packages and versions to install? (e.g. will it install setuptools > 45.x in a Python 2 virtualenv?)
• What are the new capabilities in Pip that will be enabled by this upgrade to the dependency resolver?
• What projects or features in the encompassing ecosystem will be unblocked with the introduction of this upgrade?
• What are some of the changes that users will need to make to adopt the updated Pip?
• How do you anticipate the changes in Pip impacting the viability or adoption of Python and its ecosystem within different communities or industries?
• What are some of the additional changes or improvements that you would like to see in Pip or other core elements of the Python landscape?
• What are some of the most interesting, unexpected, or challenging lessons that you have learned while working on these updates to Pip?

Keep In Touch

• Pradyun
  • Website
  • pradyunsg on GitHub
  • @pradyunsg on Twitter
• Paul
  • pfmoore on GitHub
• Tzu-Ping
  • uranusjr on GitHub
  • Website
  • @uranusjr on Twitter

Picks

• Tzu-ping
  • Python Launcher
  • Joe Abercrombie author
  • The Shattered Sea Trilogy
  • Anime
  • PipX Standalone
• Paul
  • pipx
  • Black
  • nox
  • tox
  • scoop
  • Neil Gaiman
  • Good Omens
    • Book
    • TV Series
• Pradyun
  • because my picks can be anything — things that have kept me sane in this lockdown world
    • Music: Chris Daughtry
    • Video Game: Parkitect
• Tobias
  • Language Server Protocol
  • Emacs lsp-mode

Closing Announcements

• Thank you for listening! Don’t forget to check out our other show, the Data Engineering Podcast for the latest on modern data management.
• Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
• If you’ve learned something or tried out a project from the show then tell us about it! Email [email protected]) with your story.
• To help other people find the show please leave a review on iTunes and tell your friends and co-workers
• Join the community in the new Zulip chat workspace at pythonpodcast.com/chat

Links

• Pip
  • Podcast interview with Donald Stufft
• Macdown
• Taiwan
• Pipenv
• PyPI
  • Podcast Episode
• TOML
• Python Package Metadata Standards
• iBook G4
• Acorn Computer
• distutils
• easy_install
• Python Eggs
• setuptools
• Python Wheels
• CPAN
• Conda
• Inside The Cheeseshop
• Google Summer of Code
• Zazo
• PEP517
• pip-tools
• Poetry
• resolvelib
• SAT Solver
• Trove Classifiers
• PyPA
• pyproject.toml

The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA