The Journey To Replace Python's Parser And What It Means For The Future

Summary

The release of Python 3.9 introduced a new parser that paves the way for brand new features. Every programming language has its own specific syntax for representing the logic that you are trying to express. The way that the rules of the language are defined and validated is with a grammar definition, which in turn is processed by a parser. The parser that the Python language has relied on for the past 25 years has begun to show its age through mounting technical debt and a lack of flexibility in defining new syntax. In this episode Pablo Galindo and Lysandros Nikolaou explain how, together with Python’s creator Guido van Rossum, they replaced the original parser implementation with one that is more flexible and maintainable, why now was the time to make the change, and how it will influence the future evolution of the language.

Announcements

• Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great.
• When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With the launch of their managed Kubernetes platform it’s easy to get started with the next generation of deployment and scaling, powered by the battle tested Linode platform, including simple pricing, node balancers, 40Gbit networking, dedicated CPU and GPU instances, and worldwide data centers. Go to pythonpodcast.com/linode and get a $60 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
• You listen to this show to learn and stay up to date with the ways that Python is being used, including the latest in machine learning and data analysis. For more opportunities to stay up to date, gain new skills, and learn from your peers there are a growing number of virtual events that you can attend from the comfort and safety of your home. Go to pythonpodcast.com/conferences to check out the upcoming events being offered by our partners and get registered today!
• Your host as usual is Tobias Macey and today I’m interviewing Pablo Galindo and Lysandros Nikolaou about their work on replacing the parser in CPython and what that means for the language

Interview

• Introductions
• How did you get introduced to Python?
• Can you start by discussing the role of the parser in the lifecycle of a Python program?
• What were the limitations of the previous parser, and how did that contribute to complexity and technical debt in the CPython runtime?
• What are the options for styles of parsers, and what are the benefits of using a PEG style grammar?
• How does the new parser impact the approachability of the CPython code for new contributors?
• What was the process for reimplementing the parser and guarding against regressions in the syntax?
• As developers switch to the 3.9 release, what potential edge cases/bugs might they see from introducing the new parser?
• What new syntax options does this parser provide for the Python language?
  • Are there any specific features that are planned for implementation in the 3.10 release that are enabled by the new parser grammar?
• As the language evolves due to new capabilities offered by the updated parser, how will that impact other implementations such as PyPy?
• What were the most interesting, unexpected, or challenging aspects of this project?
• What other aspects of the CPython code do you think should be reconsidered or reimplemented in light of the changes in computing and the usage of the language?

Keep In Touch

• Pablo
  • pablogsal on GitHub
  • @pyblogsal on Twitter
  • LinkedIn
• Lysandros
  • LinkedIn
  • lysnikolaou on GitHub
  • @lysnikolaou on Twitter

Picks

• Tobias
  • Annual Python Developer Survey
  • Jessica Jones TV show
• Pablo
  • Raised By Wolves TV Series
• Lysandros
  • Afterlife TV show

Links

• PEP 617 – New PEG Parser for CPython
• Podcast Episode About Parsers
• CPython
• Bloomberg
• PEG Parsers
• Seafair
• LL(1) Parsers
• Łukasz Langa
• Parser Generator
• Concrete Syntax Tree
• Abstract Syntax Tree
• PyPy
• RustPython
  • Podcast Episode
• IronPython
• Structural Pattern Matching – PEP 622
• Pylint
• ASTroid
  • Podcast Episode
• Hy
  • Podcast Episode
• Walrus Operator/Assignment Expressions
• C99
• Reference Counting
• Cycle Hunting/Generational Garbage Collection

The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA