Getting A Handle On Portable C Extensions With hpy

Summary

One of the driving factors of Python’s success is the ability for developers to integrate with performant languages such as C and C++. The challenge is that the interface for those extensions is specific to the main implementation of the language. This contributes to difficulties in building alternative runtimes that can support important packages such as NumPy. To address this situation a team of developers are working to create the hpy project, a new interface for extension developers that is standardized and provides a uniform target for multiple runtimes. In this episode Antonio Cuni discusses the motivations for creating hpy, how it benefits the whole ecosystem, and ways to contribute to the effort. This is an exciting development that has the potential to unlock a new wave of innovation in the ways that you can run your Python code.

Announcements

• Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great.
• When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With 200 Gbit/s private networking, node balancers, a 40 Gbit/s public network, fast object storage, and a brand new managed Kubernetes platform, all controlled by a convenient API you’ve got everything you need to scale up. And for your tasks that need fast computation, such as training machine learning models, they’ve got dedicated CPU and GPU instances. Go to pythonpodcast.com/linode to get a $20 credit and launch a new server in under a minute. And don’t forget to thank them for their continued support of this show!
• As a developer, maintaining a state of flow is key to your productivity. Don’t let something as simple as the wrong function ruin your day. Kite is the smartest completions engine available for Python, featuring a machine learning model trained by the brightest stars of GitHub. Featuring ranked suggestions sorted by relevance, offering up to full lines of code, and a programming copilot that offers up the documentation you need right when you need it. Get Kite for free today at getkite.com with integrations for top editors, including Atom, VS Code, PyCharm, Spyder, Vim, and Sublime.
• You listen to this show to learn and stay up to date with the ways that Python is being used, including the latest in machine learning and data analysis. For even more opportunities to meet, listen, and learn from your peers you don’t want to miss out on this year’s conference season. We have partnered with organizations such as O’Reilly Media, Corinium Global Intelligence, ODSC, and Data Council. Upcoming events include the Software Architecture Conference in NYC, Strata Data in San Jose, and PyCon US in Pittsburgh. Go to pythonpodcast.com/conferences to learn more about these and other events, and take advantage of our partner discounts to save money when you register today.
• Your host as usual is Tobias Macey and today I’m interviewing Antonio Cuni about hpy, a project aiming to reimagine the C API for Python

Interview

• Introductions
• How did you get introduced to Python?
• Can you start by describing what the hpy project is and how it got started?
  • What are the goals for the project?
  • Who else is involved?
• How much engagement have you had with CPython core contributors or the steering council?
• Who are the consumers of the current C API for the CPython implementation?
  • What are some of the pain points or shortcomings for those consumers?
  • What impact does that have for users of a given library that leverages C extensions?
• Can you talk through the structure of the hpy project?
  • What are some of the design challenges that you are facing for determining the external API?
  • What is involved in integrating the hpy interface into alternate runtimes such as PyPy or RustPython?
• What is the potential or observed performance impact for libraries that currently rely on the existing C API?
• How has the vision and scope of this project been updated as you have gotten further along in the implementation?
• What are the downstream impacts that you anticipate in projects such as PyPy and Cython?
• What have you found to be the most challenging or contentious aspects of implementing hpy so far?
• What are some of the most interesting/unexpected/useful lessons that you have learned while working on hpy?
• What do you have planned for the near to medium term for hpy?

Keep In Touch

• antocuni on GitHub
• Website
• @antocuni on Twitter

Picks

• Tobias
  • Poetry
• Antonio
  • Collapse: How Societies Choose To Fail Or Succeed by Jared Diamond

Links

• hpy
• PyPy
• Alex Martelli
  • Podcast Interview
• Python C Extensions
• EuroPython
• Victor Stinner
• Cython
  • Podcast Episode
• Armin Rigo
• NumPy
• ultrajson
• GIL == Global Interpreter Lock
• RustPython
  • Podcast Episode
• GraalPython
• hpy-rust

The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA