Simplified Data Extraction And Analysis For Current Events With Newspaper

Summary

News media is an important source of information for understanding the context of the world. To make it easier to access and process the contents of news sites Lucas Ou-Yang built the Newspaper library that aids in automatic retrieval of articles and prepare it for analysis. In this episode he shares how the project got started, how it is implemented, and how you can get started with it today. He also discusses how recent improvements in the utility and ease of use of deep learning libraries open new possibilities for future iterations of the project.

Announcements

• Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great.
• When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With the launch of their managed Kubernetes platform it’s easy to get started with the next generation of deployment and scaling, powered by the battle tested Linode platform, including simple pricing, node balancers, 40Gbit networking, dedicated CPU and GPU instances, and worldwide data centers. Go to pythonpodcast.com/linode and get a $60 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
• This portion of Python Podcast is brought to you by Datadog. Do you have an app in production that is slower than you like? Is its performance all over the place (sometimes fast, sometimes slow)? Do you know why? With Datadog, you will. You can troubleshoot your app’s performance with Datadog’s end-to-end tracing and in one click correlate those Python traces with related logs and metrics. Use their detailed flame graphs to identify bottlenecks and latency in that app of yours. Start tracking the performance of your apps with a free trial at pythonpodcast.com/datadog. If you sign up for a trial and install the agent, Datadog will send you a free t-shirt.
• You listen to this show to learn and stay up to date with the ways that Python is being used, including the latest in machine learning and data analysis. For more opportunities to stay up to date, gain new skills, and learn from your peers there are a growing number of virtual events that you can attend from the comfort and safety of your home. Go to pythonpodcast.com/conferences to check out the upcoming events being offered by our partners and get registered today!
• Your host as usual is Tobias Macey and today I’m interviewing Lucas Ou-Yang about Newspaper, a framework for easily extracting and processing online articles.

Interview

• Introductions
• How did you get introduced to Python?
• Can you start by describing what the Newspaper project is and your motivations for creating it?
• What are the main use cases that Newspaper is built for?
  • What are some libraries or tools that Newspaper might replace?
• What are the common structures in news sites that allow you to abstract across them for content extraction?
  • What are some ways of determining whether a site will be a good candidate for using with Newspaper?
• Can you talk through the developer workflow of someone using Newspaper?
  • What are some of the other libraries or tools that are commonly used alongside Newspaper?
• How is Newspaper implemented?
  • How has the design of he project evolved since you first began working on it?
  • What are some of the most complex or challenging aspects of building an automated article extraction tool?
• What are some of the most interesting, unexpected, or innovative projects that you have seen built with Newspaper?
• What keeps you interested in the ongoing support and maintenance of the project?
• What do you have planned for the future of Newspaper?

Keep In Touch

• LinkedIn
• @LucasOuYang on Twitter
• Website
• codelucas on GitHub

Picks

• Tobias
  • Million Bazillion Podcast
• Lucas
  • Hackers and Painters: Big Ideas from the Computer Age by Paul Graham

Closing Announcements

• Thank you for listening! Don’t forget to check out our other show, the Data Engineering Podcast for the latest on modern data management.
• Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
• If you’ve learned something or tried out a project from the show then tell us about it! Email [email protected]) with your story.
• To help other people find the show please leave a review on iTunes and tell your friends and co-workers
• Join the community in the new Zulip chat workspace at pythonpodcast.com/chat

Links

• Newspaper
• Los Angeles
• Reddit
• Django
• NLP == Natural Language Processing
• Web Scraping
  • Podcast Episode
• Requests
• Wintria
• Python Goose
• Diffbot
• Heuristics
• Stop Words
• RSS
• SpaCy
  • Podcast Episode
• Gensim
  • Podcast Episode
• PyTorch
  • Podcast Episode
• NLTK
• LXML
• Beautiful Soup

The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA