Put Your Whole Data Team On The Same Page With Atlan

Summary

One of the biggest obstacles to success in delivering data products is cross-team collaboration. Part of the problem is the difference in the information that each role requires to do their job and where they expect to find it. This introduces a barrier to communication that is difficult to overcome, particularly in teams that have not reached a significant level of maturity in their data journey. In this episode Prukalpa Sankar shares her experiences across multiple attempts at building a system that brings everyone onto the same page, ultimately bringing her to found Atlan. She explains how the design of the platform is informed by the needs of managing data projects for large and small teams across her previous roles, how it integrates with your existing systems, and how it can work to bring everyone onto the same page.

Announcements

• Hello and welcome to the Data Engineering Podcast, the show about modern data management
• When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
• Modern Data teams are dealing with a lot of complexity in their data pipelines and analytical code. Monitoring data quality, tracing incidents, and testing changes can be daunting and often takes hours to days. Datafold helps Data teams gain visibility and confidence in the quality of their analytical data through data profiling, column-level lineage and intelligent anomaly detection. Datafold also helps automate regression testing of ETL code with its Data Diff feature that instantly shows how a change in ETL or BI code affects the produced data, both on a statistical level and down to individual rows and values. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows. Go to dataengineeringpodcast.com/datafold today to start a 30-day trial of Datafold. Once you sign up and create an alert in Datafold for your company data, they will send you a cool water flask.
• RudderStack’s smart customer data pipeline is warehouse-first. It builds your customer data warehouse and your identity graph on your data warehouse, with support for Snowflake, Google BigQuery, Amazon Redshift, and more. Their SDKs and plugins make event streaming easy, and their integrations with cloud applications like Salesforce and ZenDesk help you go beyond event streaming. With RudderStack you can use all of your customer data to answer more difficult questions and then send those insights to your whole customer data stack. Sign up free at dataengineeringpodcast.com/rudder today.
• Your host is Tobias Macey and today I’m interviewing Prukalpa Sankar about Atlan, a modern data workspace that makes collaboration among data stakeholders easier, increasing efficiency and agility in data projects

Interview

• Introduction
• How did you get involved in the area of data management?
• Can you start by giving an overview of what you are building at Atlan and some of the story behind it?
• Who are the target users of Atlan?
• What portions of the data workflow is Atlan responsible for?
  • What components of the data stack might Atlan replace?
• How would you characterize Atlan’s position in the current data ecosystem?
  • What makes Atlan stand out from other systems for data cataloguing, metadata management, or data governance?
  • What types of data assets (e.g. structured vs unstructured, textual vs binary, etc.) is Atlan designed to understand?
• Can you talk through how Atlan is implemented?
  • How have the goals and design of the platform changed or evolved since you first began working on it?
  • What are some of the early assumptions that you have had to revisit or reconsider?
• What is involved in getting Atlan deployed and integrated into an existing data platform?
• Beyond the technical aspects, what are the business processes that teams need to implement to be successful when incorporating Atlan into their systems?
• Once Atlan is set up, what is a typical workflow for an individual and their team to collaborate on a set of data assets, or building out a new processing pipeline?
  • What are some useful steps for introducing all of the stakeholders to the system and workflow?
• What are the available extension points for managing data in systems that aren’t supported by Atlan out of the box?
• What are some of the most interesting, innovative, or unexpected ways that you have seen Atlan used?
• What are the most interesting, unexpected, or challenging lessons that you have learned while building Atlan?
• When is Atlan the wrong choice?
• What do you have planned for the future of the product?

Contact Info

• LinkedIn
• @prukalpa on Twitter

Parting Question

• From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements

• Thank you for listening! Don’t forget to check out our other show, Podcast.__init__ to learn about the Python language, its community, and the innovative ways it is being used.
• Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
• If you’ve learned something or tried out a project from the show then tell us about it! Email [email protected]) with your story.
• To help other people find the show please leave a review on iTunes and tell your friends and co-workers
• Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat

Links

• Atlan
• India’s National Data Platform
• World Economic Forum
• UN
• Gates Foundation
• GitHub
• Figma
• Snowflake
• Redshift
• Databricks
• DBT
• Sisense
• Looker
• Apache Atlas
• Immuta
• DataHub
• Datakin
• Aapache Ranger
• Great Expectations
• Trino
• Airflow
• Dagster
• Privacera
• Databand
• Cloudformation
• Grafana
• Deequ
• We Failed to Set Up a Data Catalog 3x. Here’s Why.
• Analysing the analysers book
• OpenAPI

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA