Transcript#

This transcript was generated automatically and may contain errors.

Hey, Wes, welcome to Talk Python to Me. Thanks for having me. You know, honestly, I feel like it's been a long time coming, having you on the show. You've had such a big impact in the Python space, especially the data science side of that space, and it's high time to have you on the show, so welcome. Good to have you.

Yeah, it's great to be here. I've been heads down a lot the last N years, and I actually haven't been, because I think a lot of my work has been more like data infrastructure and working at even a lower level than Python, so I haven't been engaging as much directly with the Python community. But it's been great to kind of get back more involved and start catching up on all the things that people have been building. And yeah, so being at Posit gives me the ability to sort of have more exposure to what's going on and people that are using Python in the real world.

Yeah, there's a ton of stuff going on at Posit that's super interesting, and we'll talk about some of that. You know, sometimes it's just really fun to build, you know, and work with people building things, and I'm sure you're enjoying that aspect of it. For sure.

Background and the Pandas origin story

Well, before we dive into Pandas and all the things that you've been working on after that, you know, let's just hear a quick bit about yourself for folks who don't know you. Sure. So, yeah, my name is Wes McKinney. I grew up in Akron, Ohio, mostly, and I got involved, started getting involved in Python development around 2007, 2008, and I was working in quant finance at the time. I started building a personal data analysis toolkit that turned into the Pandas project, and then open sourced that in 2009, started getting involved in the Python community, and I spent several years writing my book, Python for Data Analysis, and then working with the broader scientific Python data science community to help enable Python to become a mainstream programming language for doing data analysis and data science.

And in the meantime, I've become an entrepreneur. I've started some companies and have been working to, you know, innovate and improve the computing infrastructure that powers data science tools and libraries like Pandas. So that's led to some other projects like Apache Arrow and Ibis and some other things. And, yeah, and in recent years, I've worked on a startup, Voltron Data, which is still, you know, very much going strong and, you know, has a big team and is off to the races. And I've had a long relationship with Posit, formerly RStudio, and they were, you know, my home for doing Arrow development from 2018 to 2020. They helped me incubate the startup that became Voltron Data. And so I've gone back to work full time there as a software architect to help them with their Python strategy to make sort of their data science platform a delight to use for the Python user base.

About Posit

Yeah, so Posit started out 2009 as RStudio. And so it didn't start out intending to be a company. J.J. Allaire and Joe Chung built a new IDE, Integrated Development Environment for R, because what was available at the time wasn't great. And so they made that into, I think, probably one of the best data science IDEs that's ever been built. It's really an amazing piece of tech.

So it started becoming a company with customers and revenue in the 2013 timeframe. And they've built a whole suite of tools to support enterprise data science teams to make open source data science work in the real world. But the company itself, it's a certified B corporation, has no plans to go public or IPO. It is dedicated to the mission of open source software for data science and technical communication. And basically building itself to be a 100-year company that has a revenue generating enterprise product side and an open source side so that the open source feeds the enterprise part of the business. The enterprise part of the business generates revenue to support the open source development. And the goal is to be able to sustainably support the mission of open source data science for hopefully the rest of our lives.

And it's an amazing company. One of the most successful companies that dedicates a large fraction of its engineering time to open source software development. So it's been very impressed with the company and JJ Allaire, its founder. So I'm excited to be helping it grow and become a sustainable long-term fixture in the ecosystem.

It is dedicated to the mission of open source software for data science and technical communication. And basically building itself to be a 100-year company that has a revenue generating enterprise product side and an open source side so that the open source feeds the enterprise part of the business.

Many people know JJ Allaire created ColdFusion, which is the original dynamic web development framework in the 1990s. He and his brother Jeremy and some others built AllaireCorp to commercialize ColdFusion. And they built a successful software business that was acquired by Macromedia, which was eventually acquired by Adobe. But they did go public as AllaireCorp during the dot-com bubble. And JJ went on to found a couple of other successful startups.

I think he found himself in his late 30s 15 years ago or around the age I am now. Having been very successful as an entrepreneur, no need to make money. And looking for a mission to spend the rest of his career on. Identifying data science and statistical computing as an open source, in particular making open source for data science work, was the mission that he aligned with and something that he had been interested in earlier in his career. But he had gotten busy with other things. So I think it's really refreshing to work with people who are really mission-focused and focused on making impact in the world, creating great software, empowering people, increasing accessibility, and making most of it available for free on the internet.

And not being so focused on empire building and producing great profits for venture investors and things like that. So I think the goal of the company is to provide an amazing home for top-tier software developers to work on this software, to spend their careers, and to build families, and to be a happy and healthy culture for working on this type of software.

What is Pandas?

It's a data manipulation and analysis toolkit for Python. It's a Python library that you install that enables you to read data files, so read many different types of data files off of disk or off of remote storage or read data out of a database or some other remote data storage system. This is tabular data, so it's structured data like with columns. You can think of it like a spreadsheet or some other tabular data set. And then it provides you with this data frame object, which is kind of pandas.dataframe, that is the main tabular data object. And it has a ton of methods for accessing, slicing, grabbing subsets of the data, applying functions on it that do filtering and subsetting and selection, as well as more analytical operations like things that you might do with a database system or SQL, so joins and lookups, as well as analytical functions like summary statistics, grouping by some key and producing summary statistics.

So it's basically a Swiss army knife for doing data manipulation, data cleaning, and supporting the data analysis workflow. But it doesn't actually include very much as far as actual statistics or models or if you're doing something with LLMs or linear regression or some type of machine learning, you have to use another library, but pandas is the on-ramp for all of the data into your environment in Python. So when people are building some kind of application that touches data in Python, pandas is often the initial on-ramp for how data gets into Python, where you clean up the data, you regularize it, you get it ready for analysis, and then you feed the clean data into the downstream statistical library or data analysis library that you're using.

That whole data wrangling side of things, right? Yeah, that's right. And so it's some history. So Python had arrays like matrices and what we call tensors now, multidimensional arrays going back all the way to 1995, which is pretty early history for Python. The Python programming language has only been around since 1990 or 1991, if my memory serves. But what became NumPy in 2005, 2006 started out as numeric in 1995, and it provided numerical computing, multidimensional arrays, matrices, the kind of stuff that you might do in MATLAB, but it was mainly focused on numerical computing and not with the type of business data sets that you find in database systems, which contain a lot of strings or dates or non-numeric data.

And so my initial interest was I found Python to be a really productive programming language. I really liked writing code in it, writing simple scripts, like doing random things for my job. And you had this numerical computing library, NumPy, which enabled you to work with large numeric arrays and large data sets with a single data type. But working with this more tabular type data, stuff that you would do in Excel or stuff that you do in a database, it wasn't very easy to do that with NumPy or it wasn't really designed for that. And so that's what led to building this higher level library that deals with these tabular data sets in the Pandas library, which was originally focused on, you know, building with a really close relationship with NumPy. So Pandas itself was like a thin layer on top of NumPy originally.

The Pandas API and learning resources

One thing I find interesting about Pandas is it's almost its own programming environment these days in the sense that, you know, traditional Python, we do a lot of loops, we do a lot of attribute dereferencing, function calling, and a lot of what happens in Pandas is more functional, it's more applied to us, it's almost like set operations, right? And a lot of vector operations and so on.

Yeah, and that was behaviors inherited from NumPy. So NumPy is very array oriented, vector oriented. So you, rather than write a for loop, you would write an array expression, which would operate on whole batches of data in a single function call, which is a lot faster because you can drop down into C code and get good performance that way. And so Pandas adopted the NumPy way of, like the NumPy-like array expression or vector operations.

But it's true that that's extended to the types of like non-numeric data operations that you can do in Pandas, like, you know, vectorized set lookups, where you can say like, you would say like, oh, like this, I have this array of strings and I have this subset of strings and I want to compute a Boolean array, which says whether or not each string is contained in this set of strings. And so in Pandas, that's the is in function.

I mean, there's plenty of, I mean, I think there's great books written about Pandas. So there's my book, Python for Data Analysis. I think Matt Harrison has written an excellent book, Effective Pandas. The Pandas documentation I think has, provides really nitty gritty detail about how all the different things work.

But when I was writing this book, Python for Data Analysis, my goal was to provide a primer, like a tutorial on how to solve data problems with Pandas. And so for that, I had to introduce some basics of how NumPy works. So people, array oriented computing, basics of Python, so you know enough Python to be able to understand what things that Pandas is doing. But the, yeah, kind of, it builds incrementally. And so like, as you go through the book, the content gets more and more advanced.

And it is now freely, as you're showing there on the screen, it is freely available on the internet. So JJ Allaire helped me port the book to use Quarto, which is a new technical publishing system for writing books and blogs and website, you know, quarto.org. And yeah, so that's how I was able to publish my book on the internet as, you know, essentially you can use Quarto to write books using notebooks, which is cool. My book was written a long time ago in O'Reilly's doc XML. So not particularly fun to edit, but yeah, but because Quarto is built on Pandoc, which is a sort of markup language transpilation system.

So in the third edition, I was able to negotiate with O'Reilly and make an amendment to my very old book contract from 2011 to let me release the book for free on my website. So it's, yeah, it's just available there at westmckinney.com slash book. I find that like a lot of people really like the print book. And so I think that having the online book just available, like whenever you are somewhere and you want to look something up is great.

Quarto and Pandoc

This quarter thing looks super interesting. If you look at Pandoc, if people haven't looked at this before, the conversion matrix, how would you describe this, Wes? It can convert from, it looks like about, you know, 30 or 40 input formats to, you know, 50 or 60 output formats, maybe more than that.

Well, as history, like backstory about Quarto, so, you know, it helps to keep in mind that JJ created ColdFusion, which was this, you know, essentially publishing system, early publishing system for the internet. And so early on at RStudio, they created RMarkdown, which is basically extensions to Markdown that allow you to have code cells written in R and then eventually they added support for some other languages where it's kind of like a Jupyter Notebook in the sense you could have some Markdown and some code and some plots and output and you would run the RMarkdown render and it would, you know, generate all the output and insert it into the document.

And so you could use that to write blogs and websites and everything. But RMarkdown was written in R and so that limited, in a sense, like, it made it harder to install because you would have to install R to use it. And so in the meantime, you know, with everything that's happened with web technology, there's, it's now very easy to put a complete JavaScript engine in a small install footprint, you know, on a machine with no dependencies. And so Quarto is completely language agnostic. It's written in TypeScript and it uses Pandoc as an underlying engine and it's very easy to install and so it addresses some of the portability and, you know, extents that were in RMarkdown.

Pandas adoption and community growth

When you first started working on this and you first put it down, did you foresee a world where this was so popular and so important? I mean, it was always the aspiration of making Python this mainstream mainstream language for statistical computing and data analysis. It didn't occur to me that it would become this popular or that it would become like the, one of the main tools that people use for working with data in a business setting.

And so the fact that Pandas caught on and became as popular as it is, I think it's a combination of timing and there was a developer relations aspect that there was content available and like I wrote my book and that made it easier for people to learn about the project. But also like we had a serendipitous open source developer community that came together for the project to, you know, grow and expand like really rapidly in the early 2010s.

the different community outreach channels that the Pandas project has done to help grow new contributors. So one of the core team members, Mark Garcia, has done an amazing job organizing documentation sprints and other contributor sourcing events, essentially creating very friendly accessible events where people who are interested in getting involved in Pandas can meet each other and then assist each other in making their first pull request. And it could be something as simple as, you know, making a small improvement to the Pandas documentation because it's such a large project. The documentation is like always something that could be made better, you know, either adding more examples or documenting things that aren't documented or making, yeah, just making the documentation better.

So it's, yeah, it can be takes a while to really get to a place where you can be productive. And that can be discouraging for new contributors, especially those who don't have a lot of open source experience.

That's one of the ironies of the challenges of these big projects is they're just so finely polished. So many people are using them. Every edge case matters to somebody, right? And so to become a contributor and make changes to that, it takes a while, I'm sure.

Yeah. Yeah. I mean, I think it's definitely a big thing that helped is allowing people to get paid to work on Pandas or to be able to contribute to Pandas as a part of their job description, like as, you know, maybe part of their job is maintaining Pandas. So Anaconda, you know, was like, you know, one of the earliest companies who had engineers on staff, you know, like Brock Mendel, Tom Augsberger, Jeff Reback, who part of their job was maintaining and developing Pandas. And that was huge because prior to that, the project was purely based on volunteers.

Yeah, but I, I think, and I think many people know, but I, I haven't been involved day to day in Pandas since 2013. So that's getting on. That's a lot of years. You know, I still talk to the Pandas contributors. We had a Pandas meetup, core developer meetup here in Nashville, pre COVID, I think it was in 2019. So, you know, I'm still in active contact with the Pandas developers, but it's been a, you know, a different team of people leading the project. It's taken on a life of its own, which is amazing. And that's exactly, as a project creator, that's exactly what you want is to not be beholden to the project that you created and be responsible for it for the rest of your life.

So I think even, I don't know what it says on the bottom of GitHub, it says, you know, 3200 contributors, but that's maybe not even the full story because sometimes people, you know, they don't have their email address associated with their GitHub profile. And, you know, how GitHub counts contributors, I would say probably the true number is closer to 4000. So that's, yeah, but that's, I think that's a testament, you know, to the core team and all the outreach they've done and work making the project accessible and easy to contribute to.

Apache Arrow

So in, like around the mid 2010s, 2015, I started working at Cloudera, which is a company that was like one of the pioneers in the big data ecosystem. And yeah, I had, you know, been, spent several years working on five, six years working on pandas. And so I had gone through the experience of building pandas from top to bottom. And it was this, you know, full stack system that had its own, you know, mini query engine, all of its own algorithms and data structures and all this stuff that we had to build from scratch.

And I started thinking about, you know, what if it was possible to build some of the underlying computing technology, like data readers, like file readers, all the algorithms that power the core components of pandas, like group operations, aggregations, filtering selection, all those things. Like what if they were possible to have a general purpose library that isn't specific to Python, but it was really, really fast, really efficient and has a large community building it so that you could, you know, take that code with you and use it to build many different types of libraries, not just data frame libraries, but also database engines and, you know, stream processing engines and all kinds of things. And so that was kind of what was in my mind when I started getting interested in what turned into Arrow.

And one of the problems we realized we needed to solve, this was like a group of other open source developers and me, was that we needed to create a way to represent data that was not tied to a specific programming language. And that could be used for a very efficient interchange between components. And the idea is that you would have this immutable, like this, this kind of constant data structure, which is like, it's the same in every programming language. And then you can use that as the basis for writing all of your algorithms. So as long as it's Arrow, you have these reusable algorithms that process Arrow data.

So we started with building the Arrow format and standardizing it. And then we've built a whole ecosystem of components, like library components and different programming languages for building applications that use the Arrow format. So that includes, you know, not only tools for building and interacting with the data, but also file readers. So you can read CSV files and JSON data and Parquet files, read data out of database systems, you know, wherever the data comes from, we want to have an efficient way to get it into the Arrow format.

And then we moved on to building data processing engines that are native to the Arrow format, so that Arrow goes in, the data is processed, Arrow goes out. So DuckDB, for example, supports Arrow as a preferred input format and is, DuckDB is more or less Arrow-like in its internals. It has kind of Arrow format plus a number of extensions that are DuckDB specific for better performance within the context of DuckDB.

And the goal ultimately was that we, in the future, which is now, we don't want people to have to reinvent the wheel whenever they're building something like Pandas, that they could just pick up these off the shelf components. They can design the developer experience, the user experience that they want to create. And they can get built, you know, so if you were building Pandas now, you could build a Pandas-like library based on the Arrow components in much less time. And it would be fast and efficient and interoperable with the whole ecosystem of other projects that use Arrow.

So it's, it's, it's very cool. It's, I mean, it was really ambitious. In some ways, obvious to people, they would, they would hear about Arrow and they say, that sounds obvious. Like clearly we should have a universal way of transporting data between systems and processing it in memory. Why hasn't this been done in the past? And it turns out that as is true, many open source software problems that many of these problems are, the social problems are harder than the technical problems. And so if you can solve the kind of people coordination and consensus problems, solving the technical issues is much, much easier by comparison.

the social problems are harder than the technical problems. And so if you can solve the kind of people coordination and consensus problems, solving the technical issues is much, much easier by comparison.

WebAssembly, DuckDB, and the browser revolution

Yeah. WebAssembly and all that. Yeah. So, I'm definitely very excited about it. I've been following WebAssembly in general. And so, I guess some people listening will know about WebAssembly. But basically, it's a portable machine code that can be compiled and executed in within your browser in a sandbox environment. So, it protects against security issues and allows prevents, like, the person who wrote the WebAssembly code from doing something malicious on your machine, which is very important.

Yeah, I think it's enabled us to run the whole scientific Python stack, including Jupyter and NumPy and Pandas totally in the browser without having a client and server and needing to, you know, run a container someplace in the cloud. And so, I think in terms of creating application deployment, so, like, being able to deploy an interactive data application, like, with Shiny, for example, without needing to have a server, that's actually pretty amazing. And so, I think that, you know, simplifies and opens up new use cases, like, new, you know, application architectures and makes things a lot easier. Because setting up and running a server creates brittleness, like, it has cost. And so, if the browser is doubling as your server process, like, that's I think that's really cool.

You also have, like, other projects, like DuckDB, which is a, you know, high performance, embeddable analytic SQL engine. And so, you know, now with DuckDB compiled to Wasm, you can get a high performance database running in your browser. And so, you can get low latency interactive queries and interactive dashboards. And so, it's yeah, there's it's WebAssembly has opened up this whole kind of new world of possibilities. And so, it's transformative, I think.

Ibis and the DataFrame ecosystem

I think one of the more interesting areas in recent years has been new DataFrame libraries and DataFrame APIs that transpile or compile to different, execute on different backends. And so around the time that I was helping start Arrow, I created this project called Ibis, which is basically a portable DataFrame API that knows how to generate SQL queries and compile to pandas and polars and different DataFrame backends. And the goal is to provide a really productive DataFrame API that gives you portability across different execution backends with the goal of enabling what we call the multi-engine data stack. So you aren't stuck with using one particular system because all of the code that you've written is specialized to that system.

You have this tool, so maybe you could work with DuckDB on your laptop or pandas or polars with Ibis on your laptop. But if you need to run that workload someplace else, maybe with ClickHouse or BigQuery, or maybe it's a large big data workload that's too big to fit on your laptop, and you need to use Spark SQL or something that you can just ask Ibis, say, hey, I want to do the same thing on this larger data set over here, and it has all the logic to generate the correct query representation and run that workload for you. So it's super useful.

But there's a whole wave of work right now to help enable people to work in a pandas-like way, but work with big data or get better performance than pandas, because pandas is a Swiss army knife, but isn't a chainsaw. So if you were rebuilding pandas from scratch, it would end up a lot different. There's areas of the project that are more bloated or have performance overhead that's hard to get rid of. And so that's why you have Richie Fink started the Polars project, which is kind of a reimagining of pandas data frames written in Rust and exposed in Python. And Polars, of course, is built on Apache Arrow at its core. So building an Arrow-native data frame library in Rust and all the benefits that come with building Python extensions in Rust.

There's different schools of thought on this. So there's another project called Modin, which is similar to Ibis in many ways, in the sense of, like, transpilation and sort of dynamically supporting different backends, but sought to closely emulate the, you know, exact details of, like, you know, the API call, the function name, the function arguments must be exactly the same as pandas, with the goal of being a drop-in replacement for people's pandas code. And that's one approach, kind of the pandas emulation route.

And there's a library called Koalas for Spark, which is like a PySpark emulation layer for the pandas API. And then there's other projects like Polars and Ibis and Dask DataFrame that take, like, design cues from pandas in the sense of, like, the general way in which the API works, but has made meaningful departures in the interest of doing things better in many ways than pandas did in certain parts of the API, and making things simpler and not being beholden to decisions that were made in pandas, you know, 15 years ago.

And so Polars, for example, is very expression-based. And so everything is column expressions and is lazy and not eagerly computed, whereas pandas is eager execution, just like NumPy is, which is how pandas, you know, became eagerly executed in the first place. And so I think the mantra with Polars was, we don't want to support the eager execution by default that pandas provides. We want to be able to build expressions so that we can do query optimization and take inefficient code and under the hood rewrite it to, you know, be more efficient, which is, you know, what you can do with a query optimizer.

SQLglot

So SQLglot project started by Toby Mao. So he's a Netflix alum and, you know, really, yeah, really talented, talented developer who's created this SQL query transpilation framework library for Python and, you know, kind of underlying core library. And so the problem that's being solved there is that SQL, despite being a quote unquote standard, is not at all standardized across different database systems. And so if you want to take your SQL queries written for one engine and use them someplace else, without something like SQLglot, you would have to manually rewrite and make sure you get the typecasting and coalescing rules correct. And so SQLglot understands the intricacies and the quirks of every database dialect, SQL dialect, and knows how to correctly translate from one dialect to another.

And so Ibis now uses SQLglot as its underlying engine for query transpilation and generating SQL outputs. So originally Ibis had its own kind of bad version of SQLglot, kind of a query like SQL transpilation that was powered by I think powered by SQLalchemy and a bunch of custom code. And so I think they've been able to delete a lot in Ibis by moving to SQLglot. And I know that, you know, SQLglot is also, you know, being used to, you know, power kind of a new, yeah, being used in people building new products that are Python powered and things like that.

So very cool project and maybe a bit in the weeds, but if you've ever needed to convert a SQL query from one dialect to another, it's, yeah, SQLglot is here to save the day.

I enjoyed the conversation and yeah. Yeah, there's a lot of stuff, a lot of stuff going on and still plenty of things to get excited about. So I think often people feel like, you know, all the exciting problems in the Python ecosystem have been solved, but there's still a lot to do. And yeah, we've made a lot of progress in the last 15 plus years, but, you know, in some ways it feels like we're just getting started. So excited to see where things go next.