Transcript#
This transcript was generated automatically and may contain errors.
Welcome to The Test Set. Here we talk with some of the brightest thinkers and tinkerers in statistical analysis, scientific computing, and machine learning. Digging into what makes them tick, plus the insights, experiments, and OMG moments that shape the field.
On this episode, we talk with Leland McInnes, creator of UMAP, an algorithm that turns high-dimensional data into something easier to visualize. It's been used all over, from tracing illegally mined gold, studying solar wind, and even helping artists work with whale song.
Leland joined the Tutte Institute as a pure mathematician, but kept finding interesting problems on the data science side. He says, I just want to find ways to condense and extract information to just allow someone to explore a dataset better. We talk about the challenge of imposter syndrome and interdisciplinary work, his approach to AI, and why he thinks exploring data depends on human taste and curiosity.
Hey everyone, welcome to The Test Set. We're joined by Leland McInnes, who's a research mathematician and data scientist at the Tutte Institute for Mathematics and Computing, and contributed to HDBSCAN and created UMAP, a really interesting and popular algorithm for clustering and you've probably seen it in a lot of visualization. So, Leland, so happy to have you on. Thanks for coming on.
Thank you so much for having me. Really appreciate it.
Yeah, and then just by way of introduction, I'm the host, Michael Chow, and I'm joined by Hadley Wickham, who's chief scientist at Posit, and Wes McKinney, who's a principal architect at Posit and founder of Kenio AI Computing Tools Company. So, happy to have everyone together. Do y'all, I'm curious, have y'all met before?
Met you at a SciPy, Wes. I obviously remember it because it was a chance for me to meet one of my heroes. You obviously will not remember it because it was a long time ago and I hadn't done much by that point.
Yeah, I think the first SciPy that I went to was in 2010. And so, there've been a lot of years, a lot of conferences, and a lot of alcoholic beverages in between now and then.
It makes sense though, SciPy. I think Leland, a lot of your tools have been in Python and you have a lot of implementations of things in Python. Do you go to most of the SciPy conferences?
I went for many years. I didn't go this year and I missed a few years around COVID. Otherwise I try and get there whenever I can. It's a great conference.
I went to SciPy this year, so we could have crossed paths, but I missed out.
I think the joke is Hadley and I are trading places. I think some years he's at a Python conference, I'm in an R conference. Times have certainly evolved. A great deal is the field of data science has become more polyglot and the communities are intersecting a lot more.
Yeah, I have not made it to an R conference yet. That's on my list of things to get around to doing. I do have colleagues that do a lot of R. Python's where I landed, but I'm not wedded.
Illegal gold mining, solar wind, and the uses of UMAP
And you must go to a bunch of academic CS conferences too?
Yeah, math and CS conferences as well.
Yeah, reading about your background and a lot of things you've done, so interesting where you're kind of straddling pure mathematics and data science. And I think one thing that really caught me from some of what you said before this interview is you mentioned when thinking about the downstream impact of your work, people reaching out sometimes for help or feedback on problems. And I almost wondered if one way to really kind of introduce people to UMAP and some of this is I saw you mentioned people reaching out about things even like illegal gold mining. I don't know how much you can say about that, but I'd be curious how people are using things like UMAP.
So this is one of those things where I know nothing about the subject matter and I got someone reached out for a little advice and just consultation on using UMAP within this framework. But it was about managing to identify gold samples that had come from locations where there were not supposed to be mining. They were essentially trying to mix it in with other gold so it just gets laundered into the general supply. But there are like impurities in gold that can be traced back, and so for just managing to pull out clusters and structure in this sort of analysis of the trace minerals in gold, UMAP was one of the tools that got used in that. And so it's just really interesting to see the different ways that these tools actually get used in practice to solve real world problems.
If I understand what you're saying is sort of like they have a lot of samples and they can take a lot of measurements on the gold and that sort of makes it a really high dimensional problem, like each measurement is kind of a dimension. And they kind of want to break it down to cluster all the observations to say like what are these groupings and maybe one of these groupings is sort of like the illegally mined gold, is that?
Yeah, that's my understanding. It's been a while, but it was an interesting use case. There are a lot of very strange little use cases like that that crop up. I mean it's been used in astronomy for something to do with identifying things in solar wind. Again, I'm not an expert in astronomy and I can't comment on the specifics, but it's just intriguing to see the kinds of use cases that crop up.
Yeah, I've had to try and desperately quickly learn various different subjects as they come up. I'm never going to be an expert in it, but I've had to learn more biology than I ever wanted to know because UMAP gets used a lot in various fields in biosciences and that is 100% not my field. But it comes up often enough that I think I have had to try and learn as much as I can to at least give vaguely reasonable advice if people ask questions.
From pure math to casinos to the Tutte Institute
Yeah, I'm really curious. I know you also mentioned you studied math and then after your master's you were working in a casino a lot. It seems like you've done a lot of this sort of pure math to data science. I'm really curious almost to give people a sense of kind of how you got into all this. If you could run us through a bit of your background and how you got to the casino and where you went from there.
Yeah, so I went through an accelerated program at university that kind of put me through undergrad and then master's in a somewhat more compressed program. I started out actually in physics but ended up just being pure math by the end. So my master's was in algorithms in algebraic geometry. But by that point I was kind of burned out on academia to be honest because it had been a pretty intense environment.
So I decided to just try doing something else. So I looked around and there was a company that was hiring mathematicians which you know doesn't happen that often. So I applied there and that was a company that turned out they specialized in analytics for casinos. So it was all about doing complex data analytics on casinos and that's a really strange field to be involved in. Because their primary product in practice really was helping casino managers rearrange slot machines on the slot floor. Because apparently if you carefully rearrange the slot machines just right you can increase your revenue by one percent. And if you increase your revenue by one percent at casinos that turns out to be a lot of money.
So I was involved in that and so that's where I learned data analytics originally. I did a lot of data analytics and visualization and oddly enough that was like back in 2000 and I remember I was actually playing around with self-organizing maps there and different ways of doing visualization.
But ultimately working for casinos is kind of soul crushing and I just couldn't quite bring myself to do that anymore. So I looked back to academia and ended up heading off to university to do a PhD in pure math. That was in algebra and with topological algebra really and I also did a bunch of algebraic topology.
Once I was done that I wanted something a little bit different. I didn't want to stay in academia and the Tutte Institute here in Canada, they're a government research institute. We have people who just do really pure math stuff and data scientists doing real data science things. I actually joined as a mathematician and was in the pure math space working there but you know what pure math is hard. And you spend a while beating your head against problems that just don't want to be solved and then occasionally because we're a small group we'd chat with the other people there and the data scientists they had easy problems. So it was just a chance to try out some problems that I could at least work on attractively and so I spent a little while doing that and have now I guess I'm fully on the data science side and I try and pretend I'm still a mathematician at heart.
I guess you and I kind of have that in common like I actually started out as a pure math person. I didn't end up going to graduate school or PhD in pure math but I think one of the reasons why I stopped doing pure math was similar to your story like I had this feeling that you could spend your entire career and maybe you would happen on a handful of important problems or problems that would have impact if you were lucky. But you would spend a lot of time working on writing papers and doing work that was important within the context of the field but it would be difficult to have practical impact in the world. Whereas when I first got exposure to data science problems and data problems I had this palpable feeling of like oh there's all these problems that need to be solved and I can go and do something it's very practical and it has impact.
So I ended up a little bit in data science as an accident but it sounds like your path was a little bit similar. So not to knock on pure math like the world needs pure mathematicians like we need cryptography and all those things and all that comes out of pure mathematics. But I also found that there's people that are way smarter than I am working on doing pure math research and so once I observed some of those really incredibly super genius pure math people who are just so far out ahead of me I'm like I just don't think I can succeed quite the way I would like in this field.
Yeah no I think it's a very similar sort of story. Working in pure math is definitely a challenge. I mean it's very rewarding and I still love pure math but instead of looking for low-hanging fruit to try and pick it's pretty much just picking fruit up off the ground in data science. There are so many problems and some of them are very hard but there's just so many to choose from that are just sitting there waiting to be tried. Whereas you know math has many thousands of years of people hammering away at these problems the easy ones have been found although now you can just tell AI to do a breakthrough.
Is it common for the pure math people to collaborate with the data science people?
So I think this is one of the places where the Tutte Institute is pretty special in that it is a place where that happens. We've got others who kind of dip back and forth between each side as it goes and as I said it's a relatively small research institute we don't have that many people so everyone talks to everyone else. And you know if you can take a data science problem and turn it into a pure math problem it's not that hard to get the pure math people working on it. And as we said there are people a lot smarter in the pure math realm working on these things so if you can turn your problem into something that they will want to chew on then you can get a lot of talent working on a problem for you.
Using AI as a sounding board
Oh, caveat and say not really, not yet. But that's mostly because I'm still really trying to figure out how best to use it. So it hasn't changed my work because I'm still playing with it. I don't think I've settled on how best to make it fit because there are like, it's very powerful. It is done, I've certainly been working with it, playing with it and learning a lot and it certainly has helped but I don't think I have a settled workflow or decision about exactly where it fits.
So one of the easy ones is in software development and coding. Some of the latest models are remarkably good at that and that has been just an interesting learning curve to understand how much it can do and also trying to learn what it can't do so well. But otherwise I've mostly been using it as a way to, I guess it's a rubber duck to talk to. I'll talk to the AI and throw out some ideas of directions I'm interested in going and it throws back a bunch of ideas and they're not always great but they're not bad. It's a useful sounding board to explore ideas.
I mean, I certainly take anything that it says or does with a grain of salt. But I mostly use it as a tool to clarify my thinking, or like, what are questions that I should be asking, like is there something that I might be missing here. And so I think in terms of like helping you think laterally or like maybe look at something from a different perspective it can be really really helpful for that. Unfortunately you know even the latest generation of models seem to have like a cognitive bias where they're inclined to agree with you and tell you that you have a good idea even still.
And so for that reason like even if the AI appears to agree with what I say or say yeah that's a great idea, I'm like is it though really. And so sometimes a gentle pushback, say like you can say like is it though really, let's be critical, you're not going to hurt my feelings, and it will actually have some criticisms that it withheld in the first go around because it was worried about hurting your feelings.
My go-to approach for that is I find it's useful for me to, I'll write up an idea, it's just a little markdown document, sketch something, upload that and say I got this suggestion from someone, I think it's not very good, I think it has real problems, can you explain what's wrong with it. And then it tries to agree with you and tell you how bad it is, which is great because that's what I actually want.
Since Leland's also from New Zealand I can share my pet theory that the problem is that too much of the reinforcement learning is trained on Americans who like to be sucked up to more than like New Zealanders. So I think if they had New Zealanders do more of that there would be like less sucking up and more brutal honesty.
Do New Zealanders brutally honestly... is that a... not brutally, I'm more blunt than Americans.
Yeah, there's like a really great book that I recommend to everyone called the culture map that's basically about the way like the different working styles. Obviously it's like every individual is different but like in generality like the sort of median working style of different countries and cultures. And Americans notably have a hard time delivering critical feedback without wrapping it in a sugar coating. And this is extremely confusing for like other cultures that are used to receiving more blunt critical feedback.
That reminds me of this blog post from Kieran Healy which is about academic feedback and like how US grad students should interpret what faculty trained in the UK say. Like if they say that's not bad the US student hears that's poor but what the faculty member actually meant is like that's good. Or if you say I am confused the US PhD student thinks that means I am confused but what the faculty member actually meant to convey is you are an idiot. I just thought it was like hilarious and I read this as I was starting to advise my first US PhD students and I was like wow these PhD students are probably really confused by the feedback that I am giving them because they're hearing it very differently to how I have it.
Yeah, I think you have to disown your idea a little bit to get them to be honest about it.
Yeah I have like a pretty systematic adversarial review pipeline for everything I do. And in particular, whenever I present something for review by a fresh agent session that has no prior knowledge of what you're working on, you don't tell the agent that you wrote it, just like here is a plan that was written by another agent. And just ask for critical feedback. I think you could even say like I think there's things wrong with it, find what the things wrong with it are.
In the software engineering community there's this very alluring idea of the dark factory, like this idea of being able to produce things with AI without much human intervention, basically using feedback loops of adversarial agents. One agent writes the specification, other agents review, one agent implements, other agents verify. But I'm more of like the human assisted factory type of mindset. The clankers as we call them are still doing the work but we need to be in the loop to provide judgment and steering. And I think without that like you get slop out on the other end.
So early on in my journey building software and tools and systems with AI there were times when I delegated a little too much to the agents and was surprised to see that what came out on the other end wasn't very good. And so finding the right balance of not letting them run with it too much and certainly adversarial feedback I think helps a lot. But they can't be given too much, you know, you can't delegate your taste to them. Taste is provided by humans. And it's unfortunate because everyone is really hoping desperately that they can write a prompt and step back from the computer and go have a beer or go for a walk or something and come back to like something that is good. But in my experience like that almost never happens. And so if you want it to be good you need to be a maniac about the details and really engaged with the fine details of what you want the agents to produce for you.
The human side of data exploration
It seems like one theme with a lot of your tools and the more applied data science work is it does seem very human-centric. Like even UMAP and things like this commonly end in a visualization. It does seem like one thread is that a lot of your work has been really useful for people to like dig around and make sense of things and visualize.
Yeah, I mean, I guess if there's one theme in my work it was that when I started very early in data analytics and data science, obviously the first big problem that you come across is you get data and now you have to do all the data cleaning and EDA work and all this stuff to sort through that. And then you can finally get around to the like supervised machine learning once you've got it sorted. And I guess I never got past the first part. I'm still trying to build the tools to do that EDA framework, to do exploratory data analysis.
And I guess my focus has been on the fact that I feel like we're lacking in tools for EDA on unstructured data. I think you know tabular data is something for which we have amazing tools and Hadley and Wes have both written plenty of amazing tools for doing great EDA on tabular data. But when you have a giant pile of documents or images or videos or a mixture of all three how do you work out what's in that data set and how do you explore it in a meaningful way?
And I don't, I think the answer is you're going to need to feed that to a human somehow because I don't think an AI can just have all the answers because of exactly as Wes was saying it's about taste. You have to have some sense of what I'm looking to find or know more about and it's very hard for AI right now to do that. But you know what's the channels via which I can get the information about that data set into my brain and visualization is one of the strongest ones. But you know other means of building summarizations, that's where a lot of the clustering and things like that come in as well. I just want to find ways to condense and extract information to just allow someone to explore a dataset better and that's the through line of all the various kinds of tools I've been building. That's just, I want to help people understand their data.
I just want to find ways to condense and extract information to just allow someone to explore a dataset better and that's the through line of all the various kinds of tools I've been building. That's just, I want to help people understand their data.
Yeah, it's so interesting and I guess even the art behind you really speaks to that. It's like scientists and it's people looking for illegal gold mining but also artists have found use for this, of almost taking data in the world or like relationships and even rendering it artistically as a work of art on its own. I think that's a really powerful testament to the taste and aesthetic aspect of it.
Yeah, so the work behind me is by an artist Rafik Anadol who does a lot of digital art. He uses UMAP in some of his work, he uses a lot of AI tools, but his tagline is he wants to use data as a pigment for generating art, data is his paintbrush. And I think that's an intriguing view on this. I think there's a lot you can do with modern tools, AI tools and otherwise for actually generating art. And I don't mean in the sense where you write a prompt and it generates a picture for you, but more as a tool to do more interesting things.
There was a great artist, Kyle McDonald, who I've had back and forth with a couple of times. He's reached out and one of his projects was just analyzing whale song for a physical art installation that was going to get built, but it was just in the background of the analysis. He was using UMAP for some things and so I had some back and forth with him about that.
Fancy tooling in data science was not showing up necessarily in the end product art, but it's a useful tool in the building structure that goes into producing the art. And I mean, I think there's a lot of processing like that that I think these tools have a lot of ways they can be used that aren't obvious.
And the map, the idea of the map is such a powerful metaphor, and people love to look at maps. Not just of physical things, but it'd be like maps of the internet, and maps of music. Yeah, those spatial connections are something that feels like humans just have some in-built desire to think about and look at and experience.
Yeah, spatially navigating data is a magical thing if you can make it happen, and it's challenging to make it happen well, it turns out. I mean, I wrote UMAP quite some time ago. I still think there's better algorithms out there, and that's one of the things that I'm busy working on now, is just trying to understand better how this actually should work, because I think there's more answers out there.
And I think managing to make data spatial and the geometry of data is actually a fundamental thing in all these AI representations, like the neural embedding models and these things. That's really about the interrelationships and geometry of the data, and so it's about geometry and topology of information. And so everything is spatial for the internals of the AI, and I think if we can make that interpretable to people, that will be very, very helpful.
High-dimensional space and broken intuitions
Okay, I'll see what I can do. So if you're interacting with Claude, you have your prompt that you send it, and it basically does some processing, but it's processing stuff in pure numbers, eventually, as it's shoving it all through the neural network. And you can take an individual layer of that neural network and pull out the numbers that it has for processing a single thing. And that's a vector of numbers. And it turns out that these vectors of numbers actually organize themselves in space. In a meaningful way.
And so instead of handing something in to Claude to get a textual response out, there are other models that are designed specifically for I'm gonna hand some text in, or I'm gonna hand an image in, and it's just gonna spit out the vector of numbers associated to that. And though the interrelationships between those vectors are meaningful on a semantic level, the vectors will be similar, close to each other in space, if the semantics of the text was similar. And so this is a very powerful way of dealing with these things.
But now we're trying to do geometry in very high dimensional spaces, because these vectors that they spit out, they're not nice ones you can plot easily. They're usually, these days, thousands of dimensions. And so the geometry of high dimensional space turns out to be very important if we want to understand what's going on inside the neural networks.
That's a fun challenge that I enjoy trying to work on, because I think it's almost impossible for people to have good intuitions for what geometry in high dimensional space looks like. If you want, I can give you a slew of examples for how to break your intuition.
Yeah. Okay. So if I make a sphere, if I draw a circle and then draw a box around it, the sphere mostly fills the box. If I do a sphere and draw a cube around it, there's slightly more space in the corners of the cube. If I just keep going up in dimensions, eventually the sphere contains almost no volume whatsoever and all the volume lives in the corners of the box. So somehow everything lives in the corners.
So another way to think about it, I can take a circle and draw a little tiny thin shell around it. What's the area of the circle compared to the area of the shell? Well, I mean, it's almost entirely circle. I can do a ball and make a thin candy coating, the rind around an orange. The orange contains most of the meat. The rind is just this little thin bit. But no matter how thin you make that thin bit, there's a high enough dimension that more volume lives in the rind of the orange than inside the orange itself. So suddenly everything is living on the surface of these spheres.
If you take a sphere, ball, and draw a band around the equator, what height of band do you need to cover 99% of the surface area of the ball? In what we're used to, you have to cover most of the height of the ball. In high dimensions, that value goes to almost zero. Almost all the surface area of a ball in high dimensions lives on the equator. But I said the equator. Any rotation will do. So any equator you draw around the ball contains most of the surface area of the ball. It's very hard to conceptualize all of that. And this is the problem. High dimensional spaces just behave in ways that are run entirely counter to our intuitions about how this should work.
I appreciate you taking a stab at that because I think that it's, I don't know, it seems important if like high dimensional geometry seems like a really useful, fruitful area. And it sounds like you're really aimed at it. I feel like it's such a helpful kind of like way to prime the pump on how this behaves. If everything's in the rind of the orange, it's such a helpful analogy for at least trying to glimpse what these problems might kind of like feel like to someone.
Yeah. Your intuitions for how it should behave are all based on your everyday experience, which is in a like three dimensional world. And if those intuitions are just really wrong, it turns out for these problems. And so it's very easy to design algorithms and work with the data thinking you know how it should behave. And it turns out it doesn't. So the examples I was trying to give, they're mostly just to try and demonstrate that you just shouldn't trust any of your intuitions because your intuitions are probably wrong. And you're going to have to do the hard work of really reasoning through very carefully rather than just basing everything on, oh yeah, you know, everything should behave this way. It makes sense. Well, not in high dimensional data, unfortunately.
EVOC, Toponomy, and new tools for unstructured data
Yeah. So EVOC is for embedding vector oriented clustering. And the goal there is that essentially these neural embedding vectors are how a lot of our data is represented now. There's still always going to be lots of tabular data. But for all the things, as I said, like documents, text, video, audio, images, the way people represent those now is with these neural embedding vectors because you can do lots of machine learning with them. But now we need ways to work with those. So it's a clustering algorithm designed specifically with that kind of high dimensional data in mind to be as fast as possible and just get you to some decent clustering results quickly.
And you're working with data scientists at the Tutte Institute on it? What does that look like?
That one's primarily my project. I'm working with some people on a different project called Toponomy. That's from the Greek. It's the idea of naming places. So this is very much, as Hadley was saying, that spatial sense and the sense of a map is pretty important. But a map is meaningful when you can put some place names on it. And so if the map is of an information space generated from the space of all these documents about something, how do I put meaningful names at many different resolutions on that map?
And so with that one, LLMs and AI is playing a big role because they're turns out really good at language. But a lot of it comes down to just managing to do things like provide that multi-resolution clustering and how do I extract the right information to feed an LLM to have it give a good name to this thing? So it's kind of related to topic modeling, but its goal is to be a little bit richer. And ideally, topic modeling for whatever you can turn into embedding vectors.
That made me realize there's a little project that I've been working on that I started to use embeddings for. So at Posit, we do these monthly Ask Me Anythings where anyone in the company can ask questions of leadership. And so now we have this kind of database of questions and answers. And of course, people ask the same question, but there's no great way to say, here's all the similar questions that you asked in the past. And so I was like, okay, well, I can at least do some related. When I look at one question, show me the related question. So I was like, well, that's easy to do now. Just do some embeddings and then find the ones that are closest. But that's making me think it would be great to have a map of this as well, so you can kind of see these are the sort of clusters that questions tend to come into.
Yeah. It sounds like it's probably a fascinating dataset. There are probably some really tight clusters where people keep asking almost the same question. And then there's probably some interesting clusters where it's diffuse and it's lots of people asking variations of the same question, but heading in all sorts of different directions. I mean, there's so much you could learn and explore in that sort of data. And so yeah, this is the tooling that I want to try and build is how to explore when it's not a nice tabular dataset anymore.
Yeah. I mean, my mental model of these things, especially the 2D map views, is that it's a lens through which you're viewing the data and it's deliberately providing a distortion. I mean, a lens is not very useful unless you want a microscope lens that will show you detail or a telescope lens that'll take in a lot of information and condense it down. Whatever you do, you're going to distort the data. So you just want to have as many different lenses through which you can look at the data. And so you just want to swap between lenses so you get these different views. And that's, I think, the best way you can get to data understanding.
Because to bring it back to maps again, the classic example of this is the London tube map, which fundamentally distorts distance but gives you a much better understanding of the network. You don't want multiple views like that of your data.
I did cognitive psychology and a little neuroscience, and that does remind me of brain anatomy, where we have names for a lot of the different parts of the brain and how they connect to each other. But obviously, all our brains are different, even the shape of various places. So somehow we agreed upon these labels, but also in research, you have to figure out how to mash them onto each other, register it onto a canonical brain. And it is interesting that need to categorize the parts, but also sometimes map them onto each other and what you keep and what you lose in the process.
Imposter syndrome and interdisciplinary work
I saw that you mentioned like imposter syndrome, that even, I mean, I would say you have an incredible list of accomplishments and I would say like dancing in pure mathematics and in data science is a pretty astounding feat. I was really surprised to see you mention that you experienced imposter syndrome. I think that's really common. But I'm curious if you could say a bit about that piece.
Sure. Well, I mean, one example you said is I'm in pure math and data science, but that mostly means I feel like I'm failing to keep up with the pure math. Like I go to math conferences and it's a struggle because there are people doing amazing things that I can barely wrap my head around. And then in data science, I'm a dilettante, dabbling in data science. I'm not a data science expert and I don't really know what I'm doing. So I feel out of place in both worlds.
I have ended up writing software tools in Python, but I'm certainly not a software engineer and I don't really feel like one. And I constantly feel like I don't know what I'm doing as far as software engineering goes. I'm doing the best I can and it seems to be working, but this is an ongoing thing. I constantly feel a little bit out of place. And so yeah, imposter syndrome is certainly something that I do try to deal with. And so mostly my way to deal with that is to just try and keep my head down and focused on what I'm doing and try not to worry about how it fits into everything else.
I'm a huge fan of interdisciplinary work. I think I really feel like that's where a lot of really useful things happen. When different things end up mixing together, that's when interesting things happen. And so I'm happy to be able to be in this interdisciplinary spot. As I said, I feel like that's one of the real strengths of the Tutte Institute, the Research Institute, is that we have a bunch of this interdisciplinary. And I mean, to some people, I guess it's not very interdisciplinary because it's like math and computer science and data science, and maybe those are all the same as far as some people are concerned. But they're wildly different to people working in those fields.
Experimenting with Rust and AI coding
I was curious too, I noticed are you working on a Rust library now too?
Who isn't working on a Rust library?
So actually that one is, I said I needed projects so I could try out AI and learn how good it is at programming. So that was one of my AI projects. I am not Rust fluent at all, but I had a Python library for approximate nearest neighbor search that made use of the Numba library, which JIT compiles the Python down to LLVM machine code. So it actually runs surprisingly fast, even though it's pure Python. And so I was just curious, like this seemed like a good task for AI models. Rewrite this library in Rust, because you should be able to make it go faster.
And so what I've just put up on GitHub is the final output, but it took way, way longer than I expected. I started on this well over a year ago. And the models back then, I think it was like Opus 4.5 with Claude I started out with. It could not do it. It wrote a Rust version of the library and it did function eventually, but there was a lot of back and forth with the model saying, this is great, but please run the benchmark again, because it's still slower than the Python. And it took a long time. And so I think only the latest models have actually gotten to a point where it could make it run faster than Python code.
Well, because it was already being just in time compiled by Numba, right?
Yeah yeah. It's fast Python code, but still, I feel like a Rust native implementation specialized should be able to get at least as performant as that. And so it turns out that this is a challenge for the model. So this was a learning experience for me to learn how good are the models at this. And so I think they've gotten there in as much as I've produced it, it's slightly faster than the Python in a number of ways. So this is good. It's not as flexible, but that's your trade-off.
But that was me having a learning curve and an excuse to play around with the models.
Yeah, I think the thing that really made it work was that there was a very targeted goal and I could actually have it write tests and benchmarks. I had test suites in the pipeline. So this was very much what Wes was talking about. If you can make it very specific, you can have it run away and do a bunch of stuff. But it needed the specificity and it needed the ability to just go back and forth and benchmark against the Python code and have it run it itself and go, oh yeah, no, you're right. It is still slower. That was kind of critical to manage for me to make it work.
Asking better questions
Yeah, this was my always my pitch for why you needed visualization and modeling. Like visualization doesn't scale because a human's got to look at it, but it can surprise you. And models, like every model makes some assumptions and they can never question those assumptions. So at some level it can't surprise you.
I love that quote. I think it's fabulous. I reuse it all the time because it is very much, I think the motivation for the kind of exploratory work that I'm interested in. I want the thing that will surprise people. That's the most important part. Because I'm interested in helping people ask better questions rather than finding answers. That's, I guess, my would-be quote. Because I think there are lots of ways to squeeze data and get answers. But I think knowing what question you should be asking is a much harder problem these days.
I'm interested in helping people ask better questions rather than finding answers. Because I think there are lots of ways to squeeze data and get answers. But I think knowing what question you should be asking is a much harder problem these days.
Yeah. My colleague, John Healy, his go-to quote is just, look at your data. That's all he ever says. And he means that in the broadest possible sense of look at your data. Open it up in Excel and stare at lines in a spreadsheet. Whatever. Just actually look at what's in your data. I've seen him working with analysts who bring a data set. And he'll ask a bunch of questions about, well, what's true of this data? What do you at least know about it? Because whatever they say is true of the data, it's almost always a thing that he will then go and falsify by going through the data. No, that in fact is false. There are negative values in here. I'm sorry, but the timestamps are not contiguous and they're duplicated in some places or whatever. Whatever assumptions people make about their data turn out not to be true.
Yeah, it's honestly so fascinating to hear about your path through pure mathematics and data science and to see these tools and algorithms like UMAP just cut across so many areas. Yeah, I think I'll be thinking about for a while also just the exploratory nature of it and how we could live and we could drill into exploration way more than maybe we expect. Any parting words?
Look at your data. Ask better questions.
Yep, Leland, thanks for coming on and great seeing you.
Yeah, thanks so much for having me on. I really enjoyed it.
The Test Set is a production of Posit PBC, an open source and enterprise tooling data science software company. This episode was produced in collaboration with creative studio Agi. For more episodes, visit thetestset.co or find us on your favorite podcast platform.