Transcript#
This transcript was generated automatically and may contain errors.
Let's go ahead and get started and I'm really excited to introduce our featured lab managers for today, Daniel Chen and Kylie Ainslie. Daniel, you are going to go first today. Would you like to introduce yourself first?
Hello, yes, I am a lecturer at the University of British Columbia, which is up in Vancouver, Canada. And yeah, I'm gonna end up talking about a project that ends up being very local to the city that I'm currently living in. Sweet. I'm so excited. And you teach data science courses. And I believe that you mostly teach in Python, right? I teach data science courses. So we have the Masters of Data Science program at UBC. But that program actually gets taught in both languages, R and Python. So we have courses in both. Some of our project-based courses, they're sort of open to the students for them to pick. And then what ends up happening, at least in the last couple of years, is the students themselves just all pick Python projects. So even though we teach them both, it's up to them on what they want, because I guess they feel like Python's more popular in industry. But if you look at the TOPI index, R is slowly creeping up.
It sure is. I still have questions about that index sometimes, though.
Well, it's fantastic to get some insight into that grad program. A lot of people have questions about data science grad programs. That's something we can talk about in the Discord, too. Like, if you've been through a data science grad program, I have myself. If you've taught in one, I've taught in private programs. If you have questions, put them in the chat. We can chat about them later. And then we are also joined by our other project lab manager today, Kylie Ainsley, who's been on the lab before. Kylie, would you like to introduce yourself?
Sure. Thanks, Livvy. My name's Kylie, and I currently work at the University of Melbourne in Melbourne, Australia. But luckily, I am not currently in Australia, because it's the middle of the night. I am in Central Time, US. Hooray. And I am an infectious disease modeler by training. And I work on a lot of projects that help the government make decisions about how to slow infectious disease spread and that kind of thing. But that's not at all what I'm going to talk about today. Today, I'm going to talk about a pet project that I spent a long, dark Northern European winter creating to better find the R packages that you need when you need them. So we'll talk about that more later.
Sweet. I cannot wait. Also, I love that Michael has his cup of coffee or tea in his hands. I'm very jealous. It looks very warm and cozy, Michael. Okay, let's go ahead and get started. Daniel, would you like to share your screen and give us a little rundown on what you're going to be sharing today for your project? Everybody, I'm going to set a timer to help Dan out. I'm going to set it for about 25 minutes here, and you might hear me rudely interrupt and tell Dan how much time he has left. Please be patient with me. Take it away, Dan. Oh, wait, you're muted, Dan. Take it away after you unmute. Hello. Hi.
Cherry blossom mapping in Vancouver
So this is one of the end results of what I'm going to be showing you, which is here are the top 30 cherry blossom locations in the city of Vancouver and sort of a traveling salesman routing path to hit all 30 points. So that's sort of like the end result that was originally when I started this project. It was hey, how do we go and find like places to go see cherry blossoms? And then given all of these points, like how do we like essentially move through them? And I'll sort of talk through like the thought process of what happened.
So originally started with this LinkedIn post from Kyle Walker. So he's a good friend of ours in the art community doing all everything, everything and anything to do with maps and census data. More than he could possibly ever want to know about maps or any R package. He's amazing. Yeah. So he's got everything to do with maps in R. And then he had this post two months ago. Well, at the time, I just saw it around like this traveling salesman problem in R. And I thought that was pretty timely because it was oh, I know exactly how to apply this type of problem through like moving through like locate any set of locations in on like points on a map. So that was like sort of the starting point of like I think I have all the pieces. I know how to find like tree data, which comes from the Vancouver open data portal. That's because I knew that data set existed. And I was like, okay, we can totally map these two things together and then like route through the city.
Yeah. So one of the things is Vancouver is actually they have a big giant open data portal. And what the city also has is they have information not just on cherry blossoms, but like every single tree planted by the city in a data set. So that's, you know, there's a lot of trees in the city, which is really nice. And then, you know, the thought process is, hey, you can go and filter this by, you know, something along the lines of like flowering cherries or cherry blossoms and stuff like that. So those are all the raw pieces that I whoops, that I had. And then I can go through.
I have a question. Yes, question. So I have a question. All right. Yes, question. Why, why were you doing this? Besides seeing Kyle's LinkedIn post? Why were you like, I must map all the cherry trees in Vancouver?
Yeah, because cherry blossom season is only about a month in the city. They bloom in different points. My mom was actually coming to visit during this time. So I was like, hey, this is I need to go figure something out. So we would go and so I was like, okay, let me go find some certain locations just to be able to have a plan for where to go. And then I mapped out like the top whatever points just in case like certain place we're not in bloom, which we'll sort of talk about that as a follow up problem afterward. So that's, that's why. And then it was originally the thought process was, oh, if I map out enough of these points, maybe I can run through the city or bike through the city through all of these. And so that's sort of the thought process through all of that.
Walking through the code
All right, so here's some R codes. And this is a screen sharing session session. Could you make your text slightly bigger? So we can see, especially later on YouTube, because you know, we'll all be on a small phone.
I'm gonna have to figure something out. Okay. So the other thing about this project was, since I this was like, slowly becoming a or thought I was thinking was going to be a larger project from the beginning. I also wanted it because I'm a teacher to try to like do this in a way where this becomes like a big teaching example. So I think if you go to the repository, which was in my big giant link dump, I believe there is. Yeah, so there's this like initial analysis.r file. And I would say this file was created as I was sort of vibe coding along with Claude just to be like, hey, just help me load some data sets. Also, I was using Claude because I don't actually make maps very often. I know enough about GIS techniques. But the actual function in the SF package, I haven't been working with that in such a really long time.
So this was the initial analysis of like, oh, let's just go and load up this giant trees geo JSON file. They also had it in the shape file. But I think like I just wanted it as a single file, which is why I op and luckily the open data portal had this format, which was just a geo JSON file. So it's just one file. So it was easier to also share as well. And then the first thing I did was like, okay, go look for they actually had this data set called common name. And then it was like, okay, go look for Japanese cherry blossoms. And then I very quickly realized that is not enough because there are so many I think I can actually run the code.
I would love that. That was my first question was like, wait, when I was looking at the open data portal, it was just like scientific names and tree names and none of them said cherry.
Yeah. Oh, no. I have a Oh, no, you guys. I didn't light my candle. This is the problem. All right. Hold on. Hold on. We can do this. We can do this.
Where am I? Okay. All right. There we go. Oh, no. Oh, wait. I know what happened. There we go. Public, public, public trees. There we go. Okay. Yeah, I this was the original path. This was like at the original like vibe coded thing. And I've afterwards made it a lot. Like I formatted the data set, the folder structure, you know, to make it look prettier. So I never went back to edit this file after the initial pass. But there's a column called common name. And then I very quickly realized that like, there are a lot of things that are Japanese something there's like flower flowering ash and then, like, I clearly in very quickly realized I'm way over my head because I don't know what classifies as a cherry blossom or I realized that there were a lot of other things that were blooming in this time that wasn't exactly Japanese cherry blossom like that is like a species, one species of flowering cherries that are planted in the city, but that wasn't the only one.
So how many are there? So So here's the thing. So you can see like, there's two regex patterns. One was look for the word cherry, and then look for like a sub thing of Japanese and flowering or flower because there are like something flowering cherries like those are those are different species that were listed or Japanese flowering cherries like one but then there's like Japanese cherry trees, and then like something flowering cherries. So that's like my sort of quick process over what was happening. And I think there's I want to say I don't remember the unique number of common names, but we can totally unique.
I was just gonna say, yeah, give us a little distinct. I'm shocked that that all of the planted trees are tracked. Like, I don't know if my city tracks anything.
Super interesting thing was the city tracked all their planted trees. I don't know how often it's updated. But it's like, having this information was good enough already. Which is, you know, fund your city government. That's one of the things I also learned about this project was the city does a really good job with this stuff. For time's sake, I can sort of go through what the process is. So what I ended up finding was like 10 different tree species that had some combination of flower, cherry and Jap. And this is what the plot looks like on just like using your normal GIS techniques. And I think I can make this bigger for the internet. So these are all the trees that I'm interested in. Again, this is also based off of my very naive understanding of all of this. And then, you know, here's our starting point. How do we get from here to let's go find where in the city to go look, right? And you can visually be like, hey, there's a big cluster of points there. If you don't live in a city, that might not be very obvious. But this right here is the Inland Park. This is actually where the Japanese cherry tree fest the cherry blossom festival is actually held right here in the city. So it kind of makes sense. Like there's a big giant cluster. And this species is actually Japanese flowering cherry. The other thing that was really useful, or the reason why I want to do this in cloud was I'm also very bad at plotting, especially plotting maps. And so it was sort of like I knew kind of how to think about this problem from a data analytics perspective, but very far removed, like rusty in the actual coding. So it was like, I can kind of audit the code, but not really write it, like off the top of my head, or at least not be able to write it within like a weekend. This definitely would have taken much longer without, you know, this, these AI coding assists.
This definitely would have taken much longer without, you know, this, these AI coding assists.
Mapping by neighborhood density
So if we go through this thought process of, hey, I want to just go somewhere to go take photos of cherry blossoms, you have all the points. The next thing you I thought about was, hey, let's go and we have neighborhood boundaries. Let's just map them by neighborhood. And so this is what we have. If we do it by density, originally it was make the thing by actual counts. But if you have sort of like a data science, data thinking mind thought process of answering the question, you kind of always want to normalize it by density. And so here's like the density, like I sort of, and then I also realized like, hey, I live in this neighborhood. Isn't that great? I live in the highest density cherry blossom trees part in the city. But then, you know, as a teaching moment, especially for my undergrad students, it's like, well, if you look at downtown, it's actually one of the lower locations, even though the festival is held here. Right. And it's because look how big downtown is, but all the trees are in one location. So that's another sort of like teaching process of like, no, like when we give you open ended questions, like, you know, we're trying to get you to think through the problems, not just give you a checklist of things to work on, which is. Yeah. Question everything. Yeah.
Okay. So from there, the next question was, okay, I don't care about the neighborhood because that kind of gets me in the right area, but I still don't know where to go. So then the thought process is okay. So we, and everyone walks around the city because they're taking photos of cherry blossoms. So you need to know where the blossoms are. So that's the point. But you're going to
specific locations. And if you've anyone seen like some of the Vancouver social media posts around cherry blossom season, like we have certain streets have like two rows, like both sides of the street. So all the sidewalks are just lines of cherry blossoms. So when you're driving or walking, it's just like everything's pink and it's really cool. And so that thought process is, well, you're not really looking at neighborhood density. You actually care about individual street density. So the city government also has street segment data. And then we, I learned later on that it's probably better to use open street maps, but the city had individual street data already. So I was able to load that up as its own geo JSON file. And then I have street or line data. I have point data. I should be able to calculate density in that sense.
And then, so I want to go through one of the actual code examples.
Sweet. Okay. And as a time check, you've got about 12 minutes left. Yeah. Cool. I think I'm good. Although why is my code not showing? That's not good. Ooh, we think we're good.
Right. Are my candles still lit? They are. Okay.
All right. We're just going to restart. It's okay. Is this open? Is it too zoomed in? It's too zoomed in. Okay. So one of the things, well, so this was what I ended up asking Claude, which was I have points and lines. How do I get the points to the nearest line, right? Because sometimes the trees are not always planted on a street and on a GIS perspective is definitely not always going to be on the actual line. Or if you're in a park, the trees are pointed or planted pretty far from an actual street. So I need to get to the closest street. So I totally forgot, but there is a built in function called ST nearest feature. And so what this is allowing you to do is given these points, like, can you snap the points to the nearest line? Is this in SF? This is in SF. This is like just built in. SF is so robust. It's wild.
So because I am trying to do this and I really want to not make mistakes, I don't remember how this function worked. And so what was also nice, and again, as a good teaching moment was I made another example script where I tried to mock up, like, this exact problem because I don't remember how snap to nearest feature worked. So what I ended up doing was here's some mocked up data, and this was all kind of semi-vibe coded as well, which was like, hey, go mock up some lines, treat them as streets, randomly place some points, pretend those are trees, and then run the ST, snap to nearest, and then see what's happening. And then I also had it to, like, take point, draw a dotted line to what it actually got snapped on.
And again, this was, like, my visual confirmation of, like, is this doing the right thing? Because I don't really remember how all this GIS stuff works other than I know it's possible. And so after seeing this, this was, like, my visual check of, like, okay, snap to nearest is the right thing. Because that's so the other thing I would like to point out is, like, this calculation or this transformation of data is, like, fundamental to how the entire analysis works, right? Like, if the point is not snapped to the road correctly, and I'm trying to calculate road density, none of this will work. So there is, like, confirm for yourself that this thing is doing right.
Yeah, or it might work, but, like, not doing what you think it's doing. I think this is a really great example of, like, why I tell people it's so important to still know how to code and still know how to verify that things are right, even when you're vibe coding. Because if you can't, if you can't do this little, like, okay, I know I need to make a minimum viable example of this and make sure it runs, then, yeah, you are in grave danger of doing things that don't make sense.
if you can't do this little, like, okay, I know I need to make a minimum viable example of this and make sure it runs, then, yeah, you are in grave danger of doing things that don't make sense.
Yeah, and I would say, like, I was just scrolling down, because there's, like, this is 155 lines that was, like, even though it was vibe coded, but 155 lines was literally to confirm this one line in the actual analysis. This is real life, y'all.
Snapping trees to streets and calculating density
So that's what happened on that side. So then, okay, I have everything snapped to the nearest street, and that's great. And I was like, okay, now I can calculate my density. Now the other problem was I'm using the street data that's coming from the city, and I don't know what that looks like.
Meaning the streets in the city were already plotted by segments. And I was like, what is a segment? Like, is that, like, you know, one of those concrete pavers? Like, I have no idea what is a segment, a street segment. So I originally tried to plant, do that whole confirmation for the entire city, but I realized there's way too many street segments. My computer was running for, like, hours in the background. It still wasn't done. So I did, like, one street, which was I was thinking about, like, what is one of the longest streets in the city? And it's Granville Street.
And here I was able to be like, oh, this is actually what street segments look like. Each different color is, like, a different segment. So because my main worry was what if it treated this entire block as one block, and I have all of these cherry blossoms on it, and it'll throw it off, because I don't really need the density across the entire block. I'm looking for individual streets, because the block, like, one cross intersections into the other, that's kind of where you would go to go take photos, not the entire thing.
So I was really worried about that throwing off the analysis as well. But again, small little example, here's one street. I was like, okay, it's not perfect. It might not be every single intersection or whatever, but it's good enough in the sense that, like, it's not one giant block. So that gave me enough confidence to be like, okay, this one line is doing what I want it to do. Every tree is going to be snapped. And then I can go and calculate density over, like, 100 meters. And then you have the individual densities. And then it becomes a very tidyverse, simple thing where you're sorting by density, and then picking off the top, like, 10 or 30. And then there you go.
The other thing was, back to Kyle Walker's post around, hey, like, now that I have all of this stuff, I don't, I'm afraid to run this code. Actually, I should not be afraid. This should be totally reproducible, right? Always fear the life of code running. It might take a while, but we can, I can run it.
I have a question while you're waiting. From Anna, which was, which projection did you use to snap to nearest? The default from SF or the GeoJSON? It is the GeoJSON. And hold on, I have it right here. So, like, you see 4326, like, everywhere. And I don't remember off the top of my head what it is now. But I think it is essentially WGS 84. So the Google map one. But then it's also the view in, like, this area of Vancouver too. I don't have to worry about the curvature of the earth at this point, because we're just looking at small distances. But that's, like, another part of the projection system that you have to think about as well.
But look at that. Things are working. They are working. From, like, a project that was in existence two months ago.
It might take a while to run, but it is running. Okay. I think I can open this in one of these is the data tab variables. There it is. And one of these, one of these. Which one? Tree summary.
Yeah. So, we have each individual street segment. We have, like, and then the other thing I had to do as well is there's different types of street segments, because I realized that there were public, non-public, and then alleyways and, like, some of the bike trails. Those are all in different geo JSON files. So, I had to put them together, like, get the column names all mashed up and stuff like that. So, there's a bit of data cleaning involved there.
But for each different segment, yeah, total trees, because it's all geospatial, it knows the length of the line. So, like, the total length is, like, not something I had to calculate. It's from the actual line of the segment. And then you can do something like density. And essentially, from here, it becomes, like, sort by density, take the top whatever.
And then if we go back here, here's the top whatever, 30 points. I picked 30 because it was, like, I feel like that was, like, a lot to get all over the city. And then if it ended up being a very, like, if it ended up being close to, like, 50k routes, that's very bikeable. Or if you're an ultra runner, it's very runnable. And that became the starting point there. And then there's a lot of vibe coding to get, like, hey, like, put bounding boxes. Like, I already found all the top 30 spots. Like, make a transparent circle so people have a reference to, like, neighboring dense streets. And then I labeled them by, like, the rank order. And then this was the information that got popped in for Kyle Walker's, like, algorithm. It didn't work very well.
Routing with Google Maps URLs
Like, this is what I was able to do without doing the street level data. Because what ended up happening was I think there's, like, holes in the segments for the street data in the city. So, like, it couldn't route properly when you're, like, why couldn't it just cross the street? And it's because there was probably a hole in the intersection. Because they're all individual segments. So, like, the routing algorithm, like, went crazy.
But this was another thing that I learned through vibe coding was, like, okay, I had this routing problem now. And then Claude was actually able to be, like, hey, you can actually put coordinates in, like, a really special URL. And you can just dump it in Google Maps. What? So, this is just a very fancy, like, put a question mark, like, whatever, into Google Maps. And then you get this, like, it maxes out at 10. But this was, like, oh, now you can just do this in Google Maps. And that started the process of, like, this would be a really great blog post to share with people in the city. Because Google Maps is a very accessible medium for, like, handling all of this. And, like, Google Maps has, like, this, like, reorder thing that you can do as well. So, that was the start of all of that.
And then I did the same thing for the 30-point route. Because Google Maps only has, like, groups of 10. So, like, this became, like, sets of three. But, again, this became, like, a vibe coded function. You're just piping things in. And it's just spitting out special URLs that get put into the site.
This is so cool. Well, we have, like, this is, like, two minutes left. But for a sort of follow-up, I know that you posted this on the Vancouver Reddit. You were on the local news. And you did all of this stuff. There's so many teachable moments in here. Question one, are you using a lot of these as examples in your classes? Because I'm, like, this is so juicy and amazing. And, also, have you thought about making a Shiny app out of it?
I have thought about a Shiny app. I haven't taught a class since I did this to sort of make it a teaching example. But that's sort of, like, on the way. And so, Stephanie Hicks, actually, she's one of the founders of Our Lady Baltimore. She was here in the stats department giving a guest lecture. I had dinner with her last night. She's working on this project called Analytic Design Theory, which was, like, because of all these LLM tools, like, we still have to be able to think through the problem. And, like, she's working on, like, teaching how to do the analytical thinking. Because coding is now, like, you don't really have to worry too much about the actual code. But what was always the problem and what's a skill that's always been, that will never go away, is, like, thinking about the problem. And how do you think?
And, you know, right now it's, like, okay, now we have to change data science a little bit around, like, we have to teach you how to think more versus, like, going through these analytics problems. Because I think what ended up happening was, like, back in the day, you learn how to think this way by struggling through the code. But now, since you're not struggling through the code anymore, you're losing that cycle of, like, iteration. And so now we really have to rely on this, like, this more basic thought of, like, teaching people how to think and learn in our data science courses.
right now it's, like, okay, now we have to change data science a little bit around, like, we have to teach you how to think more versus, like, going through these analytics problems. Because I think what ended up happening was, like, back in the day, you learn how to think this way by struggling through the code. But now, since you're not struggling through the code anymore, you're losing that cycle of, like, iteration.
Amazing. Well, my timer just went off for Dan. This is so exciting. I feel like I should reach out to Stephanie. This is the whole reason why I wrote my blog post about my Ellsworth art app. Because I was like, people need to see the process, the thought process that goes into things, you know, that has nothing to do with the code. This has been so fun. Dan, there are a million links in the chat, everybody. Round of applause for Dan. Thank you so much for showing that to us. We look forward to maybe perhaps a future Shiny app that we can all play around with.
I'm going to go play around with the Google URL building for routes and stuff, because that's super fun. And now we are going to transition over to chatting with Kylie Eensley. Kylie, how's it going? You ready for me to start my 25-minute timer? Okay. Please start. All right. You have 25 minutes on the clock. Let's go. And, you know, the top of the hour, I guess, is really our limit for you, Kylie.
Introduction to Kylie's project
Yeah, well, I actually think it's great where we ended with Dan, like that whole process of thinking, because my example is going to be much more about thinking and actually letting AI help me do the heavy lifting. But me having a very clear idea of what I want and how. help me do the heavy lifting, but me having a very clear idea of of what I want and how it works and then what I've sort of learned along the way. So I'll just start by a little storytelling of how I came up with this project. So I basically was stuck in the dark over winter and was just kind of like how do you know what you don't know? So like there's all these incredible tools out there that people have built. And I'm a confident programmer, but I'm by no means
a like software developer type person. And so I always assume that there's a tool out there that someone has built that's better than what I could build myself. But how do I find it if I don't know what it's called? How do I actually discover it if I haven't heard of it before? And so that started the wheels ticking. And I thought, well, what if you could search for something by what it does and not what it's called? And so that led me to what I call the warehouse.
See if I can share my screen now that I've lost it. Oh, there it is. Okay. So let me switch. Oh, I gotta move this. Let me just switch tabs. So basically, I built this thing called the warehouse. And I started with the idea of, okay, I just want like a Quarto website, because I've used Quarto websites before. They're familiar, I like them, they're flexible. And I was like, I just want to start there. And I was like, what if it's simple? What if it just looks like Google, but it's for R packages. So I just had this, this idea in my mind, and it kind of grew from there. So I could do the Quarto web design, I knew how to do it. But then when it came to there's a there's a there's several pieces to this project. So first, you have to search something. So what do you search? So the natural first step is all CRAN. But I know for a fact that there's some very useful packages in my personal research area that are not on CRAN may never be on CRAN. And I still want to be able to discover those. So it's like it has to be broader than CRAN. And so I thought, well, what if we just go directly to, to GitHub? And, and then that starts, okay, well, so now I have to scrape things. And then I have to be able to search things. And so it led me down this whole rabbit hole of just how to actually pull off something like this, because it's more than just here's a website, there's a whole bunch of stuff in the back end. And so basically, I've come up with some visualizations to try to describe kind of what it does.
So I sort of figured out that I was like, I don't need to reinvent the wheel, right? There's things like the R universe that are fantastic. And they've curated all of these different packages. And if you're not familiar with the R universe, highly recommend it. And you can search for lots of stuff. But typically, the problem with this is that it needs to match a keyword. So let's say you wanted to search something like mixed models, that's very nonspecific, and would probably miss out on some useful packages. But I was like, I don't need to recreate our universe. So I just they have an API. So I just grabbed packages from our universe. And I also are weekly shares a lot of new packages. So I grabbed stuff from there. And like GitHub, bioconductor, I just everything I could think of. And I basically created this, this JSON file database, because JSON is very flat. So it doesn't take up a lot of space, which I also learned what a JSON file was. And so in the repo, there's this giant file full of 23,000 packages. And that's where we start. So it's like I have this JSON file. Oh, I'm 23,000. So many. Yeah.
Sometimes I forget how many R packages there are. So so there's, so there's a ton. And so it was like, this was actually a really interesting project, because it I really pushed myself to try to do something very different. I live in sort of data, data analysis, mathematical modeling world. And this is very, very different. But it was really interesting to learn all the different pieces, but have I used cloud, cloud code, but to have someone to help me actually implement it. But yet I was very strict about this is what I want, make sure that it works. Am I getting the results that I expect? Now I can't find my zoom controls. This is fun. I was gonna try to share what a JSON file looks like. But I can't figure out how to switch.
At the top of your screen, if you like hover your mouse up there, they come back wherever you hit them.
Okay, I'm just gonna, just gonna hover.
Zoom is the worst, because it puts stuff on your screen. And you're like, get out of here. And then you hide it. And it's like, Okay, I'm never coming back. You hit escape.
Oh, I got him. Thanks, Libby. Yeah.
you know, I'm just gonna, we're just gonna go with it. You're gonna see my whole desktop.
It's okay, we won't tend to you.
Okay, so this is an example of a JSON file. And it's a lot of it's empty, it kind of depended on what information was available. But you basically, it's just a very flat file. So you've just got some kind of ID, you've got the package name, you have the package title, you have the description. And then you may have more information, like a version and maintainer, I think it depends on where it comes from.
So like, for example, DuckDB has all the information. And a bunch of extra stuff, like what it exports topics, things like that. So I think if it comes from our universe, it has all this information. It even has a score to kind of give you an idea of, I guess, how good the package is.
But anyway, so just imagine this over and over and over and over and over again. So this is basically where we start. It's like I've just, the first task was just get all the packages and find them wherever I could. And then the next challenge becomes, okay, how do we search it?
This is so interesting to me, Kylie. And I'll come back to it. But like, seeing that what you're picking up is just what's available, like from somebody who actually wrote the description. When I was testing your website, some packages, I just could not find. And now that tells me why I could not find them, because maybe their description field wasn't very descriptive, or their title wasn't very descriptive of what the package did. And in my own brain, I'm like, okay, prioritize that for your packages.
Yeah, exactly. Because the more information you give, the better people are able to find it. And I know from previous packages that I've made, like, I've just been like, oh, yeah, whatever description, like something, you know, nondescript and just kind of general. And it's like, oh, well, actually, that's not very useful to someone who's trying to find it if they want to use it.
How the search works
So basically, do I have a... Yeah, so the next problem was, how do we search it? And I started by, okay, let me come up with a search algorithm, and then quickly figured out, okay, I'm not a Google software engineer, like search algorithms are, I don't know anything about them. They're not my thing. And this is where I had the idea, what if I use AI? What if I use Claude to help me search?
So the way that it does it is that you pass whatever you want to do. So in this example, on the screen, it's survival analysis. And then you say, okay, Claude, you know, there's a prompt that feeds to Claude that says, like, you're an expert R programmer, and you want to do whatever you've put into the search strategy, come up with suggested packages and search terms. So Claude then creates, like, a list of 20, 30 terms, and then passes those back. And then you use something called BM25 to search that giant JSON file based on those terms, because Claude can do semantic searching. So it'll come up with terms that aren't necessarily directly related to what you've put in, or like, they don't mirror the exact words that you use, but they're related. So like, if we go back to that mixed model, you would probably want to find the LME4 package, but you're never going to find that if all you search is mixed model.
So and for those who don't know, because I also didn't know what BM25 was, I have included some information in the Git repo readme file, just as sort of a, oh, I learned all this stuff, let me help you learn with me. And so it's just a way of scoring search results. And basically, like, it looks at how often terms are used in a particular file, and weights those terms accordingly. So if it's a very rare term, like pharmacokinetics or something, it's going to weight that highly, whereas the is very frequently used in really any document. So that's going to be weighted quite low. And the basic idea is just because terms are repeated a bunch of times doesn't mean that that's many, many more times relevant to what you're looking for.
Okay, so then basically, at the end, what happens is you've got all these search terms, you rank them based on this BM25 algorithm. And then you spit out kind of the the best results. And that's, that's the idea. And if for some reason, AI, the API fails, or AI fails, or whatever, there is a just a local fuzzy search that just kind of corrects for possible spelling mistakes. And that way, you could find something that has the words that you used in the title, or the description, but it's just not going to be as good. And then, of course, if you use the actual package name, it'll just skip all this like ggplot2, you'll just go to the end, right? And get ggplot2. So yeah, I guess we can we can give it a try.
ggplot2 is definitely what I tested. So I tested, I was like, can it find the giraffe package, which I call ggiraffe in my own brain and heart. And so at the first search term, I gave it was like add interactivity to a plot, and it did not find ggiraffe at all. And I was like, why is this? I wonder why. And it's because the word plot is buried in ggplot2. And I'm wondering if maybe like if it doesn't decompose a bunch of words into their sort of, you know, their basic structures, then maybe it's not finding that. And I think that that's possibly what it is. Because if I change my search query to add interactivity to a ggplot2, then I do find ggiraffe like immediately. Oh, nice. Okay, so add interactive.
There it is. There it is. Yeah. But if I just had add interactivity to a plot, it did not show up. I love that. Like, I love search algorithms. I love the like sort of process of figuring out how to break things down to make them searchable.
It's such a big juicy problem. It's, it's really hard to find a way to do that. It's really hard. One of the things about this project is that I learned so much about searching. And it wasn't what I expected. I just was like, I'm just going to build this cool little tool that helps me find what I want. But actually finding what you want is the hard part. And so I have had to go back and forth. And there are times when things break. And it's just been iteration on iteration on iteration of how do I get? How do I find what I want? And actually, when I first came out with this, I posted on LinkedIn, and I was like, please try it. And if you don't get the results you expect, let me know. Because then I can try to improve it. So like Libby with yours, maybe there's something I can do to improve it.
One of the things about this project is that I learned so much about searching. I just was like, I'm just going to build this cool little tool that helps me find what I want. But actually finding what you want is the hard part.
Browsing by category and package reviews
Or I do have some um, actually, I could show you. So I have these categories, because I thought, Oh, maybe you just want to browse some packages that do certain things, right? So maybe you just want to look at data wrangling, and find packages that do it. And so you can actually just search a broad category of packages that do different things. So that's, that's also possible. I've got lots of fun stuff in here.
But searching is not the only feature of the warehouse, because I noticed there are reviews on there. Oh, yeah. So the other thing that I thought about was that, and not many people have done them. And I think a lot of them are me making Claude do it for me to test out the feature. But basically, I thought, well, what if you could, what if you could review a package like, like a Google review, like there are GitHub stars and things, but not everybody uses GitHub so much. And so this is a way of being able to tell people how you used something and what was good, what was bad. And there are questions on in the reviews that are designed for the user, but also for the developer, so that it can give you give feedback for new potential users. But also if there are there are features that were really good, or didn't work so well, the developer can then take that feedback on board and, and update.
Yeah, so you can actually there's a little star up in the top right corner where it says write a review. And here's just a, we don't have to go through
a review. But these are basically the questions that it asks. So it's what was the name? Who are you? And what was your experience level to figure out like, oh, if you're a beginner, maybe you're struggling with something that was designed for more advanced users, but could, you know, be usable to a beginner with increased documentation? And then what did you use it for? And whether or not it did what you wanted?
And there's other things for developers like document quality, where the error message is helpful. Because that's always, that's always a tricky one. Or were there any bugs, things like that. And so again, this, this feature hasn't been widely used. But I really hope that people use it more, because it would be really helpful as both a developer and a user to get this kind of feedback. I think so too. Well, it's up to all of us to go post about the warehouse, share about it with people that you talk to share it with your students so that they can more easily find packages. I know that I've been in this route before. And I've had learners be in this route before. How many wheels has everybody reinvented? Like everybody in the chat? Let me know how many times have you like built something. And then you go and show it to somebody. And they're like, dude, why didn't you just use blah, blah, blah package? And you're like, what? And they're like, yeah, I like does that way easier than what you did. Exactly. So many times. Exactly. Gosh, I feel like a dummy. But how would I have known? And then when I find out about the package, the package name is something completely random that has nothing to do with what the package does. And I'm like, so how would I have ever known? Right?
the package name is something completely random that has nothing to do with what the package does. And I'm like, so how would I have ever known? Right?
Exactly. So that's, that's what this is designed to tell you. But then, okay, if we go back to our add interactivity to a ggplot, right? I mean, even dplyr. I remember when I first heard about dplyr, I was like, pliers, why?
AI chatbot feature for package recommendations
So the other feature that I thought was, okay, but here are our results, right? There's, there's a bunch of packages. I don't know which one to use. And I thought, well, what if you could just ask AI? So there's a little chatbot that you could just say, which package should I use?
I didn't realize that was there. Like when I was testing your website. Also, has anybody hopped on the posit.co website lately and been like surprised that a chatbot popped up? Because Noor mentioned last week in the chat that that happened. And I was like, wait, what? And then I opened it and I was like, whoa, I've never seen this before.
Okay. So did it come up with some pretty good ones? Yeah. So basically it's just, yeah, it just says like, there's some options and it tells you, you know, this is the most popular, here's how you can grab it. What it's best for some tradeoffs. It did grab the three top ones that I think are probably recommended. That's awesome. Yeah. And so this can really help you if you're just like, okay, I've got these results, but which one do I use? And it just gives you just a little bit of starter code. Because again, I thought there's so many tools, like we're talking about 23 plus thousand that it's quite overwhelming if you're coming to a new tool and you're not quite sure which one to use or which one is best for your use case. I love it. Share this with a beginner today, y'all.
Yeah. So I was really thinking about beginners, but I was also like, I'm lazy and I want to find what I want to use easier. Ooh, Isabella had a question. Isabella, you want to ask that? Oh, sure. I was wondering about that chatbot. Is that part of Shiny? Is it thing external?
It's external. Cool. Yeah. And it feeds into... So basically my whole plan when I started this was to just host this completely on GitHub. But because the chatbot uses the cloud API and interacts with it and that you can't have on GitHub, which I think is why things like our universe don't have this kind of interactive chat body AI integration. Because if our universe could just integrate it, then that would basically solve my problem. But you need a hosting platform to do that. You need some additional features. So I had to go down a rabbit hole to find that out. So I pay for it. So I host this. It's not very expensive, but I'm just like, well, I'm going to pay for it because I think it's cool. I do the same thing. I'll be like, this is fine. I'll just pay this $5 a month for this little thing. I want to play with it. Everybody go use Kylie's tool so that it's worth the money that she's spending every month.
Q&A: quality scores, costs, and search details
Oh, and we had another question. Anna asked... Anna, I hope your name is Anna or Anna. Let me know. Why do some packages have numbers in red at the top right-hand corner and others don't? Yeah. Great question. So this comes from the R universe. So they have what's called a quality score and they base it on certain metrics. And so if there's a quality score from our universe, I just go ahead and post it. Because again, our universe has a ton of amazing stuff. They just didn't have that particular search feature that I was looking for. So I basically had to just... I scrape all their information and then just add my own search feature. And so it just tells you how... I think it's probably has something to do with like how popular it is and other like maybe how good the documentation is, whether it's frequently maintained, those kinds of things. Is it like out of 100 is red like a certain scale and other numbers are green? I don't think so. I don't know the answer to that. I might be asking a question that doesn't have an answer.
I've definitely looked into this, but I just forgot what... Maybe we'll find out together. We do have two more questions too. Okay. While I'm doing this, fire away. One is from Andrew. Are there limits on the cloud calls so that the cost doesn't skyrocket for you if we all give it the hug of death? Well, I have a finite pot of money. So when that runs out, you can't. I'm also using a very simple model. So I'm using their Haiku, like the cheapest model. Because all I really want is for it to come up with the terms so that it... Yes, it doesn't need to do the actual searching. So it doesn't actually need to go and look through the whole database. It just comes up with the terms for me. Well, that was the other question from Victor, because Victor said, very cool project. Can you share a few more details about every search is handled by a Netlify serverless function about that part of it? I host on Netlify too.
Yeah. So I've gone back and forth with... So here's my search function if you want to see it. But basically, because I realized that it would take way too long and be very expensive if I let AI do all the searching. So I just come up with term lists and do some other things like having to deal with common nouns. And then I also set the different BM25 parameters. And then it just passes those search terms back over the dataset.
Looking at this code, I'm wondering if ggiraffes, the fact that ggplot2 is in quotation marks, might kick it out of being tokenized down to the word plot. I'm curious. This is totally outside the scope of this conversation because we have two minutes left. I'll message you later.
Wrapping up
This was so much fun. My alarm for you just went off. Oh, perfect. But we have two minutes left. Did I cut you off from anything? I interjected too many times. No, no, no. I hadn't planned to go super into detail just because there's a lot under the hood here. There is, but everybody can go check it out in your GitHub repo if they would like to. Yes. And they can play with the tool. And what's the best way to give you personal feedback? Is it in the Discord server, just tag Kylie?
I'll send you a DM. Yeah, you can contact me on Discord. You can create a comment in the GitHub repo. You can find me on LinkedIn. Really, any way you want.
Sweet. I am so excited that our timing worked out today. These projects were amazing. Thank you so much to Dan and to Kylie. This was a lot of fun. I am so glad that most of our live coding worked and our timelines worked out. Okay, everybody go check out Dan's blog post, share it with somebody. I cannot wait to have a Shiny app to play with as well. Go check out Kyle Walker's work because, wow, I mean, talk about prolific work in the R graphing space and the R mapping space. You can't find better resources than Kyle. And thank you, Kylie, for sharing your project. Everybody learned a lot more about our universe today. Crayon, descriptions, titles, why that's important. If you are a package author or maintainer, think about going and making your description field more descriptive of exactly what your package does and the problem it solves in plain language. And we will see you next time.