Transcript#
This transcript was generated automatically and may contain errors.
Welcome to The Test Set. Here we talk with some of the brightest thinkers and tinkerers in statistical analysis, scientific computing, and machine learning. Digging into what makes them tick, plus the insights, experiments, and OMG moments that shape the field.
In this episode, we talk with Leilani Battle, who likes things she can repeat, study, and refine. Whether that means honing her disc golf skills, tweaking a recipe, or deriving the perfect vegan buttermilk sourdough pancakes. This same experimental instinct runs through her research at the University of Washington. Leilani built her career straddling two very different research cultures, the computation behind databases and the human-centered, experimental world of visualization, an experience she describes as essentially getting a PhD in two disciplines. We talk about the magic that happens when those fields meet, how what loads first in a visualization can shape what people find in their data, and how our work now applies that same technical and human perspective to responsible data science. So I'm so excited for you to listen to Leilani.
Leilani, welcome to The Test Set. So Leilani Battle is the Robert E. Denning Endowed Associate Professor of Computer Science at University of Washington and co-director of the UW Interactive Data Lab, and also a member of the Database Group. And you do a lot of really interesting work. It seems like on the computational aspects of visualization, but also the cognitive human factor side of it. Why would we say speed up visualization and in what situations for people? And also more recently, it seems working a lot on responsible data science as well. So thanks. Thanks so much for coming on.
Disc golf origin story
So often I feel like I'm like, at the very end, we get to people's hobbies. But I hope it's okay that we flip it around because I think you have so many incredible, interesting hobbies. I thought maybe we could go hobby forward, which was first and foremost, I do want to note, you mentioned you do a lot of pickling and sourdough. But also, you said your most surprising hobby is probably disc golf. How did you, could you unpack for us your history with disc golf?
It's a good question. So long story short, my husband got me into disc golf, and he's been playing for a very long time, since high school, actually. But I didn't really have any interest in disc golf whatsoever until the pandemic, when we were all stuck inside, we couldn't necessarily go and do our more social hobbies. But I still wanted to kind of go out, hang out with my husband, just do something outside of my house at the time. And he had been suggesting disc golf to me for years. I think I tried it once in high school or college or something and was like, nope, no thanks. And then he managed to get me to do it again during COVID. Because he's like, we could go outside and be in nature and throw frisbees. And I was like, okay, cool. I'll do it again. And then that time something stuck, and I really fell in love with it.
Something I think that helped at the time. So I was an assistant professor at the University of Maryland College Park. And we had bought a house that was like a 15-20 minute walk from a very beginner friendly disc golf course. So I developed this habit of every weekend, going with my husband to this course and playing disc golf. And I just kept doing it. And then when we moved back to Washington, Washington has so many beautiful disc golf courses, most of them free to play actually too. I started checking out here because I wanted to be able to play with more women. Most of the people I'd see on course are men. And then I started playing tournaments in our local area. And that's really where I kind of shifted and really got obsessed with the sport.
Amateur, amateur division tournament. I could go on forever about disc golf. So there are mixed divisions, meaning you could be like any gender and play in those divisions. And then there are some gender and age protected divisions. So there are female specific amateur divisions and professional divisions, and then age protected divisions that start about every decade and eventually go about every five years as you get older.
Repeatable recipes and experimental cooking
So what I'll say is probably the most satisfying feeling for me when it comes to cooking and pickling any kind of food preparation is when I'm able to come up with a recipe or a combination that to me feels optimal. Like it's really easy to make. It tastes great and it's repeatable. Maybe it's a scientist to me. I love it when it's repeatable. If it's not repeatable to me, it's a terrible thing to cook because then it implies you can never do it again that well. So I think to me, these kinds of hobbies where even disc golf kind of falls into this, where I can just kind of do it over and over and over and hone the recipe, hone the skills.
Maybe it's a scientist to me. I love it when it's repeatable. If it's not repeatable to me, it's a terrible thing to cook because then it implies you can never do it again that well.
Yeah. And I can experiment with different techniques. I also like to do or call making base recipes where you can take kind of the core of that recipe and then modify it in different ways to get a, a different, like a fresh experience. Cause you can imagine if you ate the same food over and over, you might get bored. So having these kind of base recipes where you can mix things up, it's still repeatable, but gives you variety. Like I love things like that.
That's just practical. I like to save money too. I can't, can't risk throwing all my food away.
That's exactly the kind of thing I love. An example of a recipe that I worked on for a really long time and really enjoy doing, um, is like my buttermilk pancake recipe. So I started making buttermilk pancakes in graduate school and I got really good at this, but then I became a vegan, which kind of destroyed my buttermilk pancake recipe, at least the old version. And I had to rebuild it. But because I had all those experiences and what makes a good pancake, I figured out what makes a great vegan pancake as well.
Career path to visualization research
I'll try to maybe sprint through kind of my path, career path. Initially, when I was in high school, I was really into drawing and arts and I loved playing video games. So I had assumed that because I was passionate about those things like drawing and video games or art and video games, that I would end up there somewhere. That being said, I think the influences of, you know, interacting with that kind of technology, you know, like game consoles, my family, I always had a computer at home growing up, things like that. I think I was more open to kind of being in tech in general. And thankfully I realized right before applying to college, like universities, that, you know, there's more to life than being a game developer or an artist or a designer for games.
When I applied to the University of Washington, because I was, for me, Washington is home. I grew up in the state of Washington primarily, and the University of Washington would be considered kind of a home university for me. I knew they had a strong engineering program, including electrical engineering and computer science and engineering. I was directly admitted to the computer science and engineering program at UW because originally I wasn't even going to do computer science. I was going to do electrical engineering. And with that direct admission, I was like, oh, well, this is a pretty competitive program that's hard to get into. I can give it a try. And if I hate it, I could always do something else.
So I started going through the program, started taking the intro, competing courses, ended up loving them, did not see that happening. But I thought they were incredibly fun. You know, kudos to back then the instructors teaching the intros courses, because I'm at UW now as a faculty member. So kind of seeing the other side of it, it's been really fun to see, you know, everything that goes into designing a really good CS course, especially at the introductory level and how that had such a huge impact on my life.
My early years as an undergraduate student, I actually struggled to get good, like competitive internship offers. And my first summer after my freshman year, I didn't have any opportunities to do a tech internship. So I decided to apply to research internship programs. And I did an on-campus research internship program that summer. And that was my first introduction to computer science research. It was actually in Professor Rajesh Rao's lab here in the Allen School. And at the time, his lab was a bit more of like a traditional robotics lab. And I've mentioned this to him too. I'm really grateful for that experience, but I knew I was not a roboticist.
My next summer research experience was actually in databases, in the database group here with Magda Balazinska, who is our school's director. So she's, she's been in my life and in my career for a long time. And I really look up to her. But kind of seeing the database group, how it operates, the kinds of research projects they were working on, I started to develop an appreciation for databases. And the work that I contributed to in particular was a little bit more oriented in the human computer action direction. So thinking about how we can make people's lives easier when they're trying to use database systems. And I think that kind of set me up for graduate school.
I just felt like I didn't have enough time to explore what I wanted to learn in academia before leaving and joining the workforce. I felt like I got a decent sense of what to expect in industry. I did not get a decent sense of what, you know, what potential I had in academia. So I wanted to keep going.
Part of it was my parents not, so I'm the first person in my whole family to get a doctoral degree and including my extended family. So for them, this was really a foreign concept and it's, even when I had made the decision, it took some convincing of my parents that I wasn't like throwing my life away by not automatically taking six figures from Microsoft or Google or whoever, Amazon. And they were really worried, like, are we going to have to pay for this? We already paid to help you go to college. Like, are we going to have to pay again? Like just a lot of kind of educating my whole family on what it means to go to grad school in computer science and what the benefits are and what the value could be long-term, stuff like that.
I landed on the side of, I want to do this. I want to try. I want to see what possibilities are out there. The database group was incredibly helpful and supportive in helping me apply to graduate programs. And I ended up getting into MIT, which to me was really exciting. It also changed my parents' minds because they've heard of MIT.
It was at that point where they were like, okay. Some name brand school. I guess I get it now. I don't, no shade to my parents. I love them. It's just, they're kind of funny cause they don't really, they're not academics. So the way they see my job and academia is very different from the way I see my job in academia. But so then I went to grad school. I started working with Mike Stonebraker. He was my PhD advisor. I was in the MIT database group. Thankfully MIT also at the time and still does, it's even stronger now, but MIT had really great HCI researchers as well. And in fact, it was Rob Miller, who connected me with another professor at Tufts University, Remco Chang, who is a visualization researcher. And it was with Remco and Mike that I started, you know, charting out what I actually wanted to do for my dissertation and learning what it takes to be a researcher, not only in databases, but also in visualization.
Straddling databases and visualization
I love being in the database group and I feel like it was a strength as far as my technical ability and my strength as a researcher. I feel like I really needed that. But it's hard to say publish papers at visualization conferences when you're used to reading and learning about database papers and conferences. So I really had to find my way.
I would say a lot of people in the database community had thought that way when I started my PhD. And I think today there's much more of an appreciation for user centered database work. And I see more of it getting published at SIGMOD now than before. But on the whole, I would say database researchers do not focus on it. I think they can, if you spend time talking to them about it, they can understand the value, but they just don't prioritize it.
Absolutely. I think this is one of the most fun parts of my job. Because I love straddling that boundary. It was really difficult in the beginning, just kind of finding my way because you have to, I, I'm fond of saying I basically had to get a PhD in two disciplines to be able to talk to them. But now that I'm there, I really love talking about those research problems and understanding the strengths and weaknesses of both communities. Because when you put them together, they make magic happen. And one classic example that I love to point to that is kind of very easy, a very easy win for that intersection is, you know, Tableau and originally the Stanford Project Polaris. That was an example of how putting those two things together can make magic happen in a way that no one community could have possibly done that on their own.
Okay, cool. So what I love about this, because the Tableau paper, I think the original Tableau paper came out in like 2002. And at the time it was this research prototype called Polaris. And it was kind of funny too, when you read the paper, because the paper is, when you read the paper, the introduction is basically making the case for, hey, this community, this is why we should care about what databases do and how we can map our visualization primitives onto database primitives to automatically generate SQL queries to process our data. That's basically what that paper was bringing to the Viz community. It was saying, A, we should care about this. And B, our theory and database theory could actually be integrated to produce this really nice automated stack going top to bottom and bottom to top. And the grammar that kind of tied it all together is this, this grammar called VizQL.
Basically this mapping across the stacks, because you can think of it as a stack, right? You have your data processing and storage layers, which the database management community typically owns. And then you have the visualization and interaction and user interface design layers, which the visualization community typically owns. And that paper was saying, why, you know, bisect the problem in that way when you can actually do this more integrated approach. Nowadays people would look at that. And I even had PhD students after reading the paper, read that and be like, how is this groundbreaking? Everybody just does this. And I'm thinking, that's the point. That paper was basically saying, everybody should be doing this.
I do find it so fascinating, how we kind of build these boundaries around, this is the area in which I work, or this is the area in which we work. And then it becomes almost heretical to step outside of it or combine it with something else. Each community builds up its own kind of standards, its own vocabulary. And then when someone comes out from the outside, and they don't know how to like signal all of these, like, oh, I'm a serious whatever. I guess I never really understood why like interdisciplinary or cross disciplinary work was so hard, but it's not really about the work. It's about the people and the way they define themselves and like the communities.
Absolutely. Yeah, I think so. And lots of times a community defines itself by the problems it chooses to focus on. And that work and then also just my, I like to think the work that I focus on in my career is trying to break down some of those walls to reclaim some of what we lose by those, you know, those boundaries that we those artificial boundaries that we put up to kind of just define and protect our communities in some ways.
So as far as conferences I publish in, um, I will publish wherever I believe the work is a good fit, that could be an HCI conference, that could be a visualization conference, that could be an AI ethics conference, that could be a, you know, more programming languages oriented conference or a database conference. I think to me, the most important thing is that community cares about the work that we did.
If you're too diffuse in your portfolio, um, then people can't write those kinds of letters because they'll be like, who's that, Leilani, who's that? Or Leilani, I don't really know her work. So there's that danger, but if you can avoid that danger, I think it's better to focus more on where the work will be best received and have the most impact rather than worry about, you know, oh, I published all my papers at this.
That being said, tragically speaking, I can't jet set around the world all year. I have to be practical about what conferences I actually go to. I try to make sure when possible, try to make sure I go to at least one like Viz or HCI conference and one database conference a year. The past couple of years have been weird and I've been prioritizing sending my students to conferences instead.
Yes, I am deliberate. Uh, and I do code switch between communities because the things that they talk about, the terminology they use, the things they emphasize are different. That being said, with AI, things are blending a little more. We're sort of ending up in similar places more so than I've seen in the past because we have kind of similar goals we want to achieve, so I think things are blending more now, but I do have to code switch when I talk to different communities.
LLMs and the future of data visualization
Absolutely. I think we're already experiencing a massive shift in the way people do data science work with the introduction of large language models. One easy example I can point to is in chart generation and visualization recommendation, a lot of, a lot of my prior work has been in visualization recommendation and automated methods to support that. And now we always have to answer this question of, oh, well, how would LLM solve this problem or how would someone use an LLM to, you know, generate a chart or generate code to generate charts or even generate dashboards?
And also practically speaking, um, a lot of people I know, researchers, folks in industry, they'll just very quickly go to a model and ask it to make charts for them. So just on a practical level, people are using LLMs in a way where they would traditionally use maybe visuals and stuff like that, and also even tools like, uh, data science tools themselves are doing it as well, right? Just automatically using, providing AI recommendations. So I think huge, huge change.
I can spotlight one ongoing project in our lab around this because we are curious, you know, what this shift looks like. So Will Wong, a PhD student in our lab, who's co-advised by myself and Jeff here, he's investigating this problem by interviewing, um, statistical and, and data analysis consultants at universities. Because traditionally pre-LLM days, if you're a scientist and you weren't totally sure how to, you know, analyze and statistically test your data that you collected for an experiment or something, your university, there's a very good chance, like you could talk to the statisticians there, or there's even a consulting office you could go to. And then a consultant could work with you to figure out what questions you're trying to answer from your data, whether you collected the right data, what statistical models and tests make the most sense, how to process your data properly for it and so on. But you can imagine with LLMs, like statistical consulting must be seeing a humongous tectonic shift. So, uh, we are wrapping up an interview study with consultants to see how, like their perception of large language models and how it's affected their jobs. And it's been, it's been really interesting.
From different angles, we've been tackling this problem. So Will, he, the PhD student I mentioned, he also had a previous paper on evaluating the kinds of recommendations at the time that models were giving when you asked them to generate charts or to complete a partial visualization design. So if you had a partial design, you wanted the model to finish it, what would it recommend, or if you wanted the model to just recommend charts, you know, what, what do you get from it?
Um, at the time, so to be fair, this is a paper from 2024, it's 2026 now, models move fast, update quickly. At the time, his work found that what the models were recommending was mostly fine, but there are some cases where they would recommend weird stuff. Like they would recommend visual encodings that didn't really make sense for the context. And I think kind of the big takeaway for me was that sometimes the models were recommending designs that didn't seem to align with the data that we have in our community. So as science, as visualization researchers, as scientists, we've collected data. We've done tons of experiments seeing what kinds of business design decisions make sense, which ones don't make sense. And it just didn't seem like models, like what they were learning on was not that, it didn't seem like it was that, at least not most of the time. And for proprietary models, who knows what exactly they're learning their business recommendations from, but it seemed like the answer was not us sometimes anyway.
And for proprietary models, who knows what exactly they're learning their business recommendations from, but it seemed like the answer was not us sometimes anyway.
So kind of before, during, and now like this before and during the AI wave, I think we were already in some ways transitioning away from these kind of standalone Viz tools, at least in the specific areas I work in. We're sort of transitioning away from these standalone Viz tools and more towards like modules and things that can integrate into stuff like computational notebooks. And I personally feel like that's the future, having a standalone, unless you're providing an environment that other tools would integrate into. I don't think it's really to your advantage to have a standalone tool that you create as a Viz researcher, because then you have to argue to everybody why they should leave whatever environment they already use and use your tool instead.
You still see a lot of standalone tools, just practically speaking, cause it's easy for a grad student to make a research prototype that way than to build a, like Jupyter plugin or something, that being said with large language models, Jupyter plugins are way easier to make now than they used to be. So I think there's less of an excuse now and that integration, cause I don't, I think there's more of a question around computational notebooks years ago, like earlier in my career. Nowadays, it's like a no brainer computational notebook environment. Like why wouldn't we use it? So I think the Viz community has been forced to adopt it more.
Favorite visualization studies
So one funny takeaway from a study we did in 2022, I believe. That was the 2022 CHI paper. So, uh, this work was in collaboration with Michael Carell. He was at Tableau at the time, but now I believe he's at Northeastern. But the, it was around visualization recommendation. And our idea was to see, okay, how would people like automated recommendations versus like human curated recommendations and what arguments are people making in terms of why they like one versus the other.
So what we did was we picked up some, picked out some commercial tools like Tableau, Power BI and such, had them generate recommendations for visualizations of public health data. And this data was specifically a subset of data from NHANES. So the, um, it's basically a national health survey. We showed it to, um, students, professionals, and researchers in public health. And we showed them recommendations from, you know, automated tools and also from just people crafting recommendations and the participants of our study generally tended to prefer the human created recommendations because they felt that the human created recommendations told more of a story. There's more of a story behind why that recommendation was generated.
And at the time, the automated tools, they just, they generated pretty overly simplistic recommendations. You could tell they were just putting variables together and then generating a chart. And so that like is a, a fun, like favorite takeaway of mine. Of course, this is like pre-LLM. So maybe the automated tools would do better now. So we should redo that experiment.
And then one more study I'll highlight that was really fun. Actually, it was from the tail end of my dissertation work. Basically you wanted to see, okay, well with recommendations, there could be this issue of performance where let's say you're analyzing a large dataset. What if the thing people look at is the thing you can recommend the fastest? Basically the thing they see first. How could that potentially introduce manipulation of people? Like what, what conclusions they might draw or what findings they might find in their data? So what we wanted to do was design an abstract experiment to try to test this. Could you, in a sense, manipulate what people found in a visualization purely through latency?
So I designed this like really abstracted version of this experiment. I created this image collage and asked people to explore the collage of images. And then they were looking for certain, a certain target image. I actually put two targets, like two copies of that target in the collage. But the goal of the task was to only find one. And you can imagine kind of like Google Maps. Hey, you sometimes say you're on your phone and your data speed isn't that great and the map takes a while to load. Maybe it loads in kind of patchy. Imagine something kind of like that in this experiment, but it's an image collage so some images appear faster, slower than others.
So I designed the latency for the images such that it would look kind of random. But in fact, it was definitely not random. I was manipulating how quickly certain images would appear. And I was trying to make it such that I only use latency to try to incentivize people to find one target over another in the collage. And the, the experiments showed that if the latency was pretty low overall, like relatively low, it didn't affect people as much. And then, but gradually as you ratcheted up the latency, like if you made it the tile, the images load in slow enough, you could definitely manipulate people to find a certain target in the collage. So it was just showing how latency matters, how you design a system matters, but also how you recommend things matters. If you do it purely based on performance and don't pay attention to what, you know, what you're prioritizing, like if you're, if you have options to show A or B and you always make A show up faster, depending on how much faster A shows up, you might be incentivizing the user to just always look at A and never look at B.
Absolutely. So that to me was like a fun experiment that I enjoyed. I really enjoyed the design of the experiment. The takeaway was pretty straightforward in my opinion. I guess one thing that was surprising about it was people kind of thought of latency as a sort of binary thing, like, Oh, there's either too much or too little. But that experiment was showing it's actually kind of a gradual effect. And the more latency you have, the worse the effect becomes for people that get, become kind of biased.
Responsible data science
So responsible data science is this giant umbrella. Lots of people work in this space. At a high level response, the whole point of data responsible data science is the idea that we want the outcomes of data science work to be responsible. We want that people are not discriminated against. We want that people are not harmed by the outputs of say models, large language models and otherwise. We want people to be treated fairly.
So everybody who wants that and is working towards that, they fall under this responsibility science umbrella. What I think is really cool about the work that I'm doing, and this is in collaboration with a few professors who I want to shout out, so I want to shout out Emily Wall, kind of my longest standing collaborator on this work. Professor Amy Zhang, who Amy and I co-advised an awesome PhD student working in this space, Tiana Barrett. And then also Professor Yuichi Shoda. So Yuichi is a faculty member in psychology at UW and without Yuichi, we wouldn't be, I wouldn't be where I am now with these projects.
But the thing that always struck me about responsible data science work, where I think it's changing a bit more now, but years ago when Emily and I started talking about this problem, what struck us about it and what struck me about it especially is how it was always sort of treated as a algorithmic problem, a model training problem, or a data cleaning problem. The problem was always someone or something else. The problem was never us. And I was just thinking, wait a minute. So who does data science work? Is the data science worker the model? Is the data science worker the data or is it me and you? So if we're going to, you know, try to make data science work more responsible, should we be emphasizing the artifacts or should we be emphasizing the people doing the work?
The problem was always someone or something else. The problem was never us.
So what came out of that is this idea of behavior change interventions for responsible data science. So the idea that we should not just focus on whether or not a data set is responsible, a model's responsible. We should also be focusing on, you know, are the practices of the data scientists responsible and how does that contribute to ethical or unethical outcomes in data science? So that's the perspective that we brought to the community.
Because initially people weren't really thinking about the problem in this way. They weren't talking about it in this way. People are actually kind of skeptical too about it. It took us a bit to actually publish our very first paper in that space where it was kind of a mixture of a literature review and vision paper describing what behavior change would mean in the context of responsible data science and what kinds of tools in psychology and in computer science that we could use to actually achieve that goal of trying to help people shift their behaviors.
And then since then, we've had a couple, a few more papers on that, uh, line in that line of work where the most recent one is actually a paper at CHI, so top HCI conference, where we actually evaluated two existing behavior change intervention strategies in the context of a controlled, uh, data modeling task. So we asked people to train a machine learning model with some provided data that we gave them during the experiment. And then we, they experienced, uh, I think they experienced two different intervention strategies. So one was we primed them. So we had them read a little prompt that hopefully motivated them to be more responsible in their actions and training model. And then a separate intervention where we actually just loaded, um, like a modular plugin into the notebook they were using to train the model. And they could use that tool, that specific tool to check the fairness of their model and improve it.
A cool outcome of that work was that priming people to think responsibly did shift their behaviors. The, they did have an increase in the range of responsible activities they did during the session. But the fairness of their models did not actually get better from a significance perspective. We did not notice a significant improvement in the fairness of the models. So just motivating people to do better, basically just telling them to do better, and them feeling like they want to do better doesn't actually lead to anything getting better. So things change, but not necessarily in a measurably better way. So giving them tools to actually show them like this is what better looks like in terms of outcomes and pairing that with the motivation, that actually works.
It does run inside the notebook. It shows you some bias metrics and you can pick specific subgroups in your dataset that you want to compare against. For example, a dataset that we asked participants to train models with was sort of a classic like loan application datasets. So a demographic group you could compare would be like men versus women and their loan applications. So you could see if your model would be say biased against women and be more likely to reject their loan applications.
The prime is really just a prompt that encourages you to think like you could think about saying, hey, with the loan, the credit loan dataset, it's possible that women are more likely discriminated against than men and more likely to have their loans declined. So you could say something like that and like, oh, please keep this in mind when you're training your models.
People would say, oh yeah, I was definitely more motivated, like being reminded of that or, oh, as a woman, I can resonate, that resonates with me and I would not want people to be discriminated against in that way. It definitely motivated people. And we did notice a behavioral shift. But I think the important thing is just inducing a change in behavior does not guarantee the outcomes change. You have to induce the behavior in a deliberate direction.
Yeah. Giving them the capabilities to measure the fairness of their model. And then also to fix the model when it's not at the level that they want or if they want it to be more fair.
Yeah. So behavior change, behavior change for responsible data science is just, in my opinion, understudied. There should be way more people than the, me and the faculty I mentioned studying this problem. And there are some people studying this problem. So I'll acknowledge, like, this is not an empty bucket, but there should be way more people studying it. And I think part of the challenge in studying this problem is that we are not necessarily well-equipped with the tools that we need to tackle these kinds of behavioral problems, but there are other communities that have studied it for a long time, like psychology. Behavior change is a really important aspect of psychology research. And so what we are trying to introduce through these papers are strategies and tools that we could adopt where we maybe, we could even collaborate with researchers in psychology, like Yuichi's collaboration with us is an example of that, where we don't have to come up with all the answers as computer scientists. And in fact, we shouldn't, because a lot of us are not behavioral scientists.
African data ethics framework
Yeah. So I definitely want to acknowledge the intellectual leader of that work was Tiana Barrett. So she's a PhD student here at UW, co-advised by me and Amy Zhang. And so what's interesting to me about that work is this idea that, so, you know, in responsible data science, we care a lot about what it means to do something right and what it means to do something wrong, ethically speaking. And a lot of how various communities try to write that down are through frameworks, ethics frameworks. What we noticed is that in these ethics frameworks, you have them for a lot of different communities, but it seemed like African countries were being left out of that conversation of what, like, what is right and what is wrong in the context of interacting with, engaging with African peoples. And it seemed like we could develop a framework to reflect the values and the ethics of different African countries to fill that gap.
And Tiana worked really hard to take into account African, like Sub-Saharan African philosophy, ethics work, kind of thought leaders in terms of how AI should be incorporated into their cultures, things like that, and kind of putting it all together to review, okay, if we were to create one framework to exemplify these principles that should be upheld, what would that look like? And what gaps are we filling compared to other ethics frameworks? And I would say anyone who's really excited about it, please reach out to Tiana. She loves talking about her work and that paper.
Yeah, that's a great question. So one thing that really bothers me about ethics frameworks is that they're so hard to use in a practical sense. I could go out, read a bunch of books, learn all there is to know about these frameworks, but then how do I actually apply it to my data science work? And then also a more recent project that Tiana has been working on is this idea of, well, how exactly do data scientists operationalize their ethics and their values in their own work? And the answer is it's really dependent on the environment that they're in and the opportunities that they're afforded to express those values through their work. And some people haven't thought as deeply about it as others, but even people who think deeply about it, depending on where you work, you might need like outside projects to be able to do that, to really closely align your values with the practices in data science that you follow. Because your company might be hostile towards it. They might not care.
From a technology perspective, what can we and what can we not do to support that? Yeah. It's interesting to think about too, the, in terms of responsibility that I guess, if you're a data scientist within a company, the question of like, what is my responsibility? And if I don't have resources in the company to kind of like meet that responsibility, should I maybe do, do I need to put in work to kind of like build the skills to, to behave responsibly? Or even just how hard should I fight to try to make these resources available so that people can perform their work responsibly in the way that aligns with my values? Cause the organization has values and then the individual has values and there's this question of how they intersect.
Rationality and responsible practice
I was going to say one direction I'm really interested in moving in the future is this idea of, yeah, it might not necessarily be right versus wrong in this kind of more nuanced space. Something I think we're running into and that we see is that inaction might be the kind of the biggest problem, inaction along different dimensions, because it's pretty clear, right? That if data science, data scientists and engineers and so on do what they believe is rational or what they believe is like reasonable and are not actively doing actions where they're like, you know, I should be doing additional things to make this as responsible as possible versus, oh, I don't think I'm doing harm. So I'm, I think I'm fine. We still end up in all sorts of bad situations. So it's not that people aren't rational or people aren't reasonable. It's that the rational and reasonable, like rational behavior is not in the, in the current context, the definition of rationality might not be ethical. So if you define rational as responsible, you end up in a different frame, which might lead a totally separate line of decision-making that we come very different from what we do right now. So something I'm, Emily Wall and I are interested in following is this idea of rationality in data science and where it might get us into trouble and how other frames, like other definitions of rationality might help us kind of break out of that cycle.
Do you have advice for people maybe starting out in data science today or recommendations for them?
What I would, what I would say is you don't have to like read a bajillion books on it or anything, maybe pick one book, like Data Feminism and check it out. And you don't have to necessarily commit yourself to doing everything in that book, but just kind of, I think it's useful to know what's out there and then you yourself can decide how you want to incorporate it. And then I would say for each tool or feature that you're going to use to automate your work, see if there's another tool or feature that you can use to evaluate it from a responsible perspective. Like if you're going to train models, see if there's already tools out there that could help you check how fair they are. If you're data cleaning and you're going to do some automation, see if there's anything that can help check for like bias.
If I was starting at the very beginning, that's what I would do. I would think about, you know, almost like unit testing. If I already do the equivalent of that for any tool I want to add to my toolbox, you know, what checks and balances can I just automatically put in place as I add to the toolbox.
If you're a fan of like the good place, I think of it kind of like that. There are no perfect choices in life. You can only, you know, do better. And that's just such an antidote to like paralysis. Like, you know, you can't, you're never going to be perfect. You're never going to do the best thing you possibly could, but you can do a little bit better than what you're doing right now. And everyone can do that.
Exactly. It's so, it's relative. I think of responsibility in relative terms. There aren't good people and bad people.
Well, Leilani, thank you so much for coming on. I think you have so, so many interesting studies that span like computation, cognition, and ethics. It's been so nice having you on and hearing from your work at the intersection of so many interesting areas and really focusing on, or being sure to bring in also the human component into so much of it. And to think about the ethical side of it has been so inspiring to hear about. So yeah, really appreciate you coming on the test set. And thanks so much for talking.
Yeah. Thank you for having me. It's been a blast. I've loved talking about all the topics we covered today, especially responsible data science. I actually don't get that many opportunities to talk about it. It's not that popular of a topic in industry right now.