Transcript#

This transcript was generated automatically and may contain errors.

Mark. Thanks for having us. I would like to kick this off with a bit of a very unfair trick question of the audience. And here's what the setup looks like. Around a year ago, we were working on this agent for data analysis, and we wanted to make sure that it could see the plots that it was making. And so we asked it to make a plot and then tell us what it saw. And we said, could you please plot horsepower versus miles per gallon in empty cars? So like every time I make this plot, I have to like think through what the correlation is supposed to be. If I'm in a super high horsepower, like Ferrari or something, the miles per gallon will be low. And then the fuel efficiency of like Prius, which has low horsepower is super high. So there should be a negative correlation between those two.

So I asked the agent, can you make this plot? And the agent makes the plot, and then the plots returned. And then the agent says, there's a strong negative association. So the trick question goes like this. This agent is wrong. Why is it wrong? And this is because the plot doesn't quite look like we would have expected it to. For some reason, the association looks positive. So our Prius has very low miles per gallon. I wouldn't have expected to see this. On the internet, there are like thousands of examples that are working with this built-in empty cars dataset inside of R. And this is not what that plot looks like.

So what we were trying to do with this plot was to be able to tell whether the agent could actually see what was going on inside of the plot. And so to do that, we tried to trick it. We take the value of horsepower inside of that column, and we just invert it such that every time the agent makes a plot that has both that variable and some other variable, the association will be flipped from what it expects to see. So our thought was that if we pass this image along, and it's somehow corrupted, or we're not using the right format for the API, but the association looks like the model thought it was going to, then it's just going to say that it looks like it expected it. But if we flip it, then we can actually tell whether the agent can see the plot or not. And so for whatever reason, it can't see the plot. It still thinks that there's a strong negative association.

So we revisit the API documentation, and we switch around the fields again and again. And for some reason, it seems like the agent can just never see this plot. Out of exasperation, we're like, okay, let's just like try one other way of making a plot with base R plot instead of ggplot, and maybe that's going to help. So we say, run this code, only this code, and nothing but this code. And what this code does is it takes some number of points between three and 10, which is what's on this first line. And then on the second line, we take a random color by randomly assigning red, green, and blue. And so without knowledge of the random seed, the agent has no idea how many points there will be or what color there'll be.

So the first time the agent makes this plot, there happens to be 10 violet points. We go ahead and let the agent rip. And we tell it to run the code. It runs the code. There are three cyan points inside of the plot. And the agent says, there are three cyan points. What? That's weird. Like I would have thought one capability is being able to see the trend in a plot. Another capability is to be able to see both the trend and the color. So this is weird to me that the agent can see that there are three cyan points. And so this revealed something counterintuitive to us. It appears that LLMs and agents based on LLMs can see plots just fine.

So what's going on here? We looked a bit further into this. Sarah and I wrote this up in a blog post called When Plotting. LLMs see what they expect to see. And this is the subject of an eval called BuffBench. What we did in this eval is we take a number of these well-known data sets that LLMs have seen thousands of times during their training. And we do something to contradict their expectations whenever they make plots. There are a bunch of different plots they make. And the percentage of times that they can tell that something is amiss and the plot doesn't look like they think it will look, that's a correct answer. But at the time that we wrote this eval, which was 8 or 10 months ago, the best models were getting single-digit scores correct.

So models often report what they expect to see, not what's plotted. This is so weird. This is like so unlike humans. I think when we speak to things that seem to speak our language, like if English goes in, English comes out, we expect that the shape of the capabilities and the tendencies and the behaviors of the thing will be human-like. But this is so weird. If you put a plot in front of a human that shows a counterintuitive result, our reaction will probably most likely be to dig in further and try to figure out what's going on rather than brushing it aside.

So models often report what they expect to see, not what's plotted.

Trying to fix the eval score

So once we had this eval put together, this is really an issue because in data science we come across counterintuitive results all the time. And this is really the process of data science is learning about the world from data and contradicting our priors. So we're trying to figure out how to drive these scores up now that we have a baseline.

One thing that we tried to do is something called a model in the middle. So on the left-hand side, we have the main thread of the conversation. And on the right-hand side, we have what we call the model in the middle. So we say, can you make the plot? The agent makes the plot. And instead of returning the plot itself as an image, we return a description of the plot. And that description is generated by a separate conversation thread that has no other access to the conversation. And it will describe the plot faithfully. So this showed to us, again, we can rely on the fact that LLMs can see plots just fine, but they can't see plots when they contradict their expectations. So we don't introduce any expectations at all. All we say is, please describe this image. The model in the middle says, there's a strong positive association. And the agent in the main thread, with its priors activated, goes ahead and says, there's a strong negative association, entirely contradicting the evidence from the plot that it just made.

That's weird. That is very unlike a human.

Okay, so then we try something else. If we can't get the agent to say the real association, what if we make the agent say what the real association is? So we try a different version of the model in the middle, based on something called a prefill. We ask the agent to make a plot. It makes the plot. We return the plot. But we also do something called a prefill, where we tell the agent that it said something at the beginning of its response. And the thing that we tell it it said is the true association inside of the plot. So the agent receives the plot, and it's also told, I've already said there's a strong positive association. The agent continues that response by inserting a new line and then directly contradicting itself, saying there's a strong negative association between these two variables.

I don't know. This is so weird. This is so unlike a human collaborator. A bit exasperated, we tried one more thing. We told the agent that it had a private scratch pad, and that the user would not read the scratch pad, but that the agent could think through what it wanted to say to the user before it responded at all. So we say, could you please make the plot? And the agent makes the plot, and it doesn't look like it expects. And it says, huh, that's weird. The association is positive. And then it closes its private scratch pad and says that the association is negative.

Alien sighting.

So we tried a bunch of stuff to try to drive this eval score up, and we couldn't make it happen. Probably, as is usually the most effective way to make an eval score go up when you're working on AI agents, is to wait six months. And when we did that, we have seen, especially in the last few months, this huge jump in score on these evals. So again, at the time, the typical percent correct on this eval was something like 10%. And there were incremental improvements over that 10% over the course of the six months. But with the release of Opus 4.8 and 3.5 Flash, around a month ago or something like that, we started to see the first scores above 50%. And then with the brief release of Fable last week, we saw that score climb even higher.

So, so far, I have shown you that LLMs can see plots just fine. And yet, when those plots contradict their expectations, they seem to lose the ability to see those plots.

This is so weird. These, again, when we speak in our language to some being, and that being responds back in the same language, I think it's very easy to impart this expectation that the shape of this being is similar to a human.

And it turns out that they're actually very, very weird beings. They read 100 times faster and write 100 times faster, or probably more than that, than we can. And they also don't care about being correct.

It appears in these evals, what we're learning is that they are more concerned with making it appear as if a data analysis is progressing forward smoothly than they are about being correct. Sarah, I'll hand it off to you.

It appears in these evals, what we're learning is that they are more concerned with making it appear as if a data analysis is progressing forward smoothly than they are about being correct.

Why correctness matters

Okay. So, why is this a problem? I think probably all intuitively understand that correctness matters in data analysis. When you're working with real data, you have real world consequences, and being correct definitely matters. And correctness really is sort of a core value of data science. When you do an analysis, you want your results to actually line up with the world, or at least approximate it. And that's really sort of the whole point.

But LLM behavior in this eval, and in other ways, doesn't necessarily line up with this value of correctness. LLMs don't inherently care about correctness necessarily.

But they do put on a convincing performance, right? We've seen in this eval that the model will interpret the plot, and it will provide details and supposed evidence for what they see. And it looks pretty legitimate. It looks like if you don't inspect carefully, it looks like they've seen a plot, they're interpreting it, they sound reasonable, they provide evidence, and they act like they are moving this analysis forward. The only problem is that, like in this case, where there is a clear positive association that is described as negative, it's a convincing performance, but one that occasionally fails to match reality.

And the problem isn't just that the models are sometimes wrong. It's also that wrong answers can be indistinguishable from right ones. So here we have another plot from this eval. This is tree height versus tree circumference. And if you're just looking at this, you can clearly tell it's nonlinear, it's a parabola. And here we have three answers from the same model. And you don't really need to read these to get a sense of what they're saying. They all have kind of the same length, they have the same tone, they use sort of similar words, and they look the same at first glance.

But they actually have three very different descriptions in this plot. But if you didn't know what the plot looked like, and didn't have any sense of the data, you might read these descriptions and not be able to disentangle the correct ones from the incorrect ones.

You wouldn't be able to know that the middle one is correct, this is a parabolic relationship, and that those first two are incorrect. The text is relatively indistinguishable from each other, unless you have the plot.

Okay. So like I said, correctness matters, and it's not just a matter of reading the descriptions. It's also a matter of understanding the model.

But so do reproducibility and transparency. Reproducibility, you want to get the same results from the same conditions. And transparency, you want insight into how the analysis was done. These are all central ideas to data science, but are really often at odds with the nondeterministic and black box nature of LLM.

So what does this mean for data science if these values are often at odds with how LLMs behave? Can we still use agents for data analysis?

So in the first part, Simon introduced this eval and talked about how LLMs can behave in these like deeply weird ways that break our intuition about how the model is supposed to work and introduce new models.

And in the second half, I'm going to make the case that even though this is true, and it's not just true for this eval, but it's true in other ways, LLMs are still useful for data analysis. We don't need to throw the whole data into a bin.

Instead, the job is to figure out where they can fit and how they can be used.

We don't need to throw the whole endeavor away. We shouldn't take this kind of evidence to mean that LLMs are useless anywhere that you value correctness, reproducibility, and transparency.

Instead, the job is to figure out where they can fit and how to design around their limitations and for their strengths.

Introducing Posit Assistant

Okay, so like I said, I'm going to talk about how LLMs are still useful for data analysis.

And I'm going to talk about two things that we did for our agent, which I'll introduce in a little bit, to make it sort of as useful as we could for data analysis.

And so the first thing we're going to talk about is telling agents to care about being right, and then designing the scaffold, all the stuff that goes around the model that makes up the agent, so that it matters less if they do make an error and they are incorrect.

Okay, so we're going to be talking about agents. We already did a little bit, but just briefly, we say agent, we just generally mean a model hooked up in a way so that it can gather information about the world, like read files or see your environment, and then can also alter the world.

It has the ability to do things like run code or write code to files, move things around in a directory, any of that. So it can gather information about the world and then affect change in the world.

And specifically, I'm going to be talking about Posit Assistant. So this is one of Posit's agents for general purpose coding and data analysis, and it lives in the screenshot. It's in RStudio. You can also use it in Positron.

And we'll talk about this in a bit, but Posit Assistant can see your R or Python session, so it can run R or Python code right in your console. It can make plots that you can view in the chat pane and in the plots pane, and it can do both sort of iterative, exploratory analysis, and also longer running, more involved code.

Telling agents to care about being right

Okay. So I said we're going to talk about how LLMs are still useful for data analysis, even though they make sort of all the kinds of weird errors. And I was going to talk about how to tell them to care about being right when they don't inherently maybe care or prioritize correctness.

So an agent is more than just the model, right? It comes wrapped in a harness, and this includes things like its prompting, how it's told to act, what kinds of tasks it's told to work on, what tools it has access to. So, like, you know, broadly, what abilities does it have? Can it run code? What kind of code can it run? Can it read files? All of that.

So what abilities it has and what information it has access to. Like, is it hooked up in a way that it can see your R session? Can it, you know, gain access to a file system? All of that and more.

So this is what we control when we control Posit Assistant or really any agent. Unless you're the model builder or you're fine-tuning a model, you don't usually have control or you don't have control over the underlying LLM, but you do have control over the harness.

And so this is what we control when we, you know, adapt Posit Assistant. We have extensive prompting about prioritizing correctness, not just, you know, advancing analysis forward, but really making sure what is done is correct and sound, as well as things like statistical robustness, good modeling practices, all of that.

As well as tools that make a difference for correctness, like the ability to see your active session.

And it turns out that all of this, you know, work into the harness can make a difference. So even without affecting the underlying model.

So what this plot is showing is performance on that evaluation that Simon talked about in the beginning, the BluffBench eval.

And this is comparing Posit Assistant's performance. So the, you know, the agent with all of this extra stuff that we have put into it versus the minimal harness, which is the base case, which formed the plots that Simon showed earlier.

So generally, we find that Posit Assistant does better on this eval than the minimal harness. So all of this prompting and work that we have done into making Posit Assistant care about correctness seems to make a difference.

So all of this prompting and work that we have done into making Posit Assistant care about correctness seems to make a difference.

Designing the scaffold to handle errors

Okay, so we told the agent generally to care about being right, but likely LLMs, the agent, they're going to still make mistakes and do things that are against these values of data science.

So we also want to design the scaffold or the harness so that it matters less when they are wrong.

I'm going to talk about two things.

The first is sort of using code as a foundation. So we're going to have the model write code in a shared environment and And then also have the model pause often to involve the user in the analysis.

So let's talk about code as a foundation first.

And this might seem obvious, but it's not.

And then also have the model pause often to involve the user in the analysis.

Okay, so let's talk about code as a foundation first.

And this might seem obvious for a coding agent, but you don't actually need the agent to write code to do data analysis.

You could imagine an agent that you just give it like a CSV or some, you know, text file of your data, it ingests that and looks at it as text and then it spits out, you know, a summary of the data based on its, you know, read of the actual data file.

This is not great. It's not good for transparency. You don't know what was done to the data. You don't know how it was analyzed. It's definitely not reproducible. Every time you would give that, you know, text file of the data to the agent is probably going to give you different results.

And it is likely more error prone than writing code. Almost certainly.

Okay. And so this is probably not the right approach. I think code is a much better foundation for data analysis agents.

For a couple reasons. The first is that models are actually very good at writing code. This really is one of their strengths and we should take advantage of that.

Code, whether it's written by an agent or by a human can be run again. We know this. It's reproducible. It's one of the reasons it's so useful.

And it's also auditable, which means that it's transparent. You can look at it and you can know what was done to the data and how the analysis was done.

So code is a good choice.

And it's probably not enough just for the model to write code.

In many cases, you also want to show that code to the user so that the user can audit it and take a look at it when it matters.

So this is a screenshot of Posit Assistant.

Notice in the chat on the left hand side, it's written some code to make a plot. We don't always show the code in this way to the user, but when we think that they're going to look at it and it matters for them to know what was done, we do.

And so it's shown to the user, but we've also done all of these things to make it easy to see. So just small things like syntax highlighting, code styling, it's visible, you don't have to expand it. This might seem small, but it really helps you audit code when it matters.

Okay, and so we don't just want to write how the model write code. We also want it to share that code with the user.

A third sort of aspect of this code as foundation is operating in a shared environment. So what I mean by that is sharing the same R or Python session or whatever language your agent is writing code in with the user.

So here, the sort of circle and arrow are just pointing out that when Posit Assistant is operating here, it's using R. So it has access to your R session. If you create an object, like you created that DF object, Posit Assistant can access that same object.

If it creates an object, you can access it as well because you're operating in the same session. So when it runs code in the chat, it's actually running in the console. You can see that code, but you can also rerun it or build on it. So you're really seeing eye to eye with the agent because you're in the same session.

And this turns out to be very useful, especially for data analysis, where often a lot of the process is like turning the same object around, looking at it from multiple angles, transforming it, all of that. It's very useful for you and the agent to be working in the same environment. This lets you see eye to eye instead of occupying two different worlds.

It's very useful for you and the agent to be working in the same environment. This lets you see eye to eye instead of occupying two different worlds.

Having the model pause often

Okay. So we talked about having the agent write code, showing it to the user, and working in the same environment. The second piece of sort of catching the model if it is wrong and dealing with those consequences is having the model pause often.

In this GIF, you'll notice I just asked it to explore some data. It does a few turns of tool calling. It writes little bits of code before stopping, summarizing it, and then offering suggestions for next steps.

So Pause Assistant doesn't always behave in this way, but if it thinks you're doing something open-ended or exploring data, it's going to operate in this way where it does shorter turns. It stops and pauses and involves the user more often.

And this is in contrast to how it could behave, which is just, you know, you ask it a question and it goes off for as long as it thinks is necessary to completely solve the problem. So in this case, you know, if you ask it to explore your data, it could work for 10 minutes, explore every facet of that data, and then hand you all of that analysis at the end.

But this has the risk of losing the user, obscuring what the model is uncertain about or what assumptions it's making, and deciding its own answers to questions that it really should have asked you. This makes it more likely that it would make mistakes, but also just go off in a direction that you might not want it to go off in.

There's also another reason for this pausing pattern with the shorter turns. And it's that, in many cases, the point of analysis is building an understanding of the data for you as the person to build an understanding of the data. This might not always be true with what you're doing for analysis.

But in many cases, analysis has not been done if you don't learn something about the data. This is especially true for things like exploratory data analysis. Now, if the agent has done a bunch of work, but you have learned nothing about the data, you might say that the analysis has not been done.

So having the model pause often and involve you is a way for you to keep pace with the agent and work on building up your understanding of what is actually in your data.

Wrapping up

Okay. So in the first part, Simon talked about how LLMs can often behave in these very alien ways. They make strange errors, and they can be more concerned with advancing the analysis forward, performing progress, instead of valuing correctness.

And I tried to make the case that even though this is true, LLMs are still useful for data analysis. And we can still make a useful agent, especially if we work on telling it to care about being right and designing the scaffold so that it matters less if it's wrong.

If you want to learn more about Posit Assistant or AI happenings at Posit and in the broader world, Simon and I also write a newsletter, the AI newsletter on the Posit open source blog, and we release it roughly every other Friday.

Thank you. It's great. We're happy to answer any questions for the remaining time. I guess we can answer. Maybe there's some questions in the chat.

Yeah, I see one from Daniel Bauer. Is there any model in the middle built into the Posit Assistant harness?

So this was the approach where a model that doesn't have access to the rest of the conversation history that the main thread of the conversation does would pre-interpret the plot and then send the results of that interpretation back into the conversation history. Because we see in evals that that doesn't drive the score up substantially, that's not something that's implemented inside of Posit Assistant.

Simon and Sarah, thanks so much. We're going to tackle some Q&A now. I think you're already jumping in. Basically, if you see some questions in the chat box that you want to tackle, just feel free to answer those. If you want me to take a few more questions, feel free to do so.

So everybody, thank you so much for joining. This is a great time now in the chat. Offer up some questions, and we'll do some Q&A here for about 10 minutes or so with Simon and Sarah live.

And so just use the chat box, and then we'll take those questions over to Simon. So Simon and Sarah, if you want to pick some or want me to try to tackle. I know you've been answering them quite a bit in the chat.

Yeah, there's a new one from Mike Smith saying, any thoughts about using locally running LLMs with Posit Assistant via OLAMA?

Sarah, do you want to take this on, or should I give it a go?

Sorry, I was reading another question.

No, that's totally okay.

Do you want to go? Do you want to go ahead?

Yeah, so maybe six months ago or something, our answer was different than I think it is now. With many of the agents that Posit has worked on, the answer has been that we allow folks to hook up locally running LLMs to these agents, but we wouldn't recommend doing so. And that was because in evals, we saw that models that are small enough to run on a local machine tended not to be strong enough to really offer a good experience.

I think that has really changed in the last few months, especially with the releases of Gemma 4 and of Quen 3.5 and 3.6. These models that are small enough to run on a sort of high-end consumer laptop are now starting to be good enough to really offer a good experience for folks. So I would recommend running Posit Assistant with OLAMA. If you're using the Assistant inside of Positron, you'll be able to get up there. And just to set expectations, it's able to string together a few tool calls that are coherent, but beyond that, you're sort of gambling on whether you're going to get good responses from there.

Anthropomorphization of agents

About five or six more minutes. Do you see any good questions you want to tackle here, Sarah or Simon?

Yeah, there's a question about anthropomorphization of agents, which I thought was interesting. And I sort of take your point that it might be bad to overly anthropomorphize them, and probably some of it is an outgrowth of marketing. I do think in some ways this is often how we talk about technology. I think saying that you tell the model something I don't think is that different from how we've talked about other things on the computer. Telling R to do something, or asking your thermostat to tell you the temperature. But I don't know, I thought that was interesting. I think we generally try to avoid too much of it, and I do think it can obscure what's happening, but it's often a useful metaphor, and at least when I say things like that, I don't think we mean it too seriously, like it is sort of a metaphor or a shorthand. But I do think one reason to talk about it this way is also to point out when they do very weird things, it is sort of surprising. I definitely don't think they are working like people, and I think we've tried to point that out. These are weird ways of failing that are fundamentally different. Simon, do you have anything you want to add? I think that's always important to forefront. Because you interface with the thing via language, often the most clear shorthand for referring to pieces of that interaction is to speak about it like you would speak about the other thing that speaks to you in your language, which is a human. But again, they're not humans, they are some sort of strange bag of matrix calculus.

they're not humans, they are some sort of strange bag of matrix calculus.

Data confidentiality in pharma

There's a question from Hong that says, in pharma, a very protected field with confidential data, how will you protect this? Are you using OLAMA at all? So this is a broader question of confidentiality and trust inside of data science agents. So at least to speak for Posit assistants, if you're working in Workbench inside of an organization, the assistant can hook up to whatever your organization's trusted model provider is, and Posit isn't sort of interjecting themselves to see responses in between. So that's really a relationship, a trust relationship between your organization and their model provider. If you're using Posit Assistant with the Posit AI model provider, which is the $20 a month subscription for individuals, the default data retention setting is that we would not be able to see the conversations that you have with the assistant and the model providers that are serving the requests under the hood, which right now are the cloud series and then a private deployment of Gemma. We have zero data retention agreements with those providers, so the data is protected to the extent that those agreements make that happen.

Someone's saying, is the session going to be recorded? We do record the sessions. There's going to be some afternoon sessions that won't be recorded, and we post those onto our YouTube, but they go through all the approvals and edits and changes, and we also don't typically post the chat. So being here is a good way to interact with people, and also the videos will get posted, and probably about two to three months based on the editing, so stay tuned. There's a link to the YouTube channel for that. And Simon and Sarah, there's a couple questions coming over from the Q&A. I'm just going to see if any of these real quick are ones you want to tackle, because I'm not sure if you're going to see these. So I'll just maybe just answer these pretty quickly. I'll just go through them. It still seems you need to have a high domain expertise to know when it's inaccurate. Is that a correct assumption or thought?

I think my answer would generally be it depends. And it might vary by field. One reason is that I think the models are better at some things than they are at others, or they know about some things more than they know about others. And in some fields, like it may be harder to spot inaccuracies than others as well.

I generally feel like I only am comfortable doing analysis on data that I have some general understanding of. Just because the model can write the code does not mean I guess you should necessarily trust it to give you insights into data in a field that you literally have no experience with. But I think generally it can depend.

There's a question that came in from Kevin. I think we're going to tackle this later in Isabella's presentation, but maybe just see if there's something you want to add here. It said, can the Posit assistant use skills like a skills.md file? That's right. Yeah. So the assistant itself ships with a dozen or so built-in skills. And then you can also bring your own skills to the assistant.

AI for learning and education

Someone has a pretty interesting question. I was curious what you thought. It says, if you're new to coding or stats, do you think the assistant would be good for teaching?

What are your thoughts on that, Simon? Because I actually sometimes mention that as well, but I never quite know. Should we teach programmers programming first or use AI to teach them?

There's a line in this pretty popular AI newsletter that is repeated in almost every issue of the newsletter, which is something like, if you're very interested in learning, LLMs are one of the best tools for learning. And if you're not interested in learning, LLMs are one of the best tools for not learning. So I think the story of education with LLMs, I mean, I'm not a teacher, I'm not an instructor. So others have a much stronger expertise than I do here. But for folks that are really interested in learning with a Posit system or LLMs in general, I think you can really get a lot of mileage out of working with them.

if you're very interested in learning, LLMs are one of the best tools for learning. And if you're not interested in learning, LLMs are one of the best tools for not learning.

Sarah, Simon, do you see any others before we take a break here in a minute or two that you want to tackle or you think would be good ones to answer in the chat? Yeah. There's a question about the TUI, the terminal interface. Oh, I guess it's the same question twice. Okay. I thought someone had two people on this question. But if you mean by side chat extension, like when it's in the sidebar and the chat interface in the in an IDE, I think like generally it should function similarly. And I can paste the documentation about the TUI. I don't know. Are there any meaningful differences you want to highlight, Simon? Yeah. So I think my read of this is like, is the harness different between the terminal interface and the interface that you would have inside of like Positron or something like that? Maybe. And the answer is yes, it is the same harness.

Sourcing and testing datasets

There's a good question here about datasets. I'll try to just summarize it real quick for the sake of time. How do you recommend sourcing reliable raw datasets? We've explored public repositories, but we're having trouble finding suitable data. Are there best practices to test out data quality and reproducibility? And I want to mention, I think in the next section after this break, there's going to be a talk about the rPILOT Submissions Group PILOT 7 team that's working on a similar challenge here for pharma. How do we generate synthetic data? So you might stay tuned for that talk or potentially reach out to them to get involved. But Simon, I didn't know if, or Sarah, you had anything you wanted to add on basically synthetic data or suitable real data? How do you test some of these things?

Yeah, I don't know if we have much to add. I guess to clarify, like we're not training, we don't train any models. I think there was also a question about like data to fine-tune models for pharma or something like that. It's sort of outside of the scope of what we work on, at least at this point. But yeah, we do make these evals that use datasets. For the one that we talked about today, these are like just, well, they're sort of split into three categories. One of those is common datasets that the model was going to know about. That was part of the point. And then we simulated some data that was supposed to be in domains that the model should have expectations about. And then generally for testing, I think we just use data that we come across or find. And so then do rely on other people testing it with their actual data for their actual work. We don't really do any work with like generating large datasets for testing specifically outside these evals. I had to guess. In pharma, there tends to be one dataset that people reuse over and over called the CDIS pilot data, but it's really only applied in one therapeutic area. And people work across different areas of drug development and they work with different therapeutics and large molecule, small molecule. And so I think even when they're just using coding assistant tools or just even foundational LLMs, oftentimes they're just trying to find datasets that

represent similar to what they actually are going to be working with. And that seems to be tricky. Sometimes I oversimplify it with just asking it to generate some synthetic data, but I think people are trying to solve kind of a more niche application with that. And I think that's what the pilot seven team is trying to hone in on a bit. Somebody asked here about the positive assistant for beginners. Any like, Hey, I haven't used this before. I haven't even jumped into AI yet. And any like one-on-one advice to wrap up Simon or Sarah that you'd give to people?

If you launch RStudio or the most recent release of Positron, you'll be prompted at some point to download the assistant if you want to. And if you just want to like poke at it and get a feel for what these agents, what the feeling of working with these agents actually feels like, the TidyTuesday is a great source of just like data that you haven't seen before that you can maybe familiarize with just to get a sense for what working with the agents feels like.

I think there's actually, the PSI group has a wonderful Wednesdays maybe. I think that's the right name where it's a similar, but for pharma data. Oh, nice. Yeah. So both of them are great places to start.

Awesome. Well, I think we've got just 10 minutes here. We're going to give everyone a chance to stretch and get ready for the next set of sessions. Simon and Sarah, thank you so much for the great conversation, the great information. I'm completely overwhelmed by the chat, so I'm going to guess that's going to be fine. So hopefully you can stick around and hang out with the chat with everybody. We are going to transition over to a breakdown and then we're going to wrap up. With the chat with everybody, we are going to transition over to a breakdown and the next talk will kick off in a new Zoom presentation section within the Zoom platform. So please take a break from here. We're going to end this session and then we'll migrate over to the new one where we'll have back-to-back talks kicking off here in about nine minutes. So with that, Simon and Sarah, thank you so much for helping today and we will see everybody in the next session.