Transcript#
This transcript was generated automatically and may contain errors.
Earlier this year, we started seeing something internally and at the organizations we work with that concerned us. And that's that it's becoming normal to vibe-analyze data. A broader population of people can now convincingly do data analysis by pulling up a coding agent and asking a data question. There was an episode of Posit's Test Set podcast earlier this year that phrased this pretty hilariously. The title is, Your VP is Doing a Rogue Analysis in Cursor Right Now. And in this podcast, Danielle Thomas describes how she's seeing this in her organization.
I think data people are kind of horrified at the idea of vibe analysis. It sets a low bar for correctness, for reproducibility. You know, these coding agents, they tend to work in temp files. The harnesses that they work in tend to delete the conversation histories after a month. And so it doesn't set these models up well to write code that will be able to reproduce the same result even a month later. But at the same time, we can't just ignore this or say not to do it. People are going to reach for the technology that gives them answers.
And we think that data people have the skill set to serve these people with chatbots that give more correct and more reproducible answers. So if these models are going to make mistakes, if they have bad habits in data analysis, how do we make data analysis agents that are trustworthy?
So if these models are going to make mistakes, if they have bad habits in data analysis, how do we make data analysis agents that are trustworthy?
I think there are kind of two problems that we have to speak to in order to really answer this. One is that we need the model to write the right code. And we also need the user, the user has to know how trustworthy that code is. So if it's code that the data team is maintaining already, and these are sources of trusted analysis already that might underlie like a Shiny app or a Quarto report that exists from the data team, then that code is more trustworthy than code that the agent is sort of improvising on the spot.
Introducing the Commons package
These are exactly the problems that we're trying to solve in a new open source package called Commons. Commons is a tool for data scientists to build trustworthy data agents for their collaborators. And those data agents are designed to be more correct, more reproducible, and more transparent than coding agents that are most popular. It's open source in both R and Python, and it's built on our open source AI stack with Shiny, Elmer, and Chatless. And you can use it with whatever model from whatever provider that you want. I'm often asked why it is that there's a bird on the hex. This is a common kingfisher. So Commons, common.
One of the core ideas behind Commons is this concept of trusted code. So I mentioned earlier that the data team already has all sorts of code that produces analysis that is trusted by their collaborators and within their organization. So this could be R packages, Python packages, Quarto reports, Shiny apps, semantic views inside of databases like Databricks or Snowflake. And one of the questions that Commons tries to answer is what would it look like if the model could reach for this code directly rather than sort of hand authoring code on its own. So instead of writing code afresh, it's just reaching for this trusted code when it's around.
So in order to make that happen, Commons is doing three things. The first of them is providing a data structure that you can extract all of this trusted code and metadata about what that code means and how it's interpreted in the business context into this common data structure. And once that data structure exists, there's also a harness, so a set of prompts and tools that's provided to an LLM to try to give it the right context at the right time. And finally, once these agents exist and they're deployed in production, those agents will do a bunch of work live. And some of those conversations will result in the agent finding code that is trusted by the organization already and invoking it. But some of the other questions will result in the agent having to improvise R or Python code. And so in those cases where the agent has to improvise code, Commons provides some tools to try to push those responses up the trust ladder, allowing the data team to see what questions are being asked live and add new sources of trusted code over time.
Demo: the Commons agent in action
So let's see this in action from the perspective of the VP doing a rogue analysis, except this time the analysis is a bit less rogue. We're looking at a regular, regular chat application. We've designed Commons apps to look as much like regular coding agents as possible. So this should feel familiar to users of products like chat.com or cloud.ai. So that VP might ask, how is traffic trending for our site?
So the first way that these Commons agents work a little bit differently than most coding agents is that the agent will first search for a trusted calculation. And if it can find some piece of code that the data team uses already, then it will invoke that directly rather than writing code on its own. And so so in this case, it's found a calculation that is trusted by the data team, and it can invoke it, and then it will write a bit of context about what it sees.
The last thing you might notice is that the end of the response has this little green shield attached to the end. And if you hover over that green shield, you'll see that this answer comes from a trusted calculation. This green shield is added deterministically by the Commons system rather than by the agent, because the only way that the agent accessed and analyzed data is via the trusted calculation.
This green shield is added deterministically by the Commons system rather than by the agent, because the only way that the agent accessed and analyzed data is via the trusted calculation.
So the first thing that the agent will do in this sort of situation is look through the trusted calculations at first, and if it can't find it, it will search through context defined by the data team about how these calculations are done in the organization. So this could be data dictionaries, markdown files, if you have them existing context layers from like Databricks or Snowflake, and then it can write whatever code it needs to find an answer.
So in this case, it was able to make a plot with some R code and describe it to me, but it also knows how to interpret this plot because of some context that was defined by the data team. You'll notice at the end of this answer, there's a little blue citation icon, and if I hover over that, the agent has cited some piece of context that justifies its approach, and thus the response doesn't fall into the lowest trust tier, which I'll show now.
If I ask a question and the agent doesn't necessarily find some piece of context that justifies its approach, the end of its response will have a little yellow exclamation point, and if I hover over that icon, it reads, this answer was not produced by a trusted calculation and does not cite trusted context. So you can think of this as like a normal response from any other coding agent. In this case, the agent is improvising code and using its own intuition in order to try to answer your questions.
Revisiting the two core problems
So if we revisit those two problems that we were originally trying to solve, the model has to write the right code and the user has to know how trustworthy that code is. So in order to help the model write the right code, we provide the model with calculations and context that the data team uses already in order to produce trustworthy analysis. Then depending on how effectively the agent was able to find context or code in order to solve that problem, we can signal to the user whether it used code that exists already in the organization to answer the question or whether it had to improvise its own code in order to answer that question.
So each response that comes from a commons agent follows this sort of flow. Regardless of what the question is, the first thing that the agent will do is search for trusted calculations that can speak to this question. And if it finds one, it will use it and you'll get a green shield. Otherwise, the agent will poke around and if it finds one, it will use it and you'll get a green shield. Otherwise, the agent will poke around inside of the organizational context, freehand some R or Python or SQL code, and depending on whether or not it can justify its approach, it will get a little blue icon or a little yellow one.
Demo: DevRelAgent and Clinical Trials Agent
So I have a couple of demos to share. The first one is called DevRelAgent. This is a commons agent that looks at developer relations data for Posit inputs, so like GitHub repo stars, CRAN downloads, all of that. So this is DevRelAgent. Both the demos I'll share with you are deployed to Posit Connect Cloud. So for this one, we can ask questions like how many CRAN downloads were there for last month. Actually don't have last month's data yet, so I'm going to change this to August 2026. And again, it's going to look for a trusted calculation first, so it tries to find and go down that trusted path first before writing custom code. In this case, there is a trusted calculation for CRAN downloads, and if we expand this line that says ran a trusted calculation, we can learn more about what exactly was run. And then it gives us the answer, and because this came from a trusted calculation, we see this green shield icon.
Okay, let's ask something else. So I'm going to say plot CRAN downloads the dplyr from 2023 to 2026.
So in this case, it is going to run a trusted calculation, but that's not going to be enough. And so if in order to answer the user's question, it needs to do additional stuff, it still has the flexibility to do that. So it not only ran a trusted calculation here to get some current download data, it then, in order to plot it, wrote some custom R code. And then we have this plot, which could look a little better, but it's CRAN downloads over time.
I think one important thing for data agents to be able to do is tell you when they can't answer something. So let's see if we ask it a question that I know it can't actually answer what it does. How many times has Positron been downloaded? And in this case, it doesn't have enough data, it doesn't have the right hookups to the data that could answer this kind of question, so it says that it can't tell us how many times Positron has been downloaded. Great.
Okay, I have one other demo I want to show, which is this common agent called the Clinical Trials Agent. Again, this is deployed to Posit Connect Cloud. You can use this agent to analyze simulated clinical trials data. All of the trusted code comes from Insight Engineering's TLG catalog. And so the thing I wanted to point out about this is that this agent, the trusted calculations, don't just output tables or data frames, they output GT tables and also plots, which I'll show in a minute. So for this question, this is a GT table. You can see it's sort of styled. And then the other thing is that if you have a source for the trusted code, Commons will show you this view source thing. You can click on that and see the code source. I'm going to show you how to make a plot with a trusted calculation. So we can also ask it to make a Kaplan-Meier plot of overall survival by treatment arm. It's going to look for a trusted calculation. In this case, it has one, and so it makes a plot for us.
So I have the GitHub links to both these agents. If you want to take a look at the code, it's in that photo. We'll also have the links at the end. Okay, so after seeing these agents in action, I'm going to talk a little bit more about what it looks like for the person actually building and maintaining these Commons agents. And there's really three processes to making and maintaining a Commons agent. The first is the build process. This is when you're going to make your Commons agent. You need to hook it up to the trusted code and the data sources that you want to use.
The second is to deploy it. It's really easy to deploy Commons agents to Posit Connect Cloud or to Posit Connect. And this allows your colleagues to use the agent.
And then the third process is improvement. This is when you can review the agent's behavior and add more trusted code to cover more use cases. And Commons has a couple built-in functions that we'll talk about in a minute that help you do that. Okay, so how do you actually build a Commons agent?
Building a Commons agent
One helpful thing is that Commons ships with a built-in coding agent skill that helps you plus a coding agent like Posit Assistant or Cloud Code build a Commons agent. The skill walks you through the process of looking through all the data that you might want to make available to the Commons agent, all the code that might go into that trusted calculation layer, and then helps you construct everything together. I think this is really helpful and I would encourage you to try it out if you're building a Commons agent. I think it can help you get started faster than you would have otherwise. And there's instructions at this footer link about how to get that agent skill set up.
Commons also has this idea of information layers, so it's not just that we're making trusted calculations available to the agent. There's two different layers and they provide different kinds of information that help the agent write the right code and get answers right.
Okay, so these two layers are the semantic layer and the context layer. So, the semantic layer provides the trusted calculations and there's a couple forms that the trusted calculations can come in. They can either be in sort of normal R and Python scripts. We call these measures when they come in that format. These are basically just scripts of R or Python functions and then you add a measures tag or decorator that makes them into a measure. But they can also come from datadig.yaml definitions or from semantic models from Databricks or Snowflake.
And so, this is the semantic layer. It defines all the trusted calculations.
The second layer is the context layer and this is basically everything else. If the semantic layer is the trusted calculations, the context layer is where you put all the other information that you might want your Commons agent to have access to and it can use this information to guide how it writes custom code and also how it interprets results. The simplest form to put this in is probably markdown files. You can just sort of throw information. It doesn't have to be structured in any particular way into markdown files, but it can also come in the form of descriptive datadig.yaml fields. And datadig.yaml is just a particular specification for a data dictionary.
The actual code that you need in order to put a Commons agent together isn't particularly complicated or long. This is basically all you need. There is obviously other code happening in these files that are referenced here, but this whole thing constructs the Commons agent and then builds the app. I'm just going to step through this really quickly.
We need a connection to a model and like Simon said, you can basically use any model provider with any model that you want and this works through Elmer. Then we need to provide it the data sources. These are the pieces of data that the agent will have access to and can query and can run the trusted calculations or custom code on. And then we have our two layers that we just talked about. So, the semantic layer, we provide a semantic layer to the semantic layer argument, and then the context layer. And again, these two layers can come from different formats.
And then we build the app. If you have semantic views from Databricks or Snowflake or semantic user metric views, you can use those with Commons. I'm not going to go through this in too much detail, but this is how you would do that. Okay, so that's the build process. This is where you construct the agent and set up the layers.
Deploying and improving the agent
And then we want to deploy it so that your colleagues can use the agent and actually start to get answers from it. You can basically deploy the Commons app like any other Shiny app, anywhere that you would deploy a Shiny app. So, Posit Connect Cloud or Posit Connect. Optionally, you can enable trajectory logging. So, if you enable this, this means that you would be able to see the conversations that people have with your Commons agent. This is optional, and only if you want to and, you know, it meets your data policy. But if you do that, you can analyze those trajectories and see how the conversations are going, what kinds of things people are asking the agent, if it's giving you the right answers, all that. So, that can be really useful. It's also possible to enforce access with warehouse permissions and viewer level credentials.
Great. So, we built the agent, then we deployed it, and now you might want to improve it over time. In our experience, it sometimes takes a bit of work to get the agent set up and understand, you know, which pieces of context you want to provide to it, which data sources you want it to have. And it's possible that it's not going to work perfectly right away. So, we built in functions to Commons that help you analyze trajectories if you have that enabled, and then hopefully move stuff up the trust ladder so that the agent gets more correct over time.
So, like I mentioned earlier, you can optionally enable trajectory logging with the log equals true argument to Commons, and this means that your conversation histories will be recorded.
So, if you have enabled that, you can then use trajectory read to read in the trajectories. This either works from Connect or from local traces that you might have. And there's also a third function, trajectory review, which opens up a shiny app that ships with Commons where you can browse, flag, and annotate conversations. I'm going to show that trajectory reviewer really quick.
So, this is a trajectory reviewer app for the clinical trials agent that I showed earlier. This is useful if you have trajectory logging turned on and you want to take a look at the conversations people are actually having with the agent and see if it's working as you expect. So, you can do things like select a conversation, take a look at what happened in that conversation, and then if you want, add a note to yourself or to whoever's reviewing the trajectories. You can also flag conversations. You can sort of glance at the trust levels of the various conversations. And then there's also this plot at the top that shows you the trust levels over time. So, I think ideally what you would see is that the trusted, the amount of trusted conversations would hopefully go up over time as you move more and more things into that trusted calculation layer.
Code as the foundation
One of the primary principles that underlies Commons is that code is the foundation of what it does. And I think this sometimes can seem obvious, but isn't always obvious. And we often have to make decisions to force the model to follow this philosophy. And this is, you want code as the foundation basically of everything that the model does. And the reason this isn't a given is that LLMs can do things in their head, sort of. So, you can ask it to add two numbers together or do some kind of calculation. It doesn't have to write code to do that. It can do the calculation in its head. Then that means that that calculation is not accessible to you. It's not transparent. You can't take a look at it. You can't rerun it. And also, it's often worse.
And part of the reason is that LLMs are really good at writing code. This is one of the things that they really excel at. And we want to take advantage of that. And then the second reason is that we know from years and years of using code to do data analysis that code is reproducible, it's transparent, and it's auditable. And we don't want to throw all that away just because you have this shiny new LLM tool. So, as much as possible, we want the model to be able to write code. And so, the Commons agent runs its trusted calculation, which is code that you as the data scientists have vetted, but also when it writes custom code. It's not, you know, to the greatest extent possible, it's not doing those calculations in its head. It's writing code to get an answer. It shows you the code if you want to see it, and then it interprets it.
And the reason this isn't a given is that LLMs can do things in their head, sort of. It's not transparent. You can't take a look at it. You can't rerun it. And also, it's often worse.
Comparing Posit's AI tools
We get a lot of questions about the difference between these various AI tools that Posit makes, Posit Assistant, Commons, ShinyChat, QueryChat. So, Posit Assistant is really if you are writing code, you want to be close to the code when getting help with coding or data analysis, and you want to do that in your development environment. So, Posit Assistant might help you build a Commons agent, a ShinyChat app, or a QueryChat app. And then Commons, again, I think the differentiator is really this trusted code idea. And this is when you want to build a data analysis agent for your colleagues to use, and you have some trusted code, in particular, data sources that you want the agent to use. ShinyChat underpins Commons. This is if you want to build custom Shiny apps with chat components. So, if you want to get, you know, at the lower level and build something custom, you might want to use ShinyChat. And there's also QueryChat, which helps you explore tabular data through SQL-powered chat. QueryChat and Commons have a lot of similarities. A couple differentiators are that Commons, again, has this trusted calculation idea, and QueryChat also is the agent only writes SQL code by design, whereas a Commons agent might write custom R, Python, or SQL code.
QueryChat also might be a bit more customizable. Thanks. We have links to both of those demos, or to the code for both the demos, as well as to Commons.
Simon and I also write a newsletter every other week called the Posit.ai Newsletter, and that link goes to all the past editions on the Posit open source blog.