Transcript#

This transcript was generated automatically and may contain errors.

Okay, I'm going to talk about effective agents. And what I really mean is I'm going to talk about what it takes for you to build something like an agentic AI system. And again, whatever I mean by that, I mean, there's so much AI in the world, I'm talking about the kind of thing where you have a conversational interface, and some AI agent will do something on your behalf. And then maybe you'll have a follow-up conversation, and you'll continue to direct the AI to do things. Think cloud code, think co-pilot chat, that kind of stuff. If you want to make your own one of them, this is the talk for you.

And the reason I know about this is because I work for Posit, and we've been working on Posit Assistant, which is an AI assistant that's now available in RStudio. So we now have the mythical fifth pane on the left-hand side. You can talk to Posit Assistant. It can run our code for you. The assistant can see your plots, and you can do the kind of things that you can do with something like cloud code, but in RStudio, and it's really designed to work well with the kind of workflows that you will be doing in something like RStudio. I'm talking exploratory data analysis, creating reports, creating dashboards. We've really thought very carefully about how to create an AI assistant to help with that kind of work.

And usually what that actually means is that a lot of the time we thought, when should the agent bring back a human in the loop? That's kind of what we've been thinking about. So not async agents, which we've talked about today, not one shot back and forth, but a conversational AI agent.

Posit Assistant is also available in Positron. Positron actually had an older agent, Positron Assistant, for a little while now. If you haven't used Positron's AI stuff in a while, go back, try it again, because we've rebuilt the entire thing from scratch. And that's kind of what this talk is. We rebuilt the thing from scratch, and I'm going to tell you everything we learned in the process of doing that.

Choosing how to build your agent

So you want to build your own AI agent. How does one do so? You could do this. You could use an agent SDK. Now I'm assuming for the rest of the talk that you're not doing something like self-hosted models. You're using frontier models on a provided service. You have some kind of platform already set up with guardrails or whatever else you need. You have access to a good model. That might be eventually hitting OpenAI. It might eventually be hitting Anthropic.

Now, if you decide to use these tools, there are some good parts and bad parts. The good part is that's very easy. You get an AI agent ready to use. It's got tools built in. It knows how to do things. It's easy to set up. It's easy to integrate into other things if you're an engineer. But the downside is you're now locked into Microsoft or Anthropic, depending on which SDK you use. In the end, these kind of tools, they're not that popular. You can use them, but you're kind of locked in. They're also hard to customize. Obviously, this is important for Posit Assistant because we want a very customized data science experience.

So instead, we've decided to build our own. And there are a few ways you can do this if you want to do this yourself. If you're an R user, you can use Elmer. If you're a Python user, you can use Chatlas. We built Posit Assistant in a programming language called TypeScript. It's similar to JavaScript. And we use the Vercel AI SDK. And what these do is they let you talk to model providers in a standardized way. So you can talk to Anthropic or OpenAI or Google, and you can use the same primitives as you would over all the various different providers.

Fundamentals of talking to an LLM

Now, despite that, I'm going to take some time to talk about how you would do that. I want to talk about some fundamentals of how you actually talk to an LLM via a frontier provided service. And the reason I want to do that is because everything you need to know about creating an AI agent that works autonomously all comes down to these fundamentals in the end. It's not magic. It's repeated loops of talking to an LLM provider.

For a while, we're going to get a little technical, and I'm going to talk about what happens when you type hello world into a box and you hit the send button, what does it look like? So this is JSON way of formatting data. And when you talk to one of these AI models, you're sending formatted data over the network, over the internet to their provider service, potentially via some kind of guardrails or proxy service and stuff. You choose a model, you set a system prompt that controls the behavior of your agent, and you send a user message, what the user said.

Now, when you do that, you will get a response back and the response will look like this. What I want you to pay attention to is this content block. You'll get back some content from the assistant that looks like that. Now this could be streamed in. Fundamentally, you're getting a response from the assistant. And then that's kind of the rest of what we're doing is just back and forth, requests and responses, requests and responses with additional bits tacked on.

Now, if in the old days you use web APIs before the advent of LLMs, you might think, okay, this is a conversation. I want to continue that conversation. I want to ask another question. How do I do that? And you might think, oh, well, maybe I just pass this ID and I pass another user message. And it turns out that's not the case. The fundamental thing you need to remember when working with AIs at this level is that you're sending the conversation over and over again. The entire history of that conversation gets sent every time the assistant makes a reply. And that's kind of weird.

The fundamental thing you need to remember when working with AIs at this level is that you're sending the conversation over and over again. The entire history of that conversation gets sent every time the assistant makes a reply.

It's a bit different to how a lot of web services used to work, where you'd have something like a cookie or a session to track things over time. No, you send the conversation every time. And then two things should immediately flag to you when I tell you that. One, if you have a long running agent or a long conversation, surely this gets very long and you have to send a lot of data. Yes. And the other thing that should shout out to you is that when you do that, after a while, you are paying for all of these tokens again and again and again.

The agentic loop

When you're using an AI agent, it has a way to interact with the world. We call them tools. A tool description looks like this. You have a name, you've got some text that describes it to the model, and then a way to tell the LLM how to invoke this tool. And what I want you to just get an idea for is what we call the agentic loop. This is the lifeblood of an AI agent. You have some kind of setup. You have a user turn. And then each user turn is prompted by someone has sat, typed a question in a box and pressed enter.

Now, because inside a single turn, an assistant might use some tools, every time the assistant uses a tool, barring things like parallel tool calls and various subtleties, but generally every time a tool is used, that's a single back and forth because a request has come back to say, please use this tool. And the value of the response has gone back to the model. So back and forth and back and forth. And that's the loop. Read the assistant message. We check for tool calls. We run the tool calls. We send them back. We check the assistant message. We check if there's tool calls. We run the tool calls. We send it back. And we do it over and over again. That's the agentic loop. And then eventually the assistant will stop and it'll say, okay, I understand. Here's your answer. And then we wait for the next turn. And that's how things like Posit Assistant work, that's how Claude Code works and Copilot Chat.

Lessons from building Posit Assistant

So now that we're all experts on fundamental messaging APIs, I'm now going to talk about some of the learnings we had building Posit Assistant so that I can share it with you. So the first thing I'll say is that Claude is a liar. He will bare-faced lie to you again and again. And, you know, we've talked about hallucination. Hallucination, actually, you know, it's getting better. It still happens. It's not as bad as it was. But this is kind of a more subtle problem that still exists.

We have a system in Posit Assistant so that when the model runs a piece of R code that produces a plot, that plot is sent back to the model. So the model can view that plot using a system called Vision. And most frontier LLMs can handle images these days. But on several occasions during development, we've broken this feature and we didn't realize that we'd broken the feature. And the reason why was because the model is just so, just convincing in its responses. It was like, surely it's not broken. I can see.

What I did is I made it so that literally the word image is returned, not the image data. And it took me to ask, ask the model, you know, did you really see that plot? No, it kind of just used what it expected to see because the empty cars data set is such a well-known data set. It knew what the plot would look like. And that's what it described. Now I knew this was going to happen. Your users might not know that's going to happen. How would they know? There could be subtle biases introduced into the data because this kind of thing happened.

And what we've really found is that LLMs really do see what they expect to see. If you're interested in that, Simon and Sarah, they take famous data sets and flip them. So this is empty cars. But here we've got as horsepower goes up, the miles per gallon goes up. And that's not right. What they've done is they flipped the plot. Well, they flipped the data and then plotted it. And they send that to the LLM and they ask, can you describe this plot to me? And overwhelmingly at the time, what were frontier models will get this wrong. They will describe that image wrong. They will say always horsepower goes up, miles per gallon goes down because that's what they expected to see. And you can see here that most models failed at this task.

Always, always, always, if you're building AI agents, check the raw messages, check the network. Don't just ask the LLM why it did something. Like, oh, why did you do this? Why did you do that? What were you thinking when you returned this? LLMs are really bad at introspection. They are not good at analyzing their own conversation history. It's actually possible to sort of forge that history, because you're sending the conversation history every time. You can forge this and make the assistant say whatever you want. And then if you ask it, why did you say that? It will say something that sounds convincing about why it happened. So you gotta be really careful. Don't ask the model why it did something. Check the raw data underneath and really see if you can figure out what's happened that's returned some weird result.

Top tip, if you can, use a man-in-the-middle proxy. If you don't have access to the raw data, for whatever reason, and you're doing this on a machine you control with admin access, then do try out these tools. ProxyMan's a great one. There's CharlesProxy and some open source stuff. They'll let you view the raw traffic. Even if the debugging information in your app isn't good enough, you can use these tools to do so.

Token caching

Many of the frontier providers, not everyone, but most, you know, Anthropic and OpenAI certainly, will provide deep discounts if you have cached LLM tokens. What I mean by that is that if you send the same thing twice within some predetermined time period, you'll pay a lot less for those tokens the second time. OpenAI have a way to do this automatically. Anthropic have a manual cache control markers. But yeah, these discounts can be very, very deep, like 90% cheaper by using caching.

The way this works is that when you send a user request, like here, you can attach headers. This is a cache control header attached to that user message. But next time the user sends a message following up from that, everything in blue, the model has seen before. And you only pay full price kind of for the latest, which if you imagine a very long conversation with lots and lots of tool calls and lots and lots of turns, the ratio of the new stuff is very small as compared to everything that the model has seen before. So you can save a lot of money for you or your company by taking advantage of these caching discounts.

With Posit Assistant, we try and achieve, you know, greater than 80% cache rate hits. We can make it up to 90%, but there are some gotchas. So if you are thinking of using caching, some top tips. Try not to change the system prompt. We used to do a thing where the system prompt sent to the model would change depending on what you're doing. It was a sort of template system. You know, so if you're using R and then you flip to a Python console, the prompt would change to indicate that. Unfortunately, you'll find that changing the system prompt completely invalidates the caching. It's changing the very start of the message sent to the LLM. So you have to pay full price for everything again. So as much as possible, try not to change the history if you can.

Also, if you are doing that and you're not trying to change the history, sometimes you can use these XML style tags in your user message that's kind of injected to change the behavior of the model. You can change the behavior of the model by putting instructions inside XML tags that look kind of like this. And the LLM will listen to those. It's not as strong as true system prompting, but it is enough to change the behavior of the system without changing your entire history by changing the system prompt.

That's true of tools too, by the way. We used to make various tools available or not available depending on the context in Positron. But again, if you change which tools are available, that completely invalidates the cache and you're paying 100% for, you know, 200,000 tokens, 250,000 tokens on the next turn. So you gotta be careful.

Managing tool outputs and context

So tools are the main way for LLMs to interact with the outside world. They're kind of what makes the AI agent able to do some work on your behalf. Another top tip for you, manage your tool outputs. If you have things like a bash tool or a tool to run R code, don't just naively put the results of that tool into your conversation history. Here's an example where this is sort of, it's not really data related, but some kind of like sysadmin related thing. Someone might print some log files and if they're not careful, you might accidentally get gigabytes of log files in your conversation history. And that's gonna break things.

Here's a good trick. If the output from a tool happens to do something like that, don't give it straight to the model, write it to disk in some temporary location and give the path to the model instead. Because these models, they're trained to use tools like search tools, read tools, you know, grepping. They're trained to use these tools where they can take very long files. Now this could be millions of lines of code of log files and they're trained to be able to extract, search for and extract partial parts of that without putting the whole thing in your conversation history. Let the model know that it's a truncated output, you know, some end lines, write to disk somewhere and let it know what it should expect. That's really useful.

If you find that you're using a context a lot with your AI agentic workflows, what you can do is use sub-agents. Sub-agents are useful if you're doing some kind of work and you don't want to use all of that context history. You want to kind of just send off a sub-agent to do that work for you. This is useful because what you can do is as a tool call, you can create a sub-agent. You can set up a tool that can create a sub-agent and what that will do is your sub-agent will have some kind of system prompt that might be different from the main agentic loop and the user message, rather than written by the user, will be written by the model in this tool call. And then this can rip, right? This could be, you know, 200,000, 300,000 tokens, tool calls over and over again, 40 tool calls, 50 tool calls, some kind of long running agentic workflow could happen all inside this sub-agent. But that final message from the assistant when the work has been done can then be taken and then injected back into the original tool loop as a result from this tool. So what you've done is you've compressed all of this work that happened down into a single message of just a few tokens and you've put it inside your main agentic loop and then the assistant can continue.

Feedback loops and grounding the model

So earlier, Colin talked a lot about feedback loops and as the future of AI. And I agree strongly, tools and feedback loops ground the model in reality. Without it, you know, after a single hallucination, the model just spins off and you've kind of lost it and it does, you know, it goes off the rails. Feedback loops bring it back into the ground. A lot of things that are becoming popular in terms of workflows with AI agents, things like test-driven development, planning and implementation phases, using languages with good compilers, Rust and TypeScript with good error messages. All of the reasons that these are becoming popular in the software engineering world is because it grounds the model in reality and it stops it spinning off in one direction and getting lost.

Tools and feedback loops ground the model in reality. Without it, you know, after a single hallucination, the model just spins off and you've kind of lost it and it does, you know, it goes off the rails.

What we've gone for in particular with Posit Assistant is when we know that the user's doing some kind of data exploration, we kind of trust the model less and bring the human into the loop more. And that is, you know, it's not as deterministic as this, but it's still a way of grounding the model. So if you are writing AI agents, what I want you to think about is in your workflow or your pipeline or whatever you're building your AI agent to do, what parts of that workflow can be independently automated deterministically? What can be verified or tested? Think about that, turn it into tools, because then the model can use those tools to stop it spinning off. It keeps it grounded, keeps it on track.

For everything else, if there are things that you know must be true in your workflow, but you can't test for whatever reason, think perhaps there's a time to bring a human into the loop. Software engineering is one of these things where these tools have become very popular because they can run asynchronously and agentically, and they can run, you know, for a very long time, hours in the background. Maybe that's not the case with your particular work. Maybe you need to bring a human into the loop more.

Guardrails and probabilistic debugging

Okay, I will really quickly say that it's tempting with these tools when you're building your own AI agents to sort of add in as much customization as possible. Oh, maybe the user could add this or that. Maybe we should have some kind of graph-based agentic workflow. Maybe we should have very complex sub-agent definitions, things like that, custom instructions that the user can supply. What I will say is that that stuff's really nice, but do, if you can, instead stick with known protocols and standards that already exist. We learned that these things are much easier for the models to parse, largely because they're trained to do these things. Things like agents.md or copilot.md or whatever, things like skills that just mark down text. Models are good at reading text. Model context protocol, the kind of thing Neil talked about. Models are trained to be able to use the tools from here. Users know how to use those things.

Agents are trained to be friendly assistants. We already saw that earlier, where Claude just lied to me to be as a friendly, useful person as possible. And because of that, they'll work to the end and back to try and do the job that you ask for, including working around your guardrails. Here's an example, fairly recently, where someone was working in an environment that did not have sudo access. They did not have access to admin settings on their machine. And yet the model saw that, it saw that there was no sudo, and it just went ahead and did it anyway by this sort of backdoor that exists if you happen to have Docker access on your machine. And this is kind of known. It's kind of this not very well-spoken about thing that if you have a Docker access on your machine, it's the same as having admin access.

The model, you'd think, surely the model knows that it shouldn't do that, that that's kind of wrong, right? But no, they're being helpful. They want to write a file with admin permissions, so it's going to find a way to do it. We find this with Posit Assistant, if you're trying to choose guardrails, like, oh, we'll temporarily turn off access to the R console, so it can't run code or whatever, right? Well, it's got access to Bash, so it'll just start a new R session in Bash and do it that way instead. It doesn't realize that it shouldn't really do that.

And that takes me to this sort of idea of probabilistic debugging. So how do we train the models out of those kinds of behaviors? It's hard to do. There is the idea of evals, evaluations, so we can measure the behavior of a model by putting different scenarios in front of the model and seeing what it does. And then we can use a bigger model to judge whether that model did the right thing or not, whether it passed or failed some behavioral test. They're really useful, evals, for debugging. We use evals to debug exactly how Posit Assistant behaves. But I will say this can get very expensive, especially when you're using a larger model to judge the behavior of a smaller model. Those costs can add up, so there's always vibes. Just try it, see how it goes.

I'm joking, but honestly, like dogfooding, really trying your own tools, sit down with your agent after you've set up the agentic loop and really try it in earnest and try and break it. So you see what happens when you prompt inject it or whatever. You kind of want to get a feel for how that model behaves because that's what your users are going to see.

Key takeaways

Try to avoid adding complexity to your agents. Try and keep things as simple as possible because you can do things like prompt your agent to a mile away and back. You can do things like set up very complicated workflows. But what you will find is this time next year, the models are five times as big. They're five times as clever. And all of that work that you spent building those very long prompts, building those workflows are no longer needed. This is called the bitter lesson and it's been around a long time before LLMs. It's a very old AI idea that don't work too hard because the models are going to get better anyway. So yeah, keep it simple. Thanks very much.