Transcript#

This transcript was generated automatically and may contain errors.

What if the hardest part of artificial intelligence is not building tools, but measuring their effectiveness?

We live in a world of incredible artificial intelligence capabilities between generative assistants, co-pilots, and AI agents, things that are possible today were not even dreamed about six months ago. It's faster than ever and easier than ever to build rapidly and arrive at concepts and ideas that previously would have taken months of time and now can happen in only an afternoon.

But speed does not necessarily equal confidence. It's really easy to build something quickly, but it's much more difficult to understand if that thing that we've built, whether it's an agent, a tool, or some other AI native application is actually behaving the way that we expect and intend.

My name is James Blair and I'm a senior product manager at Posit. Today we're going to be talking about artificial intelligence evaluations and how we can measure and evaluate the behavior of these AI tools and systems, and particularly why data scientists are particularly well positioned to succeed at this evaluation process. We'll look at ways to measure the input and output of these AI systems, how to effectively understand and evaluate whether they're behaving according to the expectations that we have, and how scalable infrastructure and tools can help make sure these evaluations happen not only in the initial design and build phase, but also throughout the lifetime of these AI tools and applications.

Traditional software vs. AI systems

If we look at the difference between traditional software development and the AI systems that we see today, it's clear that they don't behave the same. Traditional software, the same input would always lead to the same output. This made it easy to test. We could create tests that verified the output that we expected given some set of input, and those tests existed to provide scaffolding around how we expect that software to behave and alert us to any sort of unexpected changes.

But AI systems behave differently. The same input can lead to various outputs and different paths that the AI tool chooses to get to that output. This doesn't signal that things are wrong necessarily. In fact, it's an inherent behavior of these AI systems that they behave in perhaps unexpected yet productive and valuable kind of ways.

This variability means that we need to think more carefully about how we evaluate the output that's coming out of these kinds of systems, not only the output, but also what's contributing to the final decision or outcome that these systems create. The key issue here is not just whether a system can respond to a given set of input, but whether we understand how that response is generated and how that matches up against the real world expectations that we have for these AI systems.

From prototype to production

Many AI projects today start with a compelling prototype. As we mentioned in the beginning, it's easier than ever to create very compelling demonstrations and proof of concepts that even six or 12 months ago were not possible. We connect a model to some sort of a tool, a knowledge base, and suddenly we have a working example that illustrates the power of these systems.

But these demos are not the same as production-grade systems. We have to face the more difficult questions. Are we using the right source? Does the model reach for the right tool at the right time? Does it confidently approach the answer when it doesn't have the appropriate source? All of these are questions that have to be considered, not just the final output, but how we arrive there.

Looking at just a few examples of different model behaviors or different agent behaviors can be misleading. We need a way to rigorously test and verify model and agent behavior across a wide spectrum of different inputs and paths that the agent might take to arrive at some sort of output.

This work follows an experimental loop, very naturally kind of aligned with traditional scientific experiments. We provide input, we evaluate what the model does and what the agent does, and then we investigate the output to determine why we arrived at a given conclusion or why a specific output was provided.

This final output, while valuable, might hide different problems that exist in the actual agent journey. We might not see the actual sources that were used or the tool calls that were invoked to arrive at a specific output. Being able to evaluate not just the final output, but everything that goes into that final decision becomes critical in understanding and trusting these agent behaviors.

Why data scientists are the right people for AI evaluation

In this kind of world, data science brings that necessary structure to this inherently uncertain type of problem. Data scientists come with a wealth of knowledge around statistical rigor, experiment management, and analysis that allow them to work very, very naturally with these AI systems. These data scientists understand how to run these experiments, how to provide targeted inputs that allow a proper evaluation of the agent system, not just its final output, but every sort of intermediate step that might happen along the way.

These data scientists can define the appropriate success criteria, interpret signal from noise, and separate meaningful improvements from random variation. These data scientists have the ability to make AI evaluation rigorous so that we move from just operating off of vibes into a world where we can operate off of trustworthy evidence that our systems are behaving the way that we expect, and over time, improving as models and other features of these systems improve as well.

These data scientists have the ability to make AI evaluation rigorous so that we move from just operating off of vibes into a world where we can operate off of trustworthy evidence that our systems are behaving the way that we expect, and over time, improving as models and other features of these systems improve as well.

There's no sort of universal metric that proves that AI is working. We've seen many different examples in pop culture and other places of AI systems that break down in unexpected ways once they hit the real world. Use cases from users interacting with chat agents and asking for things that go well outside the scope of what that agent was designed to answer to other sort of more nuanced examples where AI systems behave confidently or return confident results that are, in fact, factually inaccurate.

We have to be able to move from generic scores and evaluations to very specific, detailed investigation around how these AI systems behave, and that varies across industries and across use cases. One metric does not satisfy every sort of AI opportunity. Instead, we need to consider what the expectations are for a given agent in its specific domain. Data scientists are particularly equipped to deal with this as they bring domain-specific knowledge to play along with their understanding of statistical analyses and general data acumen.

If you consider a customer support agent or a policy assistant or a code assistant, all these things have similar ingredients, some sort of foundational model, tool calling, context, prompts, yet they're expected to behave differently, and the way that users interact with these systems are often unique. Being able to evaluate each of these systems in its own unique way becomes critical in understanding both the behavior of these systems as well as the output that they produce.

We want to make sure that every step along the way, we're evaluating and capturing the necessary information to understand that these systems are behaving in expected ways, both in initial development as well as in final production.

Evaluation never stops

That leads us to this idea that this evaluation never stops. It's important to start with this goal in mind. When we start building these AI systems, it can be very easy to get caught up in the excitement of building something new and exciting, something that feels revolutionary. It's exciting, even still today, to build a system that responds to natural language requests and does things that even three months ago maybe seemed far-fetched, but we also have to consider the fact that we need to think about evaluating these systems from the very beginning.

Capturing traces and logs along the way enable us to make sure that these agents are choosing the right path, not just today, but also tomorrow. These systems evolve as things like models change, the way the toolkits and frameworks that these models use evolve and grow over time. Our system prompts may be updated regularly based on new domain knowledge. As each of these changes are made to this system, we have to make sure that evaluations are in place that allow us to understand how these changes impact these AI systems.

Are we seeing improvement where we expect improvement? Is the agent continuing to rely on the appropriate sources for its responses? This becomes particularly critical as foundational labs introduce newer and newer models with more and more capabilities. We need to make sure that introducing a new model to our system doesn't cause the behavior of that system to dramatically drift from our expectation.

In addition to capturing these logs in our training and in our evaluation, we also need to make sure that we're capturing this information as real-world use cases build up with our systems and tools. Understanding how users are truly interacting with these systems in production enables us to understand and continue to verify that our system meets the needs of real use cases from real users, not just the test cases that we define in our initial evaluations.

Data scientists are particularly equipped to set this up for success by building any evaluations from the beginning and making sure that those evaluations continue to apply even after a model or tool or system reaches production. These data scientists can work through this data and make sure that they're evaluating model performance on a continual basis to understand and make sure that these agent systems continue to meet business expectations long after the development is initially complete.

Infrastructure and tools

In the real world, what this typically means is we need a set of infrastructure behind the scenes that enables these kinds of consistent, constant evaluations, as well as a reliable set of tools that allow us to build these evaluations and continue to iterate and test them over time. At Posit, we bring best-in-class tools for developers and data scientists who are looking to use frameworks like R and Python and the vast open-source ecosystem that is available in those languages to build rigorous evaluations for a variety of AI models, tools, and frameworks.

AWS helps by providing foundational model access through platforms like Bedrock and providing flexible and scalable infrastructure that allows these evaluations to run continuously against AI systems so that we can flag and identify opportunities for future improvement or opportunities where things have degraded and performance has perhaps not met our expectations. Together, the flexible platform that Posit provides, along with the compute infrastructure from AWS, provide a very solid and stable environment for us to build evaluations that last not only today, but well into the future.

In this world of exciting AI advancement and opportunity, we want to make sure that you're not only building with speed, but also with confidence, that you understand and know that your systems behave the way that you expect, that you've tested for edge cases, and that you are continually monitoring not just the final output of what these systems are producing, but the intermediate steps that contribute to that final output, so that you understand when these models or when these agents begin to drift or when there becomes an opportunity for new and unique improvements to enable these systems to continue to meet the needs of your end users.

If you'd like to learn more about the work that we're doing here, feel free to scan the QR code on the screen now. Thank you.