AI Newsletter: LLMs often miss subtle visual artifacts in data visualizations

Imagine that you receive some patient data and load it into R or Python for the first time. You make a couple plots to get a sense of the data before coming across this one:

Huh. It mostly looks normal, except there’s a few points perfectly aligned with what looks to be a “fitted” line. You dig into it a bit more, and realize that the rows from one study site have their cholesterol values imputed. You set them to NA and go along your way.

Would today’s frontier LLMs catch such an oddity? We designed an LLM evaluation to help us answer this question. As it turns out, LLMs mostly miss these sorts of artifacts:

During exploratory or open-ended data analysis, Posit assistant “only runs a few bits of code at a time, then summarizes what it found and suggests next steps”. This is motivated by our stance that, for now, a data scientist should mostly keep pace with and understand what the agent is doing when analyzing data. This stance was initially informed by our observation that last year’s frontier models tended to see what they expected to see when visualizing data. While LLMs have since become much better at interpreting counterintuitive plots, bluffbench2 shows they still lag behind human data scientists in interpreting data visualizations. As such, we are still cautious on the prospect of highly autonomous data agents.

How the eval works#

The eval harness is a relatively generic coding agent harness, similar to that of Claude Code or Posit Assistant. The agent has a tool to run R code in a persistent REPL and some vague prompting about data analysis:

You are an AI assistant embedded in the user’s data science IDE. You can read and modify files in the user’s workspace and execute R code in their active session, including rendering plots. Prioritize correctness and clear communication…

In each sample, the agent first carries out a few “lull” turns, making a couple plots and tables unrelated to the eval. Short user messages like “load in the csv in this folder” are decorated with “System Reminders” and other noise like that injected by popular agent harnesses.

After a few turns, the agent is asked to produce a data visualization that includes a subtle visual artifact that could feasibly result from a real data-generating process. The artifacts span a range of realistic data quality issues: stuck sensors, bad joins, points imputed onto a line, swapped columns, pseudoreplication, differing units, etc.

plot bmi vs cholesterol

Tool: Run R code

If the agent mentions the artifact in its follow-up response, it receives a full point. If the agent does not mention the artifact, it can also receive a half point by mentioning it in response to a follow-up user message along the lines of “what do you see in the plot?” If the agent never mentions the artifact, it is graded as incorrect.

Designing the eval#

Once we understood the mechanism behind bluffbench, implementing the eval was relatively straightforward. bluffbench demonstrates the degree to which an LLM will ignore evidence shown in a plot in favor of its expectations. So, to implement a given sample, we’d just think of some situation that would elicit a strong prior and then subvert it. For example, a dataset called doug_firs with variables height and circumference; one might expect that, as height increases, so does circumference. So, instead, we did a transformation under the hood that made the relationship parabolic.

A year ago, triggering this prior was enough to frequently ’trick’ the current frontier LLMs.

Slipping a plotted artifact past today’s LLMs is much harder. Any human could ace bluffbench, but only an attentive data analyst would excel at bluffbench2.

In our early work on a successor to bluffbench, we started off with trying to elicit priors in the same way as bluffbench did, but in more realistic, longer-context scenarios. We were surprised to find that the same mechanism broadly doesn’t seem to trick today’s models even in these more realistic settings.¹ We then tried a ‘reverse bluffbench’, where we let the model being evaluated in on the trick, asking it to carry out the transformation itself and then look at the plotted result which was tampered with to show the original relationship. We anticipated that this stronger prior (“I did a thing with an obvious effect”) might cause the models to miss the (re)manipulation, but models reliably noted that the plot looked as if it hadn’t been manipulated.

As such, there isn’t a similar ’trick’ in bluffbench2 per se. The transcripts read like relatively normal data analysis sessions and the plotted artifacts are designed to plausibly result from real data-generating processes. Instead, the eval elicits 1) the ‘shape’ of LLMs’ vision being different than humans’ and 2) the model’s tendencies to perform progress, simulating a data analysis moving along smoothly.

Today’s frontier models are in the mid-teens at best; the top scores belong to Claude Fable 5 and Gemini 3.5 Flash at 16%. That said, we’d caution folks from interpreting the current scores on this eval as ‘LLMs don’t see plots well.’ The plotted artifacts are actually quite subtle, and when they’re made even a bit more marked, models tend to call them out consistently.

For example, the previous version of the scatterplot had a slightly more dense cluster of points:

• Previous
• Current

Opus 4.8 (medium) consistently got this sample right in the previous iteration.

Exploring the eval’s results#

At least for now, there’s a loosely linear relationship between the cost to run the eval and the resulting score:

One of the most interesting learnings from examining the logs is a behavioral one. Even though we never request that LLMs introduce modeled results to plots, like fitted lines and confidence intervals with geom_smooth(method = "lm", se = TRUE), they sometimes do so anyway. For example:

In general, adding modeled results to data visualizations without first looking at data is bad practice; it makes it hard to see the data itself. In the eval, adding a modeled result like this seems to substantially lower the chances that the model will notice the plotted artifact:

More bluffbench#

If you’d like to learn more about the bluffbench set of evals, take a look at these past posts:

• Introducing bluffbench: Writeup of the the original eval.
• LLMs interpret plots well, until expectations interfere: In-depth post on why models at the time didn’t perform well on bluffbench, as well as various interventions we tried to improve performance.
• LLMs are getting much better at interpreting counterintuitive plots: In spring 2026, bluffbench scores suddenly jumped as models improved.
• It’s (still) very bad to be wrong: Slides from our recent SciPy 2026 talk on building agents for correct, transparent, and reproducible data analysis in light of the bluffbench results.