AI Newsletter: How to choose a model

How do you know which model to use and when? Often, it’s not a question of which model is best, but of which model suits your task and needs at a given time. You might switch between models for different projects (package development vs. data analysis), or even within a single project (planning vs. implementation). Different tasks require a different mix of cost, token usage, speed, intelligence, and capabilities.

If you want our most durable, high-level advice: start with the most expensive model you have access to from either OpenAI or Anthropic.¹ Then, once you have a sense of the “ceiling,” try less expensive models and see how they compare. Developing a feel for what’s possible with LLMs will help you make better decisions about trade-offs.

That said, we’ll try to tackle this question more thoroughly in this post.

The model landscape#

Currently, Anthropic and OpenAI set the bar for AI capabilities. Google is often discussed as a third frontier lab, though its current model lineup is less competitive.

Anthropic and OpenAI each release a “family” of models: Claude and GPT, respectively. The most capable models in both families are also the slowest and most expensive. Conversely, the least capable models are the cheapest and quickest. Other labs tend to follow this same pattern, releasing a set of models with different trade-offs along the cost-performance curve.

Models within a given family tend to share a similar shape of intelligence, with related capabilities (relative to model size), shortcomings, and idiosyncrasies. For example, Claude Fable 5, Claude Opus 5, and Claude Sonnet 5 often use the same turns of phrase and are quite good at writing code and debugging it. Models from different families can have different shapes of intelligence even when their benchmark scores and prices are very similar. For example, GPT-5.6 Terra and Claude Sonnet 5 are priced similarly and comparably capable at agentic coding, but Terra doesn’t “see” data visualizations as well as Sonnet, while Sonnet doesn’t communicate as clearly as Terra.

Other labs release models that score nearly as high as models from Anthropic and OpenAI on benchmarks. However, these evaluation scores can be deceiving. Labs can now train models to optimize for benchmarks (“benchmaxxing”). These models score well on benchmark-shaped tasks, which tend to be highly autonomous and “tricky,” but can fail to generalize to real-world tasks, which often involve more ambiguous requests. Kimi K3 and GLM 5.2 offer a counterexample: they score well on benchmarks, and we’ve also found them exceptionally well-rounded and intuitive. We recently introduced both models to Posit AI.

In previous newsletters and blog posts, we’ve shared results from targeted evaluations. Here, instead, we offer an approximate and entirely vibes-based comparison of these model families’ characteristics.

For data science applications broadly, you can loosely think of the relevant score as the average of the agentic coding and vision scores we’ve assigned here. Beyond writing R and Python code, models need to be able to interpret plots accurately and faithfully.

Find the model that fits your task#

Much of model choice is constrained by the models you have access to. Your organization may only allow a certain provider, or you might not want to pay multiple (possibly expensive!) subscriptions just to have access to all the top models.

If you do have your pick, however, here are some quick guidelines, partially shaped by what models are currently available through Posit AI.

The best open-weights model, especially for data analysis: Kimi K3.

As Simon wrote in the Kimi K3 and GLM 5.2 in Posit AI announcement post, “Kimi K3 is currently the most capable open weights model out there. In our internal testing, it feels somewhere between Opus 5 and Fable 5, and is notably well-rounded compared to other open weights releases.”

It also ranks near the top in bluffbench2 (13.46%, compared with 16.35% for the tied top scorers, Gemini 3.5 Flash and Claude Fable 5), which evaluates models’ abilities to spot subtle data quality issues in visualizations.²

A highly autonomous model for a complex coding or data task when cost and speed aren’t a concern: Claude Opus 5, Claude Fable 5, or GPT-5.6 Sol.

These are the top-of-the-line models from Anthropic and OpenAI. They are expensive and relatively slow, but can be worth using for ambitious or highly autonomous work.

A strong open-weights model for coding when you don’t need vision: GLM 5.2 (from Z.ai).

“GLM 5.2 excels at agentic coding and less so at data analysis tasks.” It is much less expensive than the proprietary models it resembles for coding tasks, but it does not natively support image inputs.

A middle-tier generalist for coding or data analysis: Claude Sonnet 5 or GPT-5.6 Terra. Both are capable across coding and routine data analysis, support vision, and are less expensive than their respective labs’ higher-tier models.

Good plot or image interpretation: One of the Gemini 3.x Flash models (3.7 was released on August 13).

This model series is particularly strong at vision and has performed well on bluffbench (3.5 Flash) and bluffbench2 (3.5 and 3.6 Flash).

Fast answers from an Anthropic model for a task that is not particularly complex: Claude Haiku 4.5.

Fast answers from an open-weights model for a task that is not particularly complex: Gemma 4.

Assorted notes from August 2026#

In late summer 2026, a few developments feel notable, but, as with much in the AI world, who knows how long these observations will hold.

• Google Gemini currently does not have any models that perform near the frontier. With the releases of Gemini 2.5 Pro (June 2025) and the Gemini 3 series (November 2025), Google seemed positioned as a third frontier lab. However, it’s been quite a while since they released a frontier model, and they’re now meaningfully behind. As of the time of writing, Google says Gemini 3.5 Pro is still testing with partners. Meanwhile, the company has begun training Gemini 4 and says it is excited by the progress.

• For a year or so, it seemed like Anthropic was solidly ahead of OpenAI in agentic coding. However, since the Claude 4.6 releases, it has become less clear that Anthropic is meaningfully ahead of OpenAI. For one, Anthropic’s current high-end models use newer tokenizers that, according to Anthropic, produce roughly 35% more tokens for the same text than their predecessors. This means the same listed price per token does not necessarily translate to the same cost for comparable text. Further, in our experience, the Claude series has become increasingly token-hungry and difficult to communicate with. At the same time, OpenAI’s models have a notably clear, concise communication style compared with the Claude 5 series. Anthropic is still likely ahead on autonomous, long-horizon coding, but OpenAI no longer feels behind for day-to-day software engineering and data science.³

• There are now a number of balanced, well-rounded open-weights models relatively close to the closed-weights frontier. Kimi K3 and GLM 5.2, in particular, combine strong capabilities with a more pleasant, intuitive feel than their predecessors. While earlier open-weights models were just as close to the frontier in benchmark scores, some newer releases are notably more well-rounded and respond to prompts about as effectively as proprietary models. These releases are also priced at a steep discount compared with the proprietary models they most resemble.

Recent past newsletters#

• Keep track of your data exploration with Posit Assistant’s EDA log
• Which models are best at spotting subtle data quality problems?

Subscribe via RSS