How to monitor costs with Posit Assistant

LLM tokens can be expensive, but figuring out how much you’ve spent can be strangely difficult. With Posit Assistant, we wanted to make your spending as transparent as possible.

This is our third post in a series on LLM costs. In the previous two posts, we explained why you need to care about token caching and how Posit Assistant makes your token budget go further. This time, we’ll look at how to actually monitor your own costs and cache status in Posit Assistant.

The short version:

• Monitor costs with Posit Assistant’s estimated cost tracker to see what you’re spending as you work.
• When the conversation-size dot turns orange or red, compact or start a new conversation.
• Understand caching. Returning to a long conversation after a few hours can be costly. Consider starting a new conversation.

Monitor spending as you work#

Posit Assistant shows you an estimate of how much you’ve spent on tokens. You can see the estimated cost of a conversation in the status bar and the cost of each turn below the response.

The same view works across providers, for models where Posit Assistant has usage and pricing data (which is a lot of them!). Note that these costs are estimates based on published prices and might not exactly match your provider’s bill.

Want more info about a specific turn? Click the cost below a response to open Turn details, where you can see how long the turn took and a breakdown of its requests, tokens, and estimated costs.

When Posit Assistant delegates work to a subagent, that work also costs tokens. The subagent’s row shows its cost. Hover over the information icon to see its model and cost without opening its conversation. Subagent costs are also included in the conversation total.

Keep an eye on conversation length#

Next to the conversation cost, you’ll see a colored dot and the number of tokens currently in your conversation context. This is the amount of information sent to the model with each request, including your messages, the model’s responses, and tool results. Because LLMs don’t “remember” previous turns, agents like Posit Assistant resend that context each time you send a message. As the conversation grows, it takes up more of the model’s context window: the maximum number of tokens the model can work with at once. Context window size varies by model.

The colored dot gives you an at-a-glance sense of the current conversation size:

Compact or start fresh when conversations grow long#

Keep an eye on the conversation size and look for a stopping point as your conversation goes on. If the conversation is getting long and you want to continue the same task, use /compact to keep around a summary of your work while reducing the context.

If you’re moving on to a different task, you might instead use /clear or click the New conversation button in the upper-left of Posit Assistant to start a new conversation. /clear starts a new conversation with your current model and thinking level.

Long conversations make each request or cache miss more expensive. Compacting or starting fresh at a natural stopping point can help control costs and avoid an automatic compaction that might interrupt your work. You don’t need to wait until the dot is red.

Detailed view of spending, caching, and context#

In Posit Assistant, you can also see more detailed cost and token information in the Session information panel. To open the panel, click on the cost/token counter next to the model picker.

There’s a lot of information in Session information. We’ll cover it top-to-bottom.

Context window#

Use the context bar to monitor context and see when compaction will automatically happen for your chosen model. This can help you understand what’s taking up your context window and not get caught off guard by an automatic compaction. Different models have different context windows.

Token usage#

You can switch between Conversation and Session, and between Cost and Tokens. A conversation is a single chat thread within Posit Assistant, while a session covers the running Posit Assistant instance and can span many conversations. This means that the session view shows usage beyond the current conversation.

The totals here count tokens used across requests, not just the tokens currently in your context window. Hover over a bar to see more details about a request. Each request is broken down by token type.

You probably want to pay the most attention to the orange bars, which represent the Input (cache write) tokens. These are input tokens (i.e., tokens sent to the model) that you’re paying to add to the cache, rather than reuse at the cheaper cache-read price. Some orange is normal, but if you keep seeing lots of orange and not much green (especially in the Tokens view, not Cost), you may not be getting much benefit from the cache. This could be because of your model and provider, but it could also mean that you’re missing your cache window by frequently sending requests after the cache has cooled.

Cache efficiency#

Below the plot is a table of your different token types, and below that table you’ll see the Cache Efficiency row. It tells you what fraction of your input tokens were cache reads. In the screenshot above, our cache efficiency is 94.1% (pretty good).

Cache efficiency is partly on Posit Assistant, but partly on how you use it. A lower percentage doesn’t necessarily mean something is wrong, but taking long breaks and then continuing the same conversation (or switching models mid-conversation) can disrupt cache reuse. It’s worth checking your cache efficiency every once in a while to see whether you’re paying more than you need to for tokens.

Manage the cache#

Beyond helping users understand what they’ve spent so far in a conversation, Posit Assistant also includes a number of tools to help users understand (and affect) how much the next request will cost. This display and its controls are experimental and likely not necessary for your daily workflow. However, if you want to learn more about your cache status or are really trying to optimize costs, they’re worth trying out.

See whether or not the cache is warm#

In the previous post, we explained how token caching works. If the cache is “warm,” your model provider can reuse parts of your conversation that it has already processed, and you pay less for those parts in your next request. If the cache goes “cold,” the provider has to process those parts again, which can cost more.

Because of this, it can be useful to see your cache status. If the cache has gone cold on a large conversation, you might want to start a new conversation. If the cache is only going to be warm for 30 more seconds, you might decide to send a new message now rather than later.

Posit Assistant’s optional, experimental cache-status display adds a flame indicator that shows whether your cache is warm and estimates how much time you have left before it goes cold. It’s off by default.

To turn it on, click the cost/token counter or type /context to open Session information. Under Prompt cache, check Show cache status in the status bar.

For supported models, hover over or click the flame in the status bar to open a popover with the details of your cache status.

Give yourself time for a break#

The cache time-to-live (TTL) tells you how long the cache stays warm after it was last refreshed. For example, if your model and provider combination has a five-minute TTL, that means you have five minutes from the last time the cache was hit, either by your message or a keepalive ping (see the next paragraph). It does not mean you have five minutes from right now.

Posit Assistant can help you keep the cache warm by sending keepalive pings in the background while you take a coffee break or think through a decision. Each successful ping starts another TTL window, so by spacing the pings out, Posit Assistant can keep the cache warm for much longer than the TTL without you doing anything at all.

Here’s how to read the popover:

Know where your tokens are going#

You don’t need to optimize every token to keep costs down, and Posit Assistant is designed to be cost-efficient even if you don’t do anything to manage costs yourself. But at a high level, here’s what we recommend keeping an eye on:

• Monitor costs using Posit Assistant’s estimated cost tracker to see what you’re spending as you work.
• Compact or start a new conversation when the conversation-size dot turns orange or red. Look for a natural stopping point rather than letting the conversation keep growing.
• Understand caching. You don’t need to closely monitor the cache, but it’s worth knowing that if you come back to a long conversation after a few hours, you’re probably going to miss the cache and pay more than you need to. It’s probably a good idea to start a new conversation.