Caching in Shared Models

During a typical multi-stage conversation with a LLM, each time you send a new message the model has to consider both your new message and all the previous messages or documents in the conversation.

Because the earlier parts of your conversation don't change, the state of the model after each message can be saved and reused, which makes answering each new message faster.

This is also called "KV caching" or "prompt caching" by other providers.

Understanding usage

Responses from Shared Models contain a usage key which describes the number of input and output token used, including whether input tokens were retrieved from the cache.

For example, after calling https://models.think.evroc.com/v1/chat/completions

{
    [...]
    "usage": {
        "prompt_tokens": 74,
        "total_tokens": 138,
        "completion_tokens": 64,
        "prompt_tokens_details": {
            "cached_tokens": 40
        }
    }
}

In this case, the first 40 tokens in the prompt were retrieved from the cache.

Reduced billing applies for tokens retrieved from cache.

There is no guaranteed lifetime for caches (and hence whether a subsequent request can take advantage of caching). Typical lifetimes are minutes to hours.

Security and Privacy

Caches are tensors representing the internal state of the model, not the original content of your messages. They can only be re-used within the same Project where the hash of earlier messages is identical.

Caches are stored only on the machine running the LLM model.

Controlling caching

If it is desirable that caches should not be shared between users within one Project, our OpenAI-compatible endpoints use the following fields in the request body (if set):

  • user
  • prompt_cache_key

The following requests will not share caches, despite using the same API key and project:

curl https://models.think.evroc.com/v1/chat/completions \
  -H "Authorization: Bearer $EVROC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "zai-org/GLM-5.3",
    "user": "tenant-1",
    "messages": [{"role": "user", "content": "Tell me a story."}]
  }'

curl https://models.think.evroc.com/v1/chat/completions \
  -H "Authorization: Bearer $EVROC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "zai-org/GLM-5.3",
    "user": "tenant-2",
    "messages": [{"role": "user", "content": "Tell me a story."}]
  }'