Skip to main content

How to tell if a model's response was truncated

Last updated 2 min read

A model's response is truncated when it stops generating output because it reached the maximum token limit you requested. Coolhand now detects this by inspecting the `finish_reason` from providers that support it. This data helps Coolhand's optimization agents make better recommendations, but is not yet exposed in the API.

A model's response is considered truncated when it stops generating output prematurely because it has hit the max_tokens limit set in the request. This can result in incomplete sentences or code snippets, and can be a frustrating and quiet failure mode.

How Coolhand detects truncation

Coolhand now automatically detects when a response was truncated. For providers that include a finish_reason in their API response, Coolhand inspects this field. If the reason indicates the output was stopped due to its length (for example, a finish_reason of length for OpenAI-compatible models), Coolhand records that the response was truncated.

This detection is only possible for providers that supply a finish reason. Truncation is not currently detected for the following providers:

  • Copilot
  • ElevenLabs
  • TypeSafe
  • Embeddings providers

How Coolhand uses this information

Currently, this truncation data is used internally by Coolhand's own systems. For example, the cost-optimization agent can use this signal to recommend increasing your max_tokens request parameter if it observes that a prompt is frequently resulting in truncated outputs. This can help you improve response quality without manual investigation.

The max_tokens you request is different from the model's own absolute maximum output token limit. You can find the model's limit in the Coolhand model catalog.

Checking for truncation in your logs

The truncation signal is stored internally on the LLM request log. However, this information is not yet exposed in the v2 REST API, so you cannot directly query for logs that were truncated. We plan to expose this in a future release.

For a complete reference on the data available via the API, see What every field on an LLM request log means.

Related articles