Skip to main content

How does evaluation sampling work?

Last updated 2 min read

Coolhand automatically samples your logs for evaluation using a formula that balances cost and coverage. The sampling rate is higher for new or low-volume prompt templates and lower for high-volume templates. Evaluations are also batched, so you may see a delay before they appear.

How Sampling Works

Coolhand uses a dynamic sampling strategy to decide which logs to evaluate for a given prompt template. The goal is to provide good coverage for new and low-volume templates without incurring excessive costs for high-volume ones.

The sampling rate is based on the number of logs a template has received in the last 30 days:

30-Day Log Volume Evaluation Rate
1 – 100 10%
101 – 1,000 10% down to 1%
More than 1,000 1%

The system calculates the target number of evaluations based on this rate and then creates new evaluations for randomly selected logs that haven't been evaluated yet. This calculation only considers logs from the past 30 days.

When Evaluations are Skipped

No evaluations are created for logs that match:

  • System templates, such as "Unmatched" and "Ignored API Calls".
  • Deprecated templates.
  • Embedding templates.

Evaluation Batching

To process evaluations efficiently, Coolhand groups them into batches. A batch is processed when it reaches a certain size.

For templates with low log volume, it may take some time to accumulate enough evaluations to form a batch. This can result in a delay between when a log is ingested and when its evaluation becomes available.

Why You Might See Fewer Evals Than Expected

If you notice that a template has fewer evaluations than you expect, it could be due to one of the following reasons:

  • The template was recently created: The sampling formula is not applied retroactively.
  • The template's log volume is low: It may be taking a while to fill a batch for processing.
  • The 30-day window: If a burst of logs is not sampled within 30 days, it will not be considered for evaluation later.
  • The template is not eligible for evaluation: It may be a system, deprecated, or embedding template.

Related articles