How Sampling Works
Coolhand uses a dynamic sampling strategy to decide which logs to evaluate for a given prompt template. The goal is to provide good coverage for new and low-volume templates without incurring excessive costs for high-volume ones.
The sampling rate is based on the number of logs a template has received in the last 30 days:
| 30-Day Log Volume | Evaluation Rate |
|---|---|
| 1 – 100 | 10% |
| 101 – 1,000 | 10% down to 1% |
| More than 1,000 | 1% |
The system calculates the target number of evaluations based on this rate and then creates new evaluations for randomly selected logs that haven't been evaluated yet. This calculation only considers logs from the past 30 days.
When Evaluations are Skipped
No evaluations are created for logs that match:
- System templates, such as "Unmatched" and "Ignored API Calls".
- Deprecated templates.
- Embedding templates.
Evaluation Batching
To process evaluations efficiently, Coolhand groups them into batches. A batch is processed when it reaches a certain size.
For templates with low log volume, it may take some time to accumulate enough evaluations to form a batch. This can result in a delay between when a log is ingested and when its evaluation becomes available.
Why You Might See Fewer Evals Than Expected
If you notice that a template has fewer evaluations than you expect, it could be due to one of the following reasons:
- The template was recently created: The sampling formula is not applied retroactively.
- The template's log volume is low: It may be taking a while to fill a batch for processing.
- The 30-day window: If a burst of logs is not sampled within 30 days, it will not be considered for evaluation later.
- The template is not eligible for evaluation: It may be a system, deprecated, or embedding template.