What is feedback match rate?

This guide explains how Coolhand matches feedback to LLM request logs, what "Match Rate" means, and how to improve it for your workloads.

What is Match Rate?#

Match Rate is the percentage of feedback submissions that Coolhand successfully links to their corresponding LLM request logs.

Match Rate = (Matched Feedbacks / Total Feedbacks) x 100%

• Matched: Feedback linked to the specific log that generated the output
• Unmatched: Feedback that couldn't be linked to any log

A high match rate is important because matched feedback enables:

• Viewing the full request/response context alongside feedback
• Accurate per-template quality metrics
• Correlation analysis between prompts and user satisfaction

---

How Matching Works#

When you submit feedback, Coolhand attempts to find the LLM request log that produced that output. The system uses a multi-tier matching strategy that balances speed with accuracy, trying each tier in order until a match is found.

Tier 0: Provider ID Matching (Immediate)#

When: At feedback creation Requires: llm_provider_unique_id submitted with the feedback

The most reliable matching method. If you include the provider's request ID with your feedback submission, Coolhand performs a direct indexed lookup — no content comparison needed.

• Speed: Instant (indexed database lookup)
• Accuracy: Exact — one feedback to one log, no ambiguity
• Best for: Any integration that can capture the provider's response ID

await coolhand.feedback({
  llm_provider_unique_id: llmResponse.id,  // e.g. "chatcmpl-abc123", "msg_abc123"
  sentiment: 'like'
});

Note: Some provider IDs are not request-unique (for example, OpenAI Assistants run_ IDs span multiple internal logs). Coolhand detects these and skips Tier 0 automatically, falling back to content matching.

Tier 1: Hash Matching (Immediate)#

When: 2 minutes after feedback is created Search window: 15 minutes before to 1 minute after feedback creation

The fastest matching method. Coolhand computes an MD5 hash of the normalized output and looks for logs with the same hash.

• Speed: Instant (indexed database lookup)
• Accuracy: Exact matches only
• Best for: Unchanged outputs submitted shortly after generation

Tier 2: Similarity Matching (Delayed)#

When: ~30 minutes after feedback creation Search window: 2 hours before to 15 minutes after feedback creation

If hash matching fails, Coolhand tries fuzzy string matching using the normalized content.

• Speed: Fast (uses trigram indexing)
• Accuracy: Finds matches even with minor formatting differences
• Best for: Outputs with slight modifications (whitespace, punctuation changes)

Tier 3: Thorough Matching (Nightly)#

When: 2 AM UTC daily Search window: 3 months before to 24 hours after feedback creation

A comprehensive search that processes unmatched feedback from the previous day.

• Speed: Slower (regex-based normalization)
• Accuracy: Catches edge cases missed by earlier tiers
• Best for: Historical feedback, heavily modified outputs

---

Content Normalization#

Before matching, both feedback outputs and log responses are normalized to increase match success:

1. JSON extraction: Values are extracted from JSON structures
2. Whitespace collapse: Multiple spaces become single spaces
3. Special character removal: Keeps only letters, numbers, and basic punctuation (., ,, !, ?, ;, :)
4. Lowercase conversion: Case-insensitive matching

Example:

Original: "Hello,   World!!!\n\nHow are you?"
Normalized: "hello, world! how are you?"

---

Common Reasons Feedback Doesn't Match#

1. Timing Mismatch#

Scenario: Feedback is submitted before the log is fully ingested.

LLM requests are processed asynchronously. If feedback arrives before the log is ingested, early matching tiers may miss it.

Solution: The system automatically retries with Tier 2 (30 min) and Tier 3 (nightly). Most timing issues resolve within 24 hours.

2. Modified Output#

Scenario: The output shown to users differs from what the LLM returned.

Common modifications that break matching:

• Adding prefixes/suffixes (e.g., "AI Assistant: …")
• Stripping content (e.g., removing tool calls)
• Reformatting (e.g., converting markdown to HTML)
• Truncating long responses

Solution: Submit the original_output exactly as returned by the LLM, before any post-processing.

Note on transformed JSON: If your application builds a summarised or projected JSON view of the LLM response (e.g. extracting a subset of fields), Coolhand can often still match it via JSON value extraction. For a match to succeed, at least half of the feedback's meaningful leaf values (strings of 6+ characters, excluding common terms like status words and booleans) must appear in the original response, with a minimum of two such matches. A feedback payload with two shared IDs and several client-generated fields will not match if the shared values are fewer than half the total. The more unique identifiers shared, the more reliable the match.

3. Generic Content#

Scenario: The output is too common across many requests.

Examples:

• "I'm sorry, I can't help with that."
• "Here's the information you requested:"
• Short confirmations like "Done" or "OK"

If more than 5 logs match the same content, Coolhand rejects the match to avoid false positives.

Solution: Include more context in your outputs, or ensure unique identifiers are present.

4. Missing or Blank Output#

Scenario: The original_output field is empty or not provided, and no llm_provider_unique_id is included.

Matching requires either a provider ID or content to match against.

Solution: Include original_output with the actual LLM response, or include llm_provider_unique_id from the provider's response object. Either field is sufficient for matching.

5. Client Mismatch#

Scenario: Feedback is submitted under a different client than the log.

Matching only searches within the same client's logs for security and accuracy.

Solution: Ensure your API credentials correspond to the same client that made the LLM request.

6. Content Too Short#

Scenario: Normalized output is fewer than 20 characters.

Very short content has high collision risk and is skipped by similarity matching.

Solution: If your outputs are legitimately short, consider including additional context.

7. Log Never Ingested#

Scenario: The LLM request was never sent to Coolhand.

This can happen if:

• The Coolhand SDK wasn't initialized
• Network issues prevented log transmission
• The request was made before Coolhand integration

Solution: Verify your SDK integration is capturing all LLM requests.

8. Streaming Response Handling#

Scenario: Streaming responses may have different content structure.

Some SDKs capture streaming responses differently, which can affect the stored response format.

Solution: Ensure your SDK configuration handles streaming appropriately.

---

Understanding Match Warnings#

When feedback matches with caveats, Coolhand adds warnings:

---

How to Improve Match Rate#

1. Submit Feedback Promptly#

The sooner feedback is submitted after the LLM response, the more likely Tier 1 (hash matching) will succeed.

Best practice: Submit feedback within 15 minutes of the response.

2. Preserve Original Output#

Send the exact output from the LLM without modifications.

// Good
await coolhand.feedback({
  original_output: llmResponse.content,  // Exact response
  sentiment: 'like'
});

// Avoid
await coolhand.feedback({
  original_output: formatForDisplay(llmResponse.content),  // Modified
  sentiment: 'like'
});

3. Include Unique Content#

If your outputs are templated, include unique elements:

// Generic (hard to match)
"Your request has been processed."

// Unique (easy to match)
"Your request #12345 has been processed at 2:30 PM."

4. Include the Provider Request ID#

The fastest and most reliable path is including the provider's response ID as llm_provider_unique_id:

const response = await openai.chat.completions.create({ ... });

await coolhand.feedback({
  llm_provider_unique_id: response.id,  // "chatcmpl-abc123"
  original_output: response.choices[0].message.content,
  sentiment: 'like'
});

This enables Tier 0 matching — a direct indexed lookup with no content comparison required.

5. Use Direct Log References#

If you have the log ID, include it directly:

await coolhand.feedback({
  llm_request_log_id: "abc123hashid",  // Direct reference
  sentiment: 'like',
  original_output: response
});

This bypasses matching entirely for 100% accuracy.

6. Verify SDK Integration#

Ensure all LLM requests are being captured:

1. Check your Coolhand dashboard for request volume
2. Verify SDK initialization in all environments
3. Test with a known request and immediate feedback

7. Monitor Match Rate Trends#

Use the Workload Feedback dashboard to track match rate over time:

• Sudden drops may indicate integration issues
• Gradual decline might suggest output format changes
• Consistently low rates for specific workloads need investigation

---

Match Rate by Workload Type#

Different workload types have different expected match rates:

---

Debugging Unmatched Feedback#

If you have persistent matching issues:

1. Check the feedback warnings - They often explain why matching failed
2. Compare timestamps - Was the log created before the feedback?
3. Verify content - Does original_output match assistant_response in any log?
4. Check client ID - Are feedback and logs from the same client?
5. Review normalization - Are special characters or encoding causing issues?

For persistent issues, contact support with:

• Feedback ID (hashid)
• Expected log ID (if known)
• Original output content
• Timestamp of the LLM request

---

Summary#

Coolhand's multi-tier matching system is designed to maximize match rate while avoiding false positives:

1. Provider ID matching gives instant, exact results when llm_provider_unique_id is included
2. Hash matching catches exact content matches immediately
3. Similarity matching handles minor formatting variations
4. JSON value extraction handles transformed or summarised JSON outputs that share identifiers with the original response
5. Nightly thorough matching catches remaining edge cases

To maximize your match rate:

• Include llm_provider_unique_id from the provider response for the fastest, most reliable matching
• Submit feedback promptly with unmodified outputs
• Include unique content in your LLM responses
• Use direct log references when available
• Monitor trends and investigate drops

A healthy match rate is typically 85% or higher. If your rate is significantly lower, review the common failure scenarios above and adjust your integration accordingly.