---
title: "How is feedback quality scored?"
answer: "Coolhand turns raw feedback into two numbers: a sentiment score drawn from likes, dislikes, and explanations, and a revision score measuring how much a user changed the output. Together they separate outputs people disliked from outputs people had to rewrite, which are different problems with different fixes."
category: "Feedback \u0026 prompts"
tags: ["feedback","scoring","analytics"]
published: 2026-08-30T13:57:52Z
updated: 2026-08-30T13:57:52Z
canonical_url: https://coolhandlabs.com/help/how-is-feedback-quality-scored
index_url: https://coolhandlabs.com/help.md
---

# How is feedback quality scored?

Coolhand turns raw feedback into two numbers: a sentiment score drawn from likes, dislikes, and explanations, and a revision score measuring how much a user changed the output. Together they separate outputs people disliked from outputs people had to rewrite, which are different problems with different fixes.

This guide explains how to interpret the feedback analytics dashboard and understand what the metrics mean for your LLM workloads.

---

## Quick Reference: Chart Glossary

### Sentiment Chart (Sentiment Breakdown View)

| Term | Definition |
|------|------------|
| **Positive** | Feedback where the user clicked thumbs up. Indicates the response met their needs. |
| **Negative** | Feedback where the user clicked thumbs down. Indicates dissatisfaction with the response. |
| **Neutral** | Feedback where the user did not provide a thumbs up/down rating. This is *not* a middle rating—it means no explicit rating was given. |
| **Positive Rate** | Percentage of rated feedback that was positive: `Positive / (Positive + Negative) × 100`. Neutral feedback is excluded from this calculation. |
| **Negative Rate** | Percentage of rated feedback that was negative: `Negative / (Positive + Negative) × 100`. |
| **Neutral Rate** | Percentage of all feedback that had no rating: `Neutral / Total × 100`. |
| **Avg Revision** | Average revision score converted to percentage: `(Revision Score / 5) × 100`. Shows how much users typically edit responses. |

### Sentiment Chart (Net Sentiment View)

| Term | Definition |
|------|------------|
| **Net Sentiment** | A combined sentiment score that weights rating sentiment (60%) and revision sentiment (40%). Rating sentiment = `(Positive - Negative) / Total × 100`. Revision sentiment converts edit behavior to -100% to +100% (minimal edits = positive, heavy rewrites = negative). Ranges from -100% to +100%. |

### Sentiment Chart (Compound Sentiment View)

| Term | Definition |
|------|------------|
| **Compound Sentiment** | A 0-100% score that combines explicit ratings (thumbs up/down) with behavioral signals (how much the user edited the response). Higher = better. See [How Compound Sentiment is Calculated](#how-compound-sentiment-is-calculated) below. |

### Match Rate Chart

| Term | Definition |
|------|------------|
| **Matched** | Feedback that was successfully linked to the original LLM request. This allows you to see the prompt and response alongside the feedback. |
| **Unmatched** | Feedback received without a link to the original request. The feedback is still valid, but you won't be able to see what prompt/response it refers to. |
| **Match Rate** | Percentage of feedback that was matched: `Matched / Total × 100`. Higher match rates mean better traceability. |

### Feedback Types Chart

| Term | Definition |
|------|------------|
| **Rating Only** | Feedback that only includes a thumbs up/down rating, with no revised text provided. |
| **Revision Only** | Feedback that only includes edited/revised text, with no thumbs up/down rating. |
| **Both** | Feedback that includes both a rating AND revised text. This provides the richest signal. |

### Collection Rate Chart

| Term | Definition |
|------|------------|
| **Collection Rate** | Percentage of LLM request logs that have received at least one piece of feedback. Measures how representative your feedback sample is of actual workload activity. |
| **Logs with Feedback** | Count of unique logs that have one or more feedback submissions linked to them. |
| **Total Logs** | Total count of client LLM request logs (excludes bakeoff comparison logs). |

---

## Understanding the Dashboard

### What Each Chart Tells You

**Sentiment Chart (Sentiment Breakdown)** — Are users happy with responses?
- High positive rate (>70%) suggests responses are meeting user needs
- Rising negative rate may indicate quality issues or changing user expectations
- High neutral rate means users aren't engaging with the rating system
- Avg Revision shows how much users typically edit responses (higher = less editing needed)

**Sentiment Chart (Net Sentiment)** — What's the overall trend?
- Combines rating sentiment (60%) with revision sentiment (40%)
- Positive values (+) mean more users are satisfied than dissatisfied
- Negative values (-) mean dissatisfaction is winning
- Track this over time to spot quality trends

**Compound Sentiment** — How good are responses when combining all signals?
- Scores above 80% indicate high-quality responses
- Scores between 60-80% suggest room for improvement
- Scores below 60% indicate significant quality issues
- This score is more nuanced than binary thumbs up/down because it incorporates editing behavior

**Match Rate** — Can we trace feedback to requests?
- Match rates below 90% may indicate integration issues
- Unmatched feedback is still useful for aggregate trends but limits debugging

**Feedback Types** — What kind of feedback are we getting?
- High "Rating Only" suggests users aren't taking time to revise
- High "Revision Only" suggests users prefer to edit rather than rate
- "Both" provides the most complete picture of user satisfaction

---

## Understanding Collection Rate

Collection Rate measures what percentage of your LLM request logs have received feedback. This is different from Match Rate:
- **Match Rate**: What % of *feedback* has been linked to logs (feedback accuracy)
- **Collection Rate**: What % of *logs* have received any feedback (feedback coverage)

Collection Rate helps you understand how representative your feedback sample is of actual workload activity. A low collection rate means your sentiment analysis may not reflect the true user experience across all requests.

### Collection Rate Benchmarks

| Collection Rate | Interpretation |
|-----------------|----------------|
| 20%+ | Exceptional - near-comprehensive coverage (e.g., integrated review workflows) |
| 5-19% | Excellent - strong feedback loop with representative sampling |
| 1-4% | Good - typical for most production workloads |
| 0.1-0.9% | Low - sparse feedback, interpret trends with caution |
| <0.1% | Very Low - insufficient data for reliable sentiment analysis |

**Note:** Collection rates vary significantly by workload type:
- Human-in-the-loop workflows (e.g., content review, summarization): 20-40%
- Automated pipelines with spot-checking: 1-5%
- Background/batch processing: Often <1%

### High-Signal vs Low-Signal Feedback

Not all feedback is equal. High-signal feedback (revisions, explanations) provides more actionable insights than binary-only feedback (thumbs up/down):

| Feedback Type | Signal Value | What It Tells You |
|---------------|--------------|-------------------|
| Revision + Rating | High | Exactly what was wrong and user sentiment |
| Revision only | High | What needed to change (actions > words) |
| Explanation + Rating | Medium-High | Why user was satisfied/dissatisfied |
| Binary rating only | Low | Directional sentiment, no actionable detail |

Consider tracking both overall collection rate AND high-signal collection rate for a complete picture.

---

## How Compound Sentiment is Calculated

The Compound Sentiment score combines two signals:

1. **Binary Rating** (60% weight) — Did the user like or dislike the response?
2. **Revision Score** (40% weight) — How much did the user edit the response?

### Why These Weights?

- **60% for explicit rating**: The user's stated opinion is the strongest signal of satisfaction
- **40% for revision behavior**: Actions complement words—heavy editing suggests the response needed work, regardless of the rating

### The Formula

```
Compound Sentiment = (Binary Score × 0.6 + Revision Score × 0.4) × 20
```

Where:
- Binary Score: 5 for thumbs up, 1 for thumbs down
- Revision Score: 1-5 based on edit percentage (see below)
- The `× 20` converts to a 0-100 scale

### Understanding the Three Revision Scores

When a user provides revised output, three scores are calculated:

#### Edit Score (Levenshtein-based)
Measures how much typing/deleting the user did (character-level edit effort).

| Score | Change % | Meaning |
|-------|----------|---------|
| 5 | 0-10% | Minimal edits—response was mostly usable as-is |
| 4 | 10-30% | Light edits—small corrections or additions |
| 3 | 30-50% | Moderate edits—significant adjustments needed |
| 2 | 50-70% | Heavy edits—major rewriting required |
| 1 | 70%+ | Near-complete rewrite |

**Best for**: Understanding user effort

#### Semantic Score (Trigram-based)
Measures how much the content/meaning changed (pattern similarity).

| Score | Similarity | Meaning |
|-------|------------|---------|
| 5 | 90%+ | Same content—minor wording tweaks |
| 4 | 70-90% | Similar content—some changes |
| 3 | 50-70% | Moderately different |
| 2 | 30-50% | Very different |
| 1 | <30% | Completely different content |

**Best for**: Understanding content quality

#### Revision Score (Combined)
Weighted combination: **40% Edit + 60% Semantic**

This prioritizes semantic change (content quality) over raw editing effort. A response that was semantically wrong is worse than one that just needed reformatting.

### When to Use Each Score

| Question | Use This Score |
|----------|---------------|
| How much work did the user do? | Edit Score |
| Was the response content correct? | Semantic Score |
| Overall response quality? | Revision Score |

### The 2x2 Matrix

Combining both dimensions reveals nuances:

| Scenario | Edit Score | Semantic Score | Interpretation |
|----------|------------|----------------|----------------|
| Minor tweaks | 4-5 | 4-5 | Response was good |
| Targeted fixes | 4-5 | 1-2 | Small edits, big meaning change (response was off) |
| Reorganization | 1-2 | 4-5 | Lots of work, same content (format was wrong) |
| Complete rewrite | 1-2 | 1-2 | Response wasn't useful |

### Example Calculations

| Scenario | Rating | Revision Score | Calculation | Final Score |
|----------|--------|----------------|-------------|-------------|
| Loved it, used as-is | Thumbs up (5) | 5 (no edits) | (5×0.6 + 5×0.4) × 20 | **100%** |
| Liked it, minor tweaks | Thumbs up (5) | 4 (15% edits) | (5×0.6 + 4×0.4) × 20 | **92%** |
| Liked it, but rewrote half | Thumbs up (5) | 3 (40% edits) | (5×0.6 + 3×0.4) × 20 | **84%** |
| Disliked, complete rewrite | Thumbs down (1) | 1 (80% edits) | (1×0.6 + 1×0.4) × 20 | **20%** |
| No rating, light edits | None | 4 | 4 × 20 | **80%** |
| Thumbs up only | Thumbs up (5) | None | 5 × 20 | **100%** |

### When Only One Signal Exists

If feedback has only a rating OR only a revision (not both), the score uses just that signal:
- Rating only: `Binary Score × 20`
- Revision only: `Revision Score × 20`

---

## How Net Sentiment is Calculated

Net Sentiment provides a single trend line showing overall user satisfaction, combining both explicit ratings and revision behavior.

### The Two Components

1. **Rating Sentiment** (60% weight) — Based on thumbs up/down votes
2. **Revision Sentiment** (40% weight) — Based on how much users edit responses

### Rating Sentiment Formula

```
Rating Sentiment = (Positive - Negative) / Total × 100
```

- Ranges from -100% (all negative) to +100% (all positive)
- A score of 0% means equal positive and negative feedback

### Revision Sentiment Formula

Revision scores (1-5) are converted to a sentiment scale (-100% to +100%):

```
Revision Sentiment = (Revision Score - 3) / 2 × 100
```

| Revision Score | Revision Sentiment | Meaning |
|----------------|-------------------|---------|
| 5 | +100% | Minimal edits = highly positive |
| 4 | +50% | Light edits = positive |
| 3 | 0% | Moderate edits = neutral |
| 2 | -50% | Heavy edits = negative |
| 1 | -100% | Complete rewrite = highly negative |

### Combined Net Sentiment Formula

```
Net Sentiment = (Rating Sentiment × 0.6) + (Revision Sentiment × 0.4)
```

### Example Calculations

| Rating Sentiment | Revision Sentiment | Calculation | Net Sentiment |
|------------------|-------------------|-------------|---------------|
| +50% (mostly positive) | +100% (score 5) | 50×0.6 + 100×0.4 | **+70%** |
| +50% (mostly positive) | -50% (score 2) | 50×0.6 + (-50)×0.4 | **+10%** |
| -20% (slightly negative) | +50% (score 4) | -20×0.6 + 50×0.4 | **+8%** |
| +30% (positive) | 0% (score 3) | 30×0.6 + 0×0.4 | **+18%** |

### When Only One Signal Exists

- **Rating only**: Uses rating sentiment directly
- **Revision only**: Uses revision sentiment directly
- **Both available**: Combines with 60/40 weighting

### Interpreting Net Sentiment

| Range | Interpretation |
|-------|----------------|
| +50% to +100% | Excellent—users consistently satisfied |
| +20% to +49% | Good—positive trend with room for improvement |
| -19% to +19% | Mixed—roughly equal satisfaction/dissatisfaction |
| -49% to -20% | Concerning—more users dissatisfied |
| -100% to -50% | Poor—significant quality issues |

---

## Common Questions

### Why is neutral not counted in positive/negative rates?

Neutral means "no rating given"—the user chose not to express an opinion. Including it would dilute the signal from users who did rate. The rates show sentiment *among users who rated*.

### Why might match rate be low?

Common causes:
- Feedback submitted without proper request correlation IDs
- Timing issues (feedback submitted long after the request)
- Integration bugs in the feedback submission flow

### What's a "good" Compound Sentiment score?

| Score Range | Interpretation |
|-------------|----------------|
| 90-100% | Excellent—responses consistently meet user needs |
| 80-89% | Good—minor improvements possible |
| 70-79% | Fair—noticeable quality gaps |
| 60-69% | Needs attention—users frequently unsatisfied |
| Below 60% | Poor—significant quality issues |

### Why weight ratings more than revisions?

Explicit ratings (60%) outweigh revision behavior (40%) because:
- Users may edit for style preferences even when satisfied
- A thumbs up with heavy edits might mean "good content, wrong format"
- The rating is the user's direct statement of satisfaction

---

## Technical Details

### API Reference

Submit feedback via the API:

```json
POST /api/v2/llm_request_log_feedbacks
{
  "llm_request_log_feedback": {
    "sentiment": "like",
    "revised_output": "The corrected response text..."
  }
}
```

**Parameters:**
- `sentiment` (string, optional): `"like"`, `"dislike"`, or `"neutral"`. Preferred rating field.
- `like` (boolean, optional, deprecated): `true` for thumbs up, `false` for thumbs down. Use `sentiment` instead. If both are provided, `sentiment` takes precedence.
- `revised_output` (string, optional): User's edited version of the response

The `revision_score` is automatically calculated server-side when `revised_output` is provided.

#### Converting `like` to `sentiment`

If you are migrating from the boolean `like` field, use this mapping:

| `like` (boolean) | `sentiment` (string) | Stored as integer |
|---|---|---|
| `true` | `"like"` | `2` |
| `false` | `"dislike"` | `0` |
| `nil` / omitted | *(no conversion)* | `nil` |

### Data Model

| Field | Type | Description |
|-------|------|-------------|
| `sentiment` | string (`"like"`, `"dislike"`, `"neutral"`) | Preferred rating field. Stored internally as integer (0=dislike, 1=neutral, 2=like). |
| `like` | boolean/null | Deprecated. `true` = positive, `false` = negative, `null` = no rating. Converted to `sentiment` on write. |
| `revised_output` | text | User's edited response (if provided) |
| `revision_score` | integer (1-5) | Computed from edit distance |
| `llm_request_log_id` | hashid string/null | Link to original request (null if unmatched). Stored internally as an integer FK. |
| `computed_sentiment_score` | decimal (-1.0 to 1.0) | Weighted rollup of parent + partial sentiment ratings. `null` if no sentiment has been submitted. Updated automatically within seconds of any sentiment change. See [How the Sentiment Score is Computed](#how-the-sentiment-score-is-computed). |

### How the Sentiment Score is Computed

`computed_sentiment_score` is a weighted average of the parent feedback's rating and any [partial feedback](https://coolhandlabs.com/help/what-is-partial-feedback.md) ratings attached to it:

| Component | Weight | Value |
|-----------|--------|-------|
| Parent `sentiment` | 2× | like = +1.0, neutral = 0.0, dislike = −1.0 |
| Each partial `sentiment` | 1× | like = +1.0, neutral = 0.0, dislike = −1.0 |

The parent carries double weight because it represents the overall response rating, while partials represent targeted section feedback.

**Example**: Parent is `like` (+1.0), one partial is `dislike` (−1.0):
```
score = (2×1.0 + 1×−1.0) / (2+1) = 1.0/3 ≈ +0.33
```

The score is recalculated automatically (usually within seconds) whenever you submit feedback, add or remove a partial, or change any sentiment rating. It is `null` until at least one sentiment value is present.


---

Related:
- [What is partial feedback?](https://coolhandlabs.com/help/what-is-partial-feedback.md)
- [How should I collect feedback on AI output?](https://coolhandlabs.com/help/feedback-collection-best-practices.md)
- [What is feedback match rate?](https://coolhandlabs.com/help/what-is-feedback-match-rate.md)
- [Creators vs. reviewers](https://coolhandlabs.com/help/creators-vs-reviewers.md)

[← All help articles](https://coolhandlabs.com/help.md) · [Feedback & prompts](https://coolhandlabs.com/help/category/feedback-and-prompts.md)