How is feedback quality scored?

This guide explains how to interpret the feedback analytics dashboard and understand what the metrics mean for your LLM workloads.

---

Quick Reference: Chart Glossary#

Sentiment Chart (Sentiment Breakdown View)#

Sentiment Chart (Net Sentiment View)#

Sentiment Chart (Compound Sentiment View)#

Match Rate Chart#

Feedback Types Chart#

Collection Rate Chart#

---

Understanding the Dashboard#

What Each Chart Tells You#

Sentiment Chart (Sentiment Breakdown) — Are users happy with responses?

• High positive rate (>70%) suggests responses are meeting user needs
• Rising negative rate may indicate quality issues or changing user expectations
• High neutral rate means users aren't engaging with the rating system
• Avg Revision shows how much users typically edit responses (higher = less editing needed)

Sentiment Chart (Net Sentiment) — What's the overall trend?

• Combines rating sentiment (60%) with revision sentiment (40%)
• Positive values (+) mean more users are satisfied than dissatisfied
• Negative values (-) mean dissatisfaction is winning
• Track this over time to spot quality trends

Compound Sentiment — How good are responses when combining all signals?

• Scores above 80% indicate high-quality responses
• Scores between 60-80% suggest room for improvement
• Scores below 60% indicate significant quality issues
• This score is more nuanced than binary thumbs up/down because it incorporates editing behavior

Match Rate — Can we trace feedback to requests?

• Match rates below 90% may indicate integration issues
• Unmatched feedback is still useful for aggregate trends but limits debugging

Feedback Types — What kind of feedback are we getting?

• High "Rating Only" suggests users aren't taking time to revise
• High "Revision Only" suggests users prefer to edit rather than rate
• "Both" provides the most complete picture of user satisfaction

---

Understanding Collection Rate#

Collection Rate measures what percentage of your LLM request logs have received feedback. This is different from Match Rate:

• Match Rate: What % of feedback has been linked to logs (feedback accuracy)
• Collection Rate: What % of logs have received any feedback (feedback coverage)

Collection Rate helps you understand how representative your feedback sample is of actual workload activity. A low collection rate means your sentiment analysis may not reflect the true user experience across all requests.

Collection Rate Benchmarks#

Note: Collection rates vary significantly by workload type:

• Human-in-the-loop workflows (e.g., content review, summarization): 20-40%
• Automated pipelines with spot-checking: 1-5%
• Background/batch processing: Often <1%

High-Signal vs Low-Signal Feedback#

Not all feedback is equal. High-signal feedback (revisions, explanations) provides more actionable insights than binary-only feedback (thumbs up/down):

Consider tracking both overall collection rate AND high-signal collection rate for a complete picture.

---

How Compound Sentiment is Calculated#

The Compound Sentiment score combines two signals:

1. Binary Rating (60% weight) — Did the user like or dislike the response?
2. Revision Score (40% weight) — How much did the user edit the response?

Why These Weights?#

• 60% for explicit rating: The user's stated opinion is the strongest signal of satisfaction
• 40% for revision behavior: Actions complement words—heavy editing suggests the response needed work, regardless of the rating

The Formula#

Compound Sentiment = (Binary Score × 0.6 + Revision Score × 0.4) × 20

Where:

• Binary Score: 5 for thumbs up, 1 for thumbs down
• Revision Score: 1-5 based on edit percentage (see below)
• The × 20 converts to a 0-100 scale

Understanding the Three Revision Scores#

When a user provides revised output, three scores are calculated:

Edit Score (Levenshtein-based)

Measures how much typing/deleting the user did (character-level edit effort).

Best for: Understanding user effort

Semantic Score (Trigram-based)

Measures how much the content/meaning changed (pattern similarity).

Best for: Understanding content quality

Revision Score (Combined)

Weighted combination: 40% Edit + 60% Semantic

This prioritizes semantic change (content quality) over raw editing effort. A response that was semantically wrong is worse than one that just needed reformatting.

When to Use Each Score#

The 2x2 Matrix#

Combining both dimensions reveals nuances:

Example Calculations#

When Only One Signal Exists#

If feedback has only a rating OR only a revision (not both), the score uses just that signal:

• Rating only: Binary Score × 20
• Revision only: Revision Score × 20

---

How Net Sentiment is Calculated#

Net Sentiment provides a single trend line showing overall user satisfaction, combining both explicit ratings and revision behavior.

The Two Components#

1. Rating Sentiment (60% weight) — Based on thumbs up/down votes
2. Revision Sentiment (40% weight) — Based on how much users edit responses

Rating Sentiment Formula#

Rating Sentiment = (Positive - Negative) / Total × 100

• Ranges from -100% (all negative) to +100% (all positive)
• A score of 0% means equal positive and negative feedback

Revision Sentiment Formula#

Revision scores (1-5) are converted to a sentiment scale (-100% to +100%):

Revision Sentiment = (Revision Score - 3) / 2 × 100

Combined Net Sentiment Formula#

Net Sentiment = (Rating Sentiment × 0.6) + (Revision Sentiment × 0.4)

Example Calculations#

When Only One Signal Exists#

• Rating only: Uses rating sentiment directly
• Revision only: Uses revision sentiment directly
• Both available: Combines with 60/40 weighting

Interpreting Net Sentiment#

---

Common Questions#

Why is neutral not counted in positive/negative rates?#

Neutral means "no rating given"—the user chose not to express an opinion. Including it would dilute the signal from users who did rate. The rates show sentiment among users who rated.

Why might match rate be low?#

Common causes:

• Feedback submitted without proper request correlation IDs
• Timing issues (feedback submitted long after the request)
• Integration bugs in the feedback submission flow

What's a "good" Compound Sentiment score?#

Why weight ratings more than revisions?#

Explicit ratings (60%) outweigh revision behavior (40%) because:

• Users may edit for style preferences even when satisfied
• A thumbs up with heavy edits might mean "good content, wrong format"
• The rating is the user's direct statement of satisfaction

---

Technical Details#

API Reference#

Submit feedback via the API:

POST /api/v2/llm_request_log_feedbacks
{
  "llm_request_log_feedback": {
    "sentiment": "like",
    "revised_output": "The corrected response text..."
  }
}

Parameters:

• sentiment (string, optional): "like", "dislike", or "neutral". Preferred rating field.
• like (boolean, optional, deprecated): true for thumbs up, false for thumbs down. Use sentiment instead. If both are provided, sentiment takes precedence.
• revised_output (string, optional): User's edited version of the response

The revision_score is automatically calculated server-side when revised_output is provided.

Converting like to sentiment

If you are migrating from the boolean like field, use this mapping:

Data Model#

How the Sentiment Score is Computed#

computed_sentiment_score is a weighted average of the parent feedback's rating and any partial feedback ratings attached to it:

The parent carries double weight because it represents the overall response rating, while partials represent targeted section feedback.

Example: Parent is like (+1.0), one partial is dislike (−1.0):

score = (2×1.0 + 1×−1.0) / (2+1) = 1.0/3 ≈ +0.33

The score is recalculated automatically (usually within seconds) whenever you submit feedback, add or remove a partial, or change any sentiment rating. It is null until at least one sentiment value is present.