---
title: "How should I collect feedback on AI output?"
answer: "The highest-signal feedback is the edit a user makes to an AI output, then their written explanation, and only then a thumbs up or down. Send the request log id when you have it and the raw response text when you do not, plus a stable reviewer identifier either way, so every submission can be attributed."
category: "Feedback \u0026 prompts"
tags: ["feedback","sdk","integration"]
published: 2026-08-30T13:57:52Z
updated: 2026-08-30T13:57:52Z
canonical_url: https://coolhandlabs.com/help/feedback-collection-best-practices
index_url: https://coolhandlabs.com/help.md
---

# How should I collect feedback on AI output?

The highest-signal feedback is the edit a user makes to an AI output, then their written explanation, and only then a thumbs up or down. Send the request log id when you have it and the raw response text when you do not, plus a stable reviewer identifier either way, so every submission can be attributed.

Guidance on how to structure your feedback collection so that it captures the highest-signal
data about your AI outputs. This guide is about *what to send Coolhand and how to send it* —
for how Coolhand links feedback to logs, see [Understanding Feedback Match Rate](https://coolhandlabs.com/help/what-is-feedback-match-rate.md);
for how revision/sentiment scores are calculated, see [Feedback Scoring Guide](https://coolhandlabs.com/help/how-is-feedback-quality-scored.md).

---

## The signal-quality hierarchy

Not all feedback fields carry equal information. When designing a feedback collection strategy,
prioritize capturing data in this order:

| Priority | Field | What it captures | Why it's valuable |
|----------|-------|-------------------|--------------------|
| **Highest** | `revised_output` | The user's own corrected version of the AI output | Shows exactly what "correct" looks like — no interpretation required |
| **Medium** | `explanation` | A human-articulated reason the output was good or bad | Explains *why*, enabling root-cause analysis of prompt or model issues |
| **Lowest** | `sentiment` (`"like"` / `"dislike"` / `"neutral"`) | A binary or ternary rating | Directional only — tells you *that* something was wrong, not *what* |

If you can only collect one signal, prefer whichever is cheapest for your users to give *without
degrading its meaning* — a forced `explanation` on every interaction tends to produce low-quality,
copy-pasted text, which is worse than a clean `sentiment` alone. The goal is matching collection
effort to genuine signal, not maximizing the number of fields populated per submission.

When a user directly edits an AI-generated result and saves the correction, that's the single
highest-signal event you can capture — send it as `revised_output` rather than discarding it or
reducing it to a `sentiment: "dislike"`.

---

## Common feedback-structuring mistakes

These patterns look like they're capturing feedback, but actually degrade signal quality by
misusing a field for something it wasn't designed to hold.

### Stuffing a canned reason into `explanation`

**The mistake:** An automated dislike action (e.g. a thumbs-down button click, or a background
job inferring dissatisfaction from a retry) writes something like `explanation: "User clicked
dislike"` or `explanation: "Auto-flagged: response discarded"`.

**Why it's wrong:** `explanation` is meant to carry a *human-articulated reason* — "the tone was
too formal for our support chat" or "missed the discount code in the second paragraph." A canned
string that just restates the sentiment adds no information beyond what `sentiment` already
conveys, but it *does* get counted as "explanation present" in feedback-quality metrics, silently
inflating your apparent signal quality without any real benefit.

**The fix:** If the dislike is automated and no human reasoning is available, submit:
```json
{
  "sentiment": "dislike"
}
```
Leave `explanation` blank. An empty `explanation` on a `sentiment`-only submission is accurate;
a populated one that just repeats the sentiment is misleading.

### Using `revised_output` for something that isn't a correction

**The mistake:** Populating `revised_output` with a value the user didn't actually produce or
approve — for example, a system-generated reformatting of the AI output, a translated copy, or a
default fallback value substituted when the AI call failed.

**Why it's wrong:** `revised_output` is the highest-weighted signal in scoring (see
[Feedback Scoring Guide](https://coolhandlabs.com/help/how-is-feedback-quality-scored.md)) precisely because it represents deliberate human
correction. Feeding it non-corrective content dilutes revision-score calculations across your
whole workload and can make a template look like it needs heavy editing when it doesn't.

**The fix:** Only set `revised_output` when a human has actually edited and confirmed a change to
the AI's output. If your pipeline reformats output for display purposes, that's a presentation
concern — don't feed it back as feedback.

### Marking synthetically-extracted feedback as `creator_type: "human"`

**The mistake:** When feedback is derived by an LLM analyzing a completed chat transcript (i.e.
`data_source: "synthetic_extraction"`), also setting `creator_type: "human"` because a human was
*present* in the underlying conversation.

**Why it's wrong:** `creator_type` should describe who or what *authored the feedback judgment*,
not who was present in the source conversation. Mislabeling synthetic feedback as human-authored
makes it indistinguishable from feedback a person deliberately submitted, which skews any
downstream analysis that weights or filters by creator type.

**The fix:** Keep `data_source: "synthetic_extraction"` paired with `creator_type: "agent"` (or
leave `creator_type` as its default, `"unknown"`, if the extracting process doesn't have a strong
opinion). Reserve `creator_type: "human"` for feedback a person directly submitted through a UI
control, form, or explicit API call attributable to that person.

### Treating `neutral` as a real opinion

**The mistake:** Defaulting every un-rated interaction to `sentiment: "neutral"` and treating that
as if the user affirmatively said "this was okay."

**Why it's wrong:** `neutral` is not a middle rating — it specifically means *no explicit rating
was given*. Submitting `neutral` in bulk for interactions nobody actually rated inflates your
total feedback count without adding real signal, and dilutes positive/negative rate calculations,
which explicitly exclude neutral from their denominator.

**The fix:** If no rating exists, don't submit a `sentiment`-only feedback item at all — omit the
field, or don't send a feedback item for that interaction. Only submit `sentiment: "neutral"` when
a user was explicitly presented with a rating choice and chose a neutral/no-opinion option.

---

## Matching feedback to the right log

However you structure your feedback payload, always include enough information for Coolhand to
match it back to the LLM call that produced the output. At minimum, every submission needs one of
`client_unique_id` or `original_output` as a backstop; better matching is available via
`llm_request_log_id` (best) or `llm_provider_unique_id` (good). See
[Understanding Feedback Match Rate](https://coolhandlabs.com/help/what-is-feedback-match-rate.md) for the full matching-tier breakdown — this
guide won't repeat that mechanics reference here.

---

## Privacy: hash your creator identifiers

`creator_unique_id` identifies who submitted feedback, for deduplication and per-user analysis —
but it should never be a raw personal identifier. Emails, names, and other directly identifying
strings should be rejected outright.

**Don't:**
```json
{ "creator_unique_id": "jane.doe@example.com" }
```

**Do:**
```json
{ "creator_unique_id": "a94a8fe5ccb19ba61c4c0873d391e987982fbbd3" }
```

A SHA-256 hash of your internal user ID is the recommended approach — it's stable (the same user
always produces the same hash, enabling deduplication) without exposing any personal information.

---

## Restructuring your feedback collection

If a review of your feedback data suggests your collection strategy needs to change — for example,
your `explanation` field is consistently populated with auto-generated text, or your workload has
a healthy submission volume but very little `revised_output` — the fastest path to fixing your
integration is the `/feedback-collection` Claude Code skill
([Coolhand-Labs/feedback-collection-skill](https://github.com/Coolhand-Labs/feedback-collection-skill)).
It plans and implements a feedback-collection strategy tailored to your codebase, prioritizing the
same signal-quality hierarchy described above.

Install it with:
```
/plugin marketplace add Coolhand-Labs/feedback-collection-skill
/plugin install feedback-collection@coolhand
```

Then run `/feedback-collection` (optionally scoped to a path, e.g. `/feedback-collection
src/ai/`) to have it review and restructure your feedback calls in place.


---

Related:
- [What is partial feedback?](https://coolhandlabs.com/help/what-is-partial-feedback.md)
- [Creators vs. reviewers](https://coolhandlabs.com/help/creators-vs-reviewers.md)
- [How is feedback quality scored?](https://coolhandlabs.com/help/how-is-feedback-quality-scored.md)
- [What is feedback match rate?](https://coolhandlabs.com/help/what-is-feedback-match-rate.md)

[← All help articles](https://coolhandlabs.com/help.md) · [Feedback & prompts](https://coolhandlabs.com/help/category/feedback-and-prompts.md)