Guidance on how to structure your feedback collection so that it captures the highest-signal data about your AI outputs. This guide is about what to send Coolhand and how to send it — for how Coolhand links feedback to logs, see Understanding Feedback Match Rate; for how revision/sentiment scores are calculated, see Feedback Scoring Guide.
The signal-quality hierarchy
Not all feedback fields carry equal information. When designing a feedback collection strategy, prioritize capturing data in this order:
| Priority | Field | What it captures | Why it's valuable |
|---|---|---|---|
| Highest | revised_output |
The user's own corrected version of the AI output | Shows exactly what "correct" looks like — no interpretation required |
| Medium | explanation |
A human-articulated reason the output was good or bad | Explains why, enabling root-cause analysis of prompt or model issues |
| Lowest | sentiment ("like" / "dislike" / "neutral") |
A binary or ternary rating | Directional only — tells you that something was wrong, not what |
If you can only collect one signal, prefer whichever is cheapest for your users to give without
degrading its meaning — a forced explanation on every interaction tends to produce low-quality,
copy-pasted text, which is worse than a clean sentiment alone. The goal is matching collection
effort to genuine signal, not maximizing the number of fields populated per submission.
When a user directly edits an AI-generated result and saves the correction, that's the single
highest-signal event you can capture — send it as revised_output rather than discarding it or
reducing it to a sentiment: "dislike".
Common feedback-structuring mistakes
These patterns look like they're capturing feedback, but actually degrade signal quality by misusing a field for something it wasn't designed to hold.
Stuffing a canned reason into explanation
The mistake: An automated dislike action (e.g. a thumbs-down button click, or a background
job inferring dissatisfaction from a retry) writes something like explanation: "User clicked
dislike" or explanation: "Auto-flagged: response discarded".
Why it's wrong: explanation is meant to carry a human-articulated reason — "the tone was
too formal for our support chat" or "missed the discount code in the second paragraph." A canned
string that just restates the sentiment adds no information beyond what sentiment already
conveys, but it does get counted as "explanation present" in feedback-quality metrics, silently
inflating your apparent signal quality without any real benefit.
The fix: If the dislike is automated and no human reasoning is available, submit:
{
"sentiment": "dislike"
}
Leave explanation blank. An empty explanation on a sentiment-only submission is accurate;
a populated one that just repeats the sentiment is misleading.
Using revised_output for something that isn't a correction
The mistake: Populating revised_output with a value the user didn't actually produce or
approve — for example, a system-generated reformatting of the AI output, a translated copy, or a
default fallback value substituted when the AI call failed.
Why it's wrong: revised_output is the highest-weighted signal in scoring (see
Feedback Scoring Guide) precisely because it represents deliberate human
correction. Feeding it non-corrective content dilutes revision-score calculations across your
whole workload and can make a template look like it needs heavy editing when it doesn't.
The fix: Only set revised_output when a human has actually edited and confirmed a change to
the AI's output. If your pipeline reformats output for display purposes, that's a presentation
concern — don't feed it back as feedback.
Marking synthetically-extracted feedback as creator_type: "human"
The mistake: When feedback is derived by an LLM analyzing a completed chat transcript (i.e.
data_source: "synthetic_extraction"), also setting creator_type: "human" because a human was
present in the underlying conversation.
Why it's wrong: creator_type should describe who or what authored the feedback judgment,
not who was present in the source conversation. Mislabeling synthetic feedback as human-authored
makes it indistinguishable from feedback a person deliberately submitted, which skews any
downstream analysis that weights or filters by creator type.
The fix: Keep data_source: "synthetic_extraction" paired with creator_type: "agent" (or
leave creator_type as its default, "unknown", if the extracting process doesn't have a strong
opinion). Reserve creator_type: "human" for feedback a person directly submitted through a UI
control, form, or explicit API call attributable to that person.
Treating neutral as a real opinion
The mistake: Defaulting every un-rated interaction to sentiment: "neutral" and treating that
as if the user affirmatively said "this was okay."
Why it's wrong: neutral is not a middle rating — it specifically means no explicit rating
was given. Submitting neutral in bulk for interactions nobody actually rated inflates your
total feedback count without adding real signal, and dilutes positive/negative rate calculations,
which explicitly exclude neutral from their denominator.
The fix: If no rating exists, don't submit a sentiment-only feedback item at all — omit the
field, or don't send a feedback item for that interaction. Only submit sentiment: "neutral" when
a user was explicitly presented with a rating choice and chose a neutral/no-opinion option.
Matching feedback to the right log
However you structure your feedback payload, always include enough information for Coolhand to
match it back to the LLM call that produced the output. At minimum, every submission needs one of
client_unique_id or original_output as a backstop; better matching is available via
llm_request_log_id (best) or llm_provider_unique_id (good). See
Understanding Feedback Match Rate for the full matching-tier breakdown — this
guide won't repeat that mechanics reference here.
Privacy: hash your creator identifiers
creator_unique_id identifies who submitted feedback, for deduplication and per-user analysis —
but it should never be a raw personal identifier. Emails, names, and other directly identifying
strings should be rejected outright.
Don't:
{ "creator_unique_id": "jane.doe@example.com" }
Do:
{ "creator_unique_id": "a94a8fe5ccb19ba61c4c0873d391e987982fbbd3" }
A SHA-256 hash of your internal user ID is the recommended approach — it's stable (the same user always produces the same hash, enabling deduplication) without exposing any personal information.
Restructuring your feedback collection
If a review of your feedback data suggests your collection strategy needs to change — for example,
your explanation field is consistently populated with auto-generated text, or your workload has
a healthy submission volume but very little revised_output — the fastest path to fixing your
integration is the /feedback-collection Claude Code skill
(Coolhand-Labs/feedback-collection-skill).
It plans and implements a feedback-collection strategy tailored to your codebase, prioritizing the
same signal-quality hierarchy described above.
Install it with:
/plugin marketplace add Coolhand-Labs/feedback-collection-skill
/plugin install feedback-collection@coolhand
Then run /feedback-collection (optionally scoped to a path, e.g. /feedback-collection
src/ai/) to have it review and restructure your feedback calls in place.