Skip to main content

Collecting feedback and writing better prompts

How to capture high-signal human feedback on AI output, how Coolhand scores and attributes it, and the prompt patterns that hold across every provider.

  • How should I collect feedback on AI output?

    The highest-signal feedback is the edit a user makes to an AI output, then their written explanation, and only then a thumbs up or down. Send the request log id when you have it and the raw response text when you do not, plus a stable reviewer identifier either way, so every submission can be attributed.

  • What is feedback match rate?

    Match rate is the share of your feedback submissions that Coolhand successfully linked back to the LLM request log that produced the output. Unmatched feedback is still stored, but it cannot be attributed to a prompt, so it never reaches the analysis that turns feedback into a proposed fix.

  • How is feedback quality scored?

    Coolhand turns raw feedback into two numbers: a sentiment score drawn from likes, dislikes, and explanations, and a revision score measuring how much a user changed the output. Together they separate outputs people disliked from outputs people had to rewrite, which are different problems with different fixes.

  • What is partial feedback?

    Partial feedback rates a specific section of an AI response instead of the whole thing. You send the highlighted text and its character offsets, and Coolhand attributes the sentiment to that passage — so a mostly-good answer with one bad paragraph is recorded as exactly that, rather than as a dislike.

  • Creators vs. reviewers

    A creator is one raw identifier attached to a single feedback submission. A reviewer is the person behind it, assembled automatically by grouping every creator identifier and fingerprint belonging to the same individual. Reviewer-level analytics tell you how many people gave feedback, not how many sessions did.

  • Prompt engineering best practices

    Prefer your provider's structured-output API over asking for JSON in prose, keep instructions in a system prompt rather than inline with the data, delimit variable input explicitly, and phrase rules as what to do rather than what to avoid. These patterns hold across every provider Coolhand ingests.