# Open sourcing our AI feedback loop playbook
Collecting useful human-in-the-loop feedback on AI outputs is a core part of what we do at Coolhand. We've spent the last year figuring out what actually works — which signals matter, how to capture them without disrupting users, how to match feedback back to the exact LLM call that produced it. Today we're open sourcing everything we've learned into a Claude Code skill.
## 95% of enterprise AI projects fail. Low-quality feedback loops are why.
MIT research puts the enterprise AI failure rate at 95%. The reasons vary, but a common thread runs through most failed projects: teams ship an AI feature, watch it quietly underperform, and never find out why. The model returns outputs. Users silently accept them, work around them, or churn. The signal that would fix the system disappears.
The teams that succeed treat AI deployment as a loop, not a launch. They collect feedback, match it to the calls that produced it, and feed it back into the system — with fixes being as simple as a prompt update created by an automated AI loop. This isn't complicated in principle. In practice, it's a side project that nobody has time to build well — instrumenting calls, designing capture points, hashing user IDs, building somewhere to store it all.
Today we're releasing the [Coolhand Feedback Collection Skill](https://github.com/Coolhand-Labs/feedback-collection-skill) to make that side project a single command.
## Why we're open sourcing this
Feedback and AI self-improvement loops are a big part of our business. We've spent countless hours thinking about and refining how to build the interfaces — both human and API — that power this. It's the core of what we do.
We believe more AI and agentic projects should succeed in production. That has real implications for how we operate. A rising tide raises all boats — the more teams that get good at closing the feedback loop, the better the whole ecosystem gets. We're committed to open source and to the broader goal of seeing the field move forward, not just our own platform.
## What it does
The skill is a [Claude Code](https://claude.ai/code) plugin. Install it once and run `/feedback-collection` in any project. It scans your codebase for LLM inference calls — OpenAI, Anthropic, LangChain, LlamaIndex, Ollama, Bedrock, and more — identifies where feedback could be collected, and proposes a concrete plan before it touches a line of code. You stay in the driver's seat; the agent does the legwork.
It doesn't just wire up a thumbs-up button. It designs feedback collection the way you'd want a thoughtful engineer to: preferring passive signals that require zero UI changes, prioritizing high-quality signals like an edited-and-saved output over a simple sentiment tap, and guaranteeing every feedback item can be matched back to the original LLM call. Privacy-safe handling of user identifiers is enforced out of the box — directly identifying strings are rejected in favor of hashed IDs.
## Use Coolhand, or self-host — your call
The skill works great as a front door to the Coolhand platform. Connect an API key and every feedback item flows into our analytics, where our agent team can surface patterns, identify underperforming prompts, and propose fixes as pull requests.
But you don't have to use us. The skill can also scaffold the same endpoints onto your own servers so you log and own your feedback data entirely. We'll help you set it all up. When you reach the point where you need advanced analytics across thousands of feedback points — pattern recognition, automated PR generation, cross-workload analysis — we'll be here.
## A field that's moving — and so will this
Feedback collection for AI systems is still an emerging discipline. We'll be updating the skill frequently as new best practices take shape, both from what we learn alongside our clients and from what emerges across the broader ecosystem. Because the skill pulls the latest SDK documentation each time it runs, your recommendations stay current without you chasing changelogs.
## Released alongside the Coolhand CLI
The skill ships hand-in-hand with the new [Coolhand CLI](https://github.com/Coolhand-Labs/coolhand-cli), which we're releasing and updating on the same cadence. The CLI makes it easy and token-efficient for any agent to authenticate with and use Coolhand. It also powers a beta capability: fetching fixes proposed by Coolhand directly, so your coding agent can apply them under your direction. You review, you approve, the agent does the work.
## Level up your AI game in minutes, for free
Add the marketplace and install the plugin in Claude Code:
```
/plugin marketplace add Coolhand-Labs/feedback-collection-skill
/plugin install feedback-collection@coolhand
```
Then run `/feedback-collection` in any project. Grab an API key at [coolhandlabs.com](https://coolhandlabs.com), and tell us what you think — the whole point, after all, is feedback.