General guidance for writing effective LLM prompts, regardless of provider. These patterns apply across all source APIs Coolhand ingests.
Structured outputs over JSON instructions
Asking a model to "respond in JSON" in your prompt is fragile — models may add prose before or after the JSON, wrap it in markdown fences, or silently omit fields under token pressure. Use your provider's native structured output API instead.
Why it's better:
- The schema is enforced at the API level, not by the model's instruction-following
- No need to parse or validate the response format in your application code
- Fields are guaranteed to be present (required fields won't be silently dropped)
- Eliminates a whole class of prompt-engineering effort
Provider support: See each provider's best practices guide for structured output availability and documentation links.
Use description fields to improve results
When defining a JSON schema for structured output, populate the description property on every field. Models use these descriptions to understand what each field should contain — they are effectively in-schema instructions.
Less effective:
{
"type": "object",
"properties": {
"score": { "type": "integer" },
"label": { "type": "string" }
}
}
More effective:
{
"type": "object",
"properties": {
"score": {
"type": "integer",
"description": "Quality score from 1–10. Use 1–3 for responses that miss the question, 4–6 for partial answers, 7–9 for complete answers, 10 for exceptional clarity and correctness."
},
"label": {
"type": "string",
"enum": ["positive", "neutral", "negative"],
"description": "Overall sentiment of the user's message, not the assistant's response."
}
}
}
Think of descriptions as mini-instructions scoped to each field. They are read at generation time and directly influence what the model produces.
Conflicting instructions between prompt and system prompt
When a user prompt and system prompt give contradictory instructions, model behavior is unpredictable — it may follow either one, blend them, or refuse. Common conflict patterns:
- System prompt says "always respond in English"; user prompt is in French and asks for a French response
- System prompt sets a persona; user prompt asks the model to ignore its persona
- System prompt specifies a response length; user prompt asks for something shorter or longer
- System prompt defines output format; user prompt asks for a different format
How to identify this in Coolhand: Review logs where the assistant response doesn't match the expected format or behavior. Check both the System Prompt and User Prompt fields — if they contain competing instructions, that's the likely cause.
Fix: System prompts should define stable constraints (persona, format, output language, safety rules). User prompts should carry only the variable content for that request. If a constraint needs to flex per request, make it a parameter substituted into the system prompt rather than an override in the user prompt.
Redundant and unnecessary instructions
Instructions that repeat across the system prompt and user prompt, or that restate things the model already knows, add tokens without adding value. This increases cost and can dilute the weight the model gives to the instructions that matter.
Common redundancy patterns:
- Restating the output format in both system and user prompt
- Reminding the model of its persona in every user turn of a multi-turn conversation
- Including boilerplate like "You are a helpful assistant" when it adds nothing specific
- Repeating constraints that were already set in the system prompt (e.g., "Remember, keep responses concise")
How to spot it: Compare the system prompt and user prompts side by side in Coolhand. If you find the same constraint stated twice, the user-prompt copy is almost certainly redundant.
Fix: State each constraint once, in the system prompt. User prompts should carry only what changes per request.
Tool use
Evaluate tool errors
Tool errors often indicate a mismatch between what the model thinks a tool does and what it actually does. When a tool call fails, check:
- Did the model pass the right argument types and structure?
- Did it call the tool at a point in the task where calling it makes sense?
- Is the error recoverable — did the model handle it gracefully in the next turn?
Repeated tool errors on the same tool suggest the tool description or parameter schema needs clarification, not just error handling in your application.
Evaluate tool misusage
A tool call that succeeds but shouldn't have been made is harder to catch than one that errors. Signs of misusage:
- The model calls a tool and then ignores its result
- The model calls the same tool multiple times with the same arguments
- The model calls a more expensive or destructive tool when a simpler one would do
- The model calls a tool to retrieve information it was already given in the context
Review tool call patterns in Coolhand logs to identify these cases. They typically point to unclear tool descriptions or a missing instruction about when tools should and shouldn't be used.
Write useful tool descriptions — and front-load key guidance
The tool description is the model's primary signal for when and how to use a tool. Treat it like a docstring that a senior engineer would write before handing off an API:
- Start with what the tool does and when to use it — the first sentence is weighted most heavily
- Explain when NOT to use it — this prevents the misusage patterns above
- Describe the output — what does a successful call return, and what does the caller need to do with it?
- Clarify argument semantics — especially for parameters that are ambiguous (e.g., is
ida user ID or a session ID?)
If you notice the model frequently misusing a tool, consider whether the key guidance should appear earlier in the description rather than later.
Prompt design for specific scenarios
Rubrics for scoring prompts
When asking the model to score something (quality, relevance, tone, etc.), vague criteria produce inconsistent scores. Define a rubric directly in the prompt or schema description.
Less effective:
Rate the response quality on a scale of 1–5.
More effective:
Rate the response quality on a scale of 1–5 using this rubric:
- 1: Does not address the question at all, or is factually wrong
- 2: Addresses the topic but misses the main question
- 3: Answers the question but is incomplete or unclear
- 4: Answers the question fully and clearly
- 5: Answers the question fully, clearly, and with notable insight or concision
Rubrics also make scores comparable across different evaluators and over time — a 3 means the same thing regardless of which model version produced it.
For automated scoring workflows, embed the rubric in the schema description field rather than repeating it in the prompt on every request. This keeps the prompt clean and the rubric cache-friendly.
Examples (few-shot)
Adding examples (few-shot prompting) is one of the highest-leverage techniques for improving consistency on classification, scoring, extraction, and formatting tasks. Guidelines:
- Place examples where they are stable and early — the key is that examples appear before variable content so the cached prefix can extend over them. The system prompt is a natural place for this, but examples early in the user turn also work; the caching benefit comes from positional stability, not specifically from the system field
- Include at least one example per class or edge case you care about — a single example of the "positive" class teaches less than one positive and one borderline case
- Format examples the same way as real requests — the model learns format from examples as much as it learns content
- Include a reasoning step in the example output before the final label or score when your task requires judgment — chain-of-thought examples produce more consistent judgments than direct-answer examples
For classification tasks: Include at least one example per category, plus one example of a case that might be ambiguous between two categories, with an explanation of why it falls where it does.
For scoring tasks: Pair the rubric with concrete examples at each score level. A rubric with no examples anchors to language; a rubric with examples anchors to observed behavior.