Skip to main content

Reading your logs and querying them programmatically

What every field on an LLM request log means, when batching is worth it, and how to query your observability data from an AI agent over MCP.

  • How can I get help articles in Markdown format?

    Every Coolhand help article, category page, and product update is available in a Markdown format suitable for programmatic use. You can get the Markdown version of any page by appending `.md` to its URL, or by sending an `Accept: text/markdown` header with your request.

  • How do I create or update a template using an external key?

    Create and manage templates programmatically using the `PUT /api/v2/llm_request_templates/by_key/{external_key}` endpoint. This allows you to assign a stable, user-defined external key to a template. Templates created this way are 'declared' and bypass regex matching. Instead, you associate logs with them by providing the external key in your log requests.

  • How do I list and analyze my workloads using the API?

    A Coolhand workload is a group of prompt templates that make up a single task or agent. Use the `/api/v2/workloads` endpoint to list your workloads and return performance and cost metrics for them. This allows you to programmatically monitor your AI agents and tasks from your own tools.

  • LLMs and System One models: what's different, and when to use which

    TypeSafe's Jev is the first of its kind: use an LLM when you need reasoning, prose or conversation; use a System One model when the step is a narrow judgment that happens at scale. Coolhand can help you analyze System One models as a part of your AI toolset.

  • Node.js/JavaScript SDK Integration Best Practices

    coolhand-node patches fetch and https.request at initialization time. Import order, module system (CJS vs native ESM), and when you construct AI client objects all affect whether that patch actually intercepts your calls. This guide covers the three most common ways Node/JS integrations silently stop being monitored, and how to verify interception in CI instead of trusting a 200 response.

  • Python SDK Integration Best Practices

    coolhand-python patches httpx and requests in the interpreter that imports it. That patch doesn't automatically exist in every worker process, survives fork() but not spawn(), and can interact with other instrumentation or logging setup. This guide covers the process-model and ordering pitfalls that leave Python integrations silently unmonitored.

  • Ruby SDK Integration Best Practices

    coolhand-ruby patches Net::HTTP via Module#prepend and requires an explicit Coolhand.configure call — it does not auto-initialize on require. Require order, initializer placement, and duplicate provider gems all affect whether monitoring actually captures your AI calls. This guide covers the Ruby-specific ordering pitfalls to check first.

  • What does the 'decision' output modality mean?

    The 'decision' output modality is a Coolhand-specific term for models that return a structured choice rather than generative content. Models with this modality, like TypeSafe Jev, are designed to provide classifications, scores, or other structured data based on a schema you provide in the prompt.

  • What is Coolhand's 90-day data retention policy?

    After 90 days, Coolhand automatically removes the content of your LLM logs and feedback, including prompts, responses, and explanations. De-identified records and metadata, such as token counts, latency, and scores, are retained. This process is irreversible and applies to all customers.

  • What every field on an LLM request log means

    Coolhand now captures additional details about your LLM requests. The `wire_format` field indicates the specific API format used (e.g., `anthropic_messages`, `bedrock_converse`) for providers that support multiple formats. The `ingest_evidence` field records detected modalities (like text, audio, or image) and observed capabilities (such as tool_calling or reasoning). A new `system_one` request type is also available for specific system-level requests.

  • When should I use batch requests?

    Batch APIs process a set of independent LLM requests asynchronously, usually within 24 hours, at roughly half the per-token price. They suit any workload where nobody is waiting on the result. Coolhand flags templates whose traffic pattern suggests batching would cut cost without hurting quality.