Skip to main content

What is a workload?

Last updated 6 min read

A Coolhand workload is a container that groups related prompt templates belonging to a single task, feature, or AI agent. While individual templates capture specific prompt patterns, a workload aggregates performance, cost, token usage, latency, error rates, and quality scores across all templates that make up an entire capability.

A Coolhand workload is a logical container that groups related prompt templates belonging to a single task, feature, or AI agent.

While an individual prompt template captures a specific prompt structure or version, an AI feature often involves multiple steps—such as input classification, tool invocation, summarization, and response generation. Workloads group these related templates together so you can observe, evaluate, and optimize the overarching agent or workflow as a cohesive unit.

How workloads relate to templates

Prompt templates represent prompt patterns in your application, whereas workloads represent the feature or agent running those prompts:

  • One workload, many templates: Every prompt template in Coolhand belongs to exactly one workload. A workload can contain multiple active templates as well as draft or deprecated versions.
  • Unified metrics rollups: In the Coolhand dashboard and API, viewing a workload rolls up metrics—such as total request counts, token consumption, total cost, average latency, error rates, and feedback scores—across all templates assigned to that workload.
  • Streamlined navigation: If a workload has only a single active template, navigating to the workload in the dashboard routes directly to that template's view.

When creating a template programmatically using an external key (see How do I create or update a template using an external key?), you must specify the workload ID or workload key that the template belongs to.

Application workloads vs. system workloads

Coolhand separates user-created application workloads from internal system workloads:

  • Application workloads: Workloads created for your application's features and agents, either automatically when templates are created or declared explicitly via the API and SDKs.
  • System workloads: Built-in workloads maintained automatically by Coolhand:
    • Unmatched: Captures LLM request logs that did not match any existing prompt template pattern.
    • Embedding Requests: Collects requests sent to vector embedding endpoints.
    • Ignored API Calls: Holds requests excluded from template matching.

System workloads are automatically excluded from evaluation sampling, correctness scoring, and optimization scans. By default, API queries also omit system workloads unless explicitly requested.

Workload maturity and lifecycle

Every workload carries a maturity status that indicates how settled the feature is:

  • Development: New workloads or workloads whose prompt templates are changing frequently. Coolhand prioritizes rapid feedback review on development workloads.
  • Maturing: Active workloads in production where quality scores, sentiment, or prompt revisions indicate the workflow is still being calibrated.
  • Stable: High-traffic, high-quality workloads where prompt templates have settled. Optimization agents focus on cost reduction and require stronger evidence before suggesting quality changes.
  • Dormant: Workloads with little or no recent traffic. Coolhand pauses automated optimization scans on dormant workloads and suggests archiving them after extended inactivity.

Coolhand recalculates workload maturity automatically every night. If you want to lock a workload's status, you can set a manual maturity override in the dashboard to pin it to Development, Maturing, or Stable.

When a workload is no longer in active use, you can archive it. Workloads that represent duplicate or overlapping tasks can also be merged together, consolidating their historical logs and templates into a single active workload.

Declaring workloads in logs

When ingesting requests via SDKs or the log ingestion API (POST /api/v2/llm_request_logs), you can declare the workload directly on the request payload. Providing a declared workload attaches the incoming log straight to the target workload and bypasses pattern-based template matching:

{
  "llm_request_log": {
    "raw_request": {
      "model": "gpt-4o",
      "messages": [{ "role": "user", "content": "Analyze this invoice..." }]
    },
    "declared": {
      "workload": {
        "key": "invoice-extraction",
        "name": "Invoice Data Extraction",
        "description": "Extracts structured data from uploaded invoices."
      },
      "template": {
        "key": "invoice-header-extractor",
        "name": "Invoice Header Extractor"
      }
    }
  }
}

If the declared workload key does not yet exist, Coolhand automatically creates it upon receiving the first request.

Managing workloads with the API

You can inspect and manage workloads programmatically using Coolhand's private API.

Listing workloads

Send a GET request to /api/v2/workloads to retrieve your account's workloads, ordered alphabetically by name:

curl -H "X-API-Key: $COOLHAND_PRIVATE_API_KEY" \
  "https://api.coolhandlabs.com/api/v2/workloads?include_metrics=true&include_templates=true"

Common query parameters include:

  • search: Case-insensitive substring match against workload names.
  • external_key: Filter to a workload with a specific declared external key.
  • reporter: Filter workloads reported by a specific tool or SDK.
  • include_templates: When true, includes an array of active templates with their prompt regex patterns.
  • include_metrics: When true, includes aggregated cost, token, latency, error rate, and feedback metrics.
  • days_back, since, until: Control the time window for the computed metrics (defaults to 28 days).
  • include_archived: Include archived workloads (default false).
  • include_system: Include system workloads like Unmatched and Embedding Requests (default false).

The endpoint returns pagination metadata in the X-Page, X-Per-Page, X-Total-Count, and X-Total-Pages response headers.

Example response:

[
  {
    "id": "wkld_8x92k1mz",
    "name": "Invoice Data Extraction",
    "description": "Extracts structured data from uploaded invoices.",
    "external_key": "invoice-extraction",
    "reporter": "app-backend",
    "archived": false,
    "system": false,
    "merged": false,
    "maturity": "stable",
    "maturity_override": null,
    "archive_suggested": false,
    "template_count": 3,
    "draft_template_count": 0,
    "log_count": 14250,
    "last_activity": "2026-09-24T14:30:00Z",
    "templates": [
      {
        "id": "tmpl_3f82kd9a",
        "name": "Invoice Header Extractor",
        "status": "published",
        "user_prompt_pattern": "^Extract header from:.*",
        "system_prompt_pattern": null
      }
    ],
    "metrics": {
      "request_count": 14250,
      "failure_count": 25,
      "error_rate": 0.0018,
      "error_rate_change": -0.0004,
      "avg_latency_ms": 1180.5,
      "total_cost": 42.85,
      "avg_cost_per_request": 0.003,
      "total_input_tokens": 12850000,
      "total_output_tokens": 1420000,
      "avg_input_tokens": 901,
      "avg_output_tokens": 100,
      "correctness_score": 92.4,
      "sentiment_score": 0.88,
      "days_back": 28
    }
  }
]

Creating or updating a workload by key

To create or update a workload programmatically with a stable external identifier, send a PUT request to /api/v2/workloads/by_key/{external_key}:

curl -X PUT \
  -H "X-API-Key: $COOLHAND_PRIVATE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Customer Support Agent",
    "description": "Multi-turn assistant handling tier 1 customer inquiries.",
    "reporter": "support-service"
  }' \
  "https://api.coolhandlabs.com/api/v2/workloads/by_key/support-agent-v1"

The external_key remains fixed once assigned. If you update the workload's display name or description later, any existing logs or SDK declarations pointing to that external key continue to resolve seamlessly.

Related articles