Skip to main content

When should I use batch requests?

Last updated 4 min read

Batch APIs process a set of independent LLM requests asynchronously, usually within 24 hours, at roughly half the per-token price. They suit any workload where nobody is waiting on the result. Coolhand flags templates whose traffic pattern suggests batching would cut cost without hurting quality.

Batch APIs let you submit a large set of independent LLM requests for asynchronous processing, typically at around half the cost of standard per-token pricing in exchange for a delayed turnaround (usually within 24 hours). This guide covers when batching is a good fit, when it isn't, and how to structure a workload so it can take advantage of it.

What a Batch API is

Instead of sending one request and waiting for one response, you submit a whole set of requests together (often as a JSONL file or an array of requests). The provider processes them asynchronously — outside the normal request/response cycle — and you poll for completion or receive a callback, then download the results once the batch finishes.

The tradeoff is consistent across providers:

  • ~50% of standard per-token pricing
  • Results within 24 hours (occasionally longer, depending on the provider and current capacity)
  • No synchronous response — your application can't wait on the call in the same request cycle

Good fit

Batch APIs work well for workloads that are:

  • Asynchronous by nature — nothing on the other end is waiting in real time for the result
  • Non-interactive — no user is watching a chat window or expecting an immediate reply
  • Delay-tolerant — a same-day or next-day turnaround doesn't break anything downstream

Typical examples: bulk document classification, nightly data extraction, running evals over a large sample set, backfilling summaries or enrichment over historical records, and any scheduled or offline processing job.

Poor fit

Batch APIs are a poor fit for:

  • Chat or conversational workloads — a user typing a message needs a response in that same request, not tomorrow
  • Real-time or interactive workloads — anything a person or system is actively waiting on
  • User-facing latency-sensitive flows — even if the interaction isn't a chat, if a person is blocked waiting on the result, batching will break the experience

If your workload is a mix of both — most requests are fine to delay, but some subset genuinely needs an immediate response — consider splitting it into two separate workloads or templates so the batch-eligible portion can be optimized independently.

Provider-specific details

Batch support, limits, and turnaround vary by provider. See the relevant best-practices guide for exact details:

Some providers have no batch API at all — check the relevant guide before assuming support.

Structuring a batch-friendly workload

A few practices make a workload easier to batch effectively:

  • Keep each request independent. Batch items shouldn't depend on the output of another item in the same batch.
  • Give each request a stable identifier (a custom_id or key, depending on the provider) so you can map results back to the original item once the batch completes.
  • Keep the system prompt static across requests. Variable content belongs in the per-item user prompt, not the system prompt — this also keeps prompt caching effective if you use both.
  • Design for idempotent reprocessing. If a batch job fails partway or needs to be resubmitted, reprocessing the same item twice should be safe.

Is this workload actually latency-sensitive?

Our cost optimization agent looks at signals like the template type, your workload's stated purpose, and observed response times before recommending Batch API — but automated signals don't catch everything. If you get a Batch API recommendation for a workload that actually needs a fast response, you can flag it directly: open the workload and update its description to state the latency requirement (e.g. "results must return within seconds — used in a live user-facing flow"). Future optimization recommendations will take that into account.

Related articles