Skip to main content

OpenAI API best practices

Last updated 3 min read

OpenAI renamed the system-instruction role to `developer` for its o-series reasoning models and silently maps `system` to it on newer models. Coolhand captures both. Send a system or developer instruction on every request so your prompts group into templates instead of landing on Unmatched.

Role deprecation: system → developer

OpenAI renamed the system-instruction role for its o-series reasoning models:

  • system — original role, works on all models. On o1-2024-12-17 and later o-series models, OpenAI maps it to developer silently. Note: o1-preview and o1-mini support neither role.
  • developer — preferred for o1-2024-12-17 and later. Accepted on GPT-4o too, but system is still more widely compatible across integrations.

Coolhand captures both as the System Prompt field. No action needed for existing integrations — both roles land in the same place.

Structured outputs

OpenAI offers two JSON output modes:

  • Structured Outputs (response_format: { type: "json_schema", json_schema: ... }) — strict schema enforcement with no extra fields or omissions. Requires gpt-4o-2024-08-06, gpt-4o-mini, or later models.
  • JSON Mode (response_format: { type: "json_object" }) — guarantees valid JSON but does not enforce a schema. Compatible with older models including gpt-3.5-turbo-1106 and gpt-4-*, but the model may produce any JSON structure.

Use Structured Outputs when schema adherence matters. JSON Mode is a fallback for older deployments that don't support schema constraints.

Docs: https://platform.openai.com/docs/guides/structured-outputs

Caching

Automatic — no explicit API required. OpenAI caches the longest stable prefix of your prompt.

  • Minimum prefix: 1,024 tokens
  • TTL: typically 5–10 minutes of inactivity; up to 1 hour maximum (treat as short-lived — a prompt inactive for 10 minutes may already be evicted)
  • Invalidation: any change to the cached prefix breaks the cache
  • Scope: per organization and per model version — different organizations or model versions do not share a cache

To maximize hit rate: put your entire system prompt first, followed by any stable context, and keep all variable content at the end of the prompt (or in the user turn). A single token change anywhere before the variable content resets the cache prefix.

Coolhand captures cache hits as Cached Input Tokens. Cache creation tokens are not reported by OpenAI (no separate creation event).

Batch processing

The OpenAI Batch API processes up to 50,000 requests from a JSONL file asynchronously, returning results within 24 hours at 50% of standard per-token pricing. Suitable for any workload that doesn't require real-time responses: evals, document processing, bulk classification.

  • Submit a JSONL file via the Files API, then create a batch job referencing the file ID
  • Poll the batch job status; download the output JSONL when complete
  • Each line in the input/output maps via a custom_id you supply

Docs: https://platform.openai.com/docs/guides/batch

Non-standard token fields

Field When present
Reasoning Tokens o-series models only — internal chain-of-thought tokens, billed as output
Audio Input Tokens Multimodal requests that include audio input

Latency

Three-tier fallback in order of preference:

  1. Assistants API runs — wall-clock time derived from completed_at - started_at timestamps in the run object. Note: the Assistants API is deprecated and scheduled for sunset on 2026-08-26; the Responses API is now the recommended path for stateful workflows.
  2. Chat Completions — openai-processing-ms response header (server-side processing time, does not include network transit)
  3. Fallback — collector-provided duration_ms when the header is absent (wall-clock)

Most requests use tier 2.

Related articles

  • Google Gemini best practices

    The Gemini API carries system instructions in a top-level `system_instruction` field rather than a system-role messag...

  • Azure OpenAI best practices

    Azure logs record your deployment name, not the underlying model version, so `my-gpt4o-prod` appears where you might ...

  • Anthropic API best practices

    Anthropic's Messages API carries the system prompt in a top-level `system` field rather than a message with a system ...

  • Google Vertex AI best practices

    Vertex AI's preferred request format puts system instructions in a top-level `system_instruction` field, outside the ...