Skip to main content

Google Vertex AI best practices

Last updated 4 min read

Vertex AI's preferred request format puts system instructions in a top-level `system_instruction` field, outside the `contents` array. Coolhand reads there first and falls back to system or developer roles inside `contents`, so role-based instructions may still be captured — but the top-level field is the reliable one.

System instructions are a top-level field

The Vertex AI preferred format uses a top-level system_instruction field (or systemInstruction in camelCase), outside the contents array. Coolhand reads from there first. As a fallback, Coolhand also checks the contents array for messages with a system or developer role — so role-based system messages may be captured, but system_instruction is the reliable path.

Structured outputs

Supported via the response_schema parameter (part of generation_config). Enforces a JSON schema on the response.

Docs: https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/control-generated-output

Thinking budgets

Vertex-hosted Gemini models use the same thinking_config.thinking_budget (2.5 models) / thinking_config.thinking_level (3.x models) mechanics as the direct Gemini API — see Google Gemini Best Practices for the per-model valid ranges, the two parameters' incompatibility on 3.x models, and why a shared/hardcoded budget value is the most common way to trip the HTTP 400 "thinking budget … is invalid" error.

Turning thinking off on Gemini 2.5 Flash and Flash-Lite

To disable thinking, send thinking_budget: 0 by itself. Do not pair a zero budget with include_thoughts: true: Vertex AI rejects that request with HTTP 400:

Unable to submit request because Thinking_config.include_thoughts is only enabled when thinking is enabled.

A budget of 0 on its own is accepted, and so is a budget of 0 with include_thoughts: false. This applies to gemini-2.5-flash and gemini-2.5-flash-lite alike, with or without a response schema.

gemini-2.5-pro is different: it cannot be turned off, so any request that sets a budget of 0 returns HTTP 400 ("The model does not support setting thinking_budget to 0"), whatever else is in thinking_config. Use a budget of at least 128 there.

The include_thoughts error most often comes from a client library or wrapper whose "disable thinking" option sends include_thoughts: true together with a budget of 0. If a library's off switch returns this error, look at the request body it sends, and set the budget directly if the library allows it.

Last checked 2026-10-23 against Vertex AI. This has not been confirmed on the Gemini API, so test there before relying on it.

Model retirement timing can differ between the two surfaces for the same model — Vertex AI's own model lifecycle documentation has listed retirement dates for some Gemini 2.5 models where the Gemini API's deprecations page had none announced. Check Vertex's model lifecycle page directly rather than assuming the two surfaces retire a model in lockstep.

Caching

Vertex AI supports the same implicit and explicit context caching as the direct Gemini API — see Google Gemini Best Practices for the full mechanics, including the two-part billing (a per-token storage fee charged hourly, plus the discounted per-token rate on cache hits) for explicit caching. Coolhand's inference API pricing catalog tracks both rates for Vertex-hosted Gemini models where confirmed.

Vertex returns a cachedContentTokenCount field, but Coolhand's ingestor does not currently extract it. Cached Input Tokens and Cache Creation Tokens are always blank for Vertex logs even when caching is active — this affects only per-request usage capture, not the catalog's published rates above.

Batch processing

Vertex AI batch prediction submits a JSONL input file to Cloud Storage, processes the requests asynchronously, and writes results to a Cloud Storage output path. There is no fixed completion window — jobs run to completion based on available capacity.

  • No per-request overhead; well-suited for large one-off processing jobs
  • Input and output stay in your GCP project (Cloud Storage buckets you control)
  • Typical completion ~24 hours; maximum queue time is 72 hours
  • Supports most Gemini models available on Vertex — tuned models must use a tuned endpoint MODEL_PATH rather than the global endpoint

Docs: https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/batch-prediction-gemini

Latency

Not available. Vertex AI does not include timing information in API responses, and Coolhand does not read a collector duration_ms for Vertex text logs. Latency will always be blank.

Related articles

  • Google Gemini best practices

    The Gemini API carries system instructions in a top-level `system_instruction` field rather than a system-role messag...

  • OpenAI API best practices

    OpenAI renamed the system-instruction role to `developer` for its o-series reasoning models and silently maps `system...

  • Anthropic API best practices

    Anthropic's Messages API carries the system prompt in a top-level `system` field rather than a message with a system ...

  • AWS Bedrock best practices

    AWS Bedrock exposes two API surfaces and Coolhand records them as separate sources: `bedrock` for the OpenAI-compatib...