Docs/Batch Inference/Vertex Ai Setup

Vertex AI Batch Inference Setup

The AI Optimizer can replay an agent's traces against a Gemini model on Vertex AI through Vertex AI batch prediction instead of real-time calls. This page lists what to create in Google Cloud and what to enter in the LLM connection.

Batch is turned on per optimizer run with Use batch inference model on the Select Models step. See Model Replay for how replay works for every provider.

How it works

For each Vertex AI comparison model, the platform collects the traces that have not already been replayed for that model.
It creates a new Cloud Storage bucket in the Google Cloud project of your service account key, in the connection's Location (us-central1 when empty), uploads the requests as a JSONL file and creates a Vertex AI batch prediction job for the model publishers/google/models/<model>. Only Google publisher models (Gemini) can be used.
The platform checks the job status about once a minute. When the job succeeds, it reads the output from the bucket, stores the comparison results and runs the LLM-as-a-Judge scoring.
Jobs are checked for up to 2 days after they are submitted. Results of a job that finishes later are not collected.

Each request uses a maximum of 1000 output tokens and temperature 0.7.

What the platform creates in your project

Resource	Name	Notes
Cloud Storage bucket, one per optimizer batch job	`finops-vertex-<timestamp>-<project suffix>-<random>`	Created in the connection's location with uniform bucket-level access.
Input and output files	`batch-inference/input/...` and `batch-inference/output/...` in that bucket	Contain the trace inputs and the model outputs.
Cloud Storage bucket for LLM-as-a-Judge batches	`<gcp project id>-vertex-batch`	Created once, if missing, when the judge model is on Vertex AI and its evaluations run as a batch.
Batch prediction job	`finops-<project suffix>-<job suffix>-<timestamp>`	Visible in the Vertex AI console under batch predictions.

The platform does not delete these buckets or files. Remove them according to your own retention policy, for example with a Cloud Storage lifecycle rule.

Prerequisites

A Google Cloud project with the Vertex AI API enabled.
Access to the Gemini models you want to compare, with batch prediction available in the location you will use. See Google: batch prediction for Gemini and Google: Vertex AI locations.
Permission to create a service account, grant it roles and create a key.

Setup

Step 1: Enable the APIs

bash

gcloud services enable aiplatform.googleapis.com storage.googleapis.com --project=PROJECT_ID

Step 2: Create a service account

bash

gcloud iam service-accounts create ants-batch-inference \ --display-name="ANTS Platform Batch Inference" \ --project=PROJECT_ID

Step 3: Grant the roles

The service account calls Vertex AI and creates, writes, reads and deletes Cloud Storage buckets and objects. Grant Vertex AI User and Storage Admin, or a custom role with the same permissions (storage.buckets.create, storage.buckets.get, storage.buckets.delete, storage.objects.create, storage.objects.get, storage.objects.list).

bash

gcloud projects add-iam-policy-binding PROJECT_ID \ --member="serviceAccount:ants-batch-inference@PROJECT_ID.iam.gserviceaccount.com" \ --role="roles/aiplatform.user" gcloud projects add-iam-policy-binding PROJECT_ID \ --member="serviceAccount:ants-batch-inference@PROJECT_ID.iam.gserviceaccount.com" \ --role="roles/storage.admin"

Storage Object Admin alone is not enough, because it cannot create buckets.

Step 4: Create a service account key

bash

gcloud iam service-accounts keys create key.json \ --iam-account=ants-batch-inference@PROJECT_ID.iam.gserviceaccount.com

⚠

The key gives access to your Google Cloud project. Store it securely and delete the local file after you paste it into the connection.

Step 5: Add the LLM connection

Go to Project Settings > LLM Connections > Add LLM Connection and fill in:

Field	Value
LLM adapter	`google-vertex-ai`
Provider name	Any name that identifies this connection
API Key	The full contents of `key.json`. The Google Cloud project is taken from its `project_id`; there is no separate project field.

Then click Show advanced settings and set:

Field	Value
Location (Optional)	The Vertex AI location, for example `us-central1` or `europe-west4`. When empty, `us-central1` is used.
I intend to use Batch Inference for LLM optimizer	Checked
Enable default models / Custom models	The Gemini models you want to use

Click Create connection. See LLM Connections for the other fields.

Step 6: Turn on batch in the AI Optimizer

When you create an optimizer, check Use batch inference model on the Select Models step. See AI Optimizer.

What is checked when you save

Create connection checks that the API Key is a valid service account JSON key and sends a short test message to the first model. With I intend to use Batch Inference for LLM optimizer checked, it also creates a temporary bucket named ants-platform-vertexai-test-<timestamp> in the selected location and deletes it right away, to confirm the service account can manage buckets.

Troubleshooting

Message or symptom	Cause
`Invalid GCP service account JSON key`	The API Key is not a complete service account JSON key.
`GCS bucket creation validation failed.`	The service account cannot create or delete buckets. Grant Storage Admin or the permissions listed in Step 3.
Permission denied when the job is created	The service account is missing Vertex AI User, or the Vertex AI API is not enabled.
Batch job fails in the Vertex AI console	The model name is wrong, the model does not support batch prediction in the selected location, or the project has no quota for it. Check the job's error in the Vertex AI console.

See Batch Inference Limits for the limits the platform applies.