Model Replay
Model replay is the step of an AI Optimizer run that produces the comparison outputs. For every trace in the run's snapshot, the platform sends the trace's input to each comparison model you picked and stores the answer, its cost and its latency, so the evaluator can score it next to the original output.
Replay calls run on your own provider credentials from Settings → LLM Connections. Your provider bills you for every replayed call, and the trace inputs are sent to that provider under your account. On credit-based plans each replayed trace also uses one credit (see Evaluation & Credits).
What is replayed
- The input of each trace in the snapshot. A text input becomes one user message. A list of messages, or an object that holds messages, is sent as a chat conversation.
- One model call per trace and comparison model. Replay does not rerun tools, retrieval or other steps of a multi-step agent; it compares the model's answer to the same input.
- Default generation settings: temperature 0.7 and up to 1,000 output tokens.
Each successful answer is stored as a new trace in your project, linked to the optimizer job and the comparison model, with its cost in USD, latency in milliseconds and token counts. A call that fails is not stored; it counts as an error for that model and shows up in the model's progress on the job page.
Supported providers
The comparison model list on the Select Models step is built from your LLM connections. For connections that use default models, only models that support the provider's batch API are offered; custom models you added to a connection are always listed.
| Provider (LLM connection) | Real-time replay | Batch replay | Notes |
|---|---|---|---|
| OpenAI | Yes | OpenAI Batch API | |
| Azure OpenAI | Yes | Azure OpenAI Batch | The model is your deployment name. Batch needs a deployment of type Global Batch, otherwise the job fails. |
| Anthropic | Yes | Message Batches API | Only Claude 3 and later models are offered. |
| Google AI Studio (Gemini API key) | Yes | Gemini batch | |
| Google Vertex AI | Yes | Vertex AI batch prediction | Batch supports versioned model names only, not aliases. See Vertex AI Setup. |
| AWS Bedrock | Yes | Bedrock batch inference | Batch needs a minimum of 100 traces and a batch IAM role and S3 bucket for the connection. See Bedrock Setup. |
Real-time or batch
On the Select Models step, Use batch inference model decides how replay runs.
| Real-time | Batch | |
|---|---|---|
| How | Direct API calls with retries | One batch job per comparison model, polled until it finishes |
| Provider price | Standard | Up to 50% cheaper on most providers |
| Speed | Results arrive as each call returns | Results arrive when the provider finishes the batch job, which can be much later |
| Latency recommendation | Yes | No. Batch latency does not reflect live latency, so only cost is compared. |
When you turn batch on, the platform checks that every selected comparison model supports batch for its provider. If one does not, you see Batch Inference Not Supported with the model names, and you can uncheck batch or pick other models. Provider size limits are listed in Batch Inference Limits.
The same setting also lets the LLM-as-a-Judge step submit its evaluations through the judge provider's batch API when there are at least 5 evaluations to run.
How replay cost is calculated
Replay cost uses the same model definitions as the rest of the platform. The model name is matched against your definitions (Bedrock regional IDs such as us.anthropic.claude-...-v1:0 and Azure batch deployment names are normalized first), and the price tier is chosen from the token usage, so long-context requests are charged at the long-context rate when the definition has one. A model without a price definition is costed at zero, which makes it look free; add a definition before you compare it.
Troubleshooting
| Symptom | Likely cause |
|---|---|
| A model shows trace generation errors | Invalid or expired key on the LLM connection, the model is not enabled for your account or region, or a provider rate limit. |
| Azure batch job fails | The deployment is not of type Global Batch. |
| Bedrock batch job does not start | Fewer than 100 traces in the run, or the connection has no batch IAM role and S3 bucket. |
| A comparison model costs $0 | No model definition matches the model name. |
Next steps
- Evaluation & Credits: how the replayed outputs are scored.
- Recommendations: how a comparison model wins.
- Provider API Keys: add the LLM connections replay uses.