Docs/Llmops/Annotation

Human Annotation

Human annotation lets people on your team review AI output and record scores. Use it to build a quality baseline, check that LLM-as-a-Judge scores agree with human judgement, and label cases for datasets.

There are two ways to review by hand:

  • Annotate. Open a trace, observation or session and select Annotate to add scores.
  • Review in a case. Collect items in an Investigations case and work through them one at a time.
ℹ

Annotation queues are now cases on the Investigations page, under Discover AI → Investigations. Seeing cases needs the Tracing & Sessions permission at Read; creating cases and adding items need Read & Write. See Investigations.

Annotate a trace, observation or session

Open any trace, observation or session and select Annotate. Pick score configs and enter values. The scores appear on the item next to API and evaluator scores. Create score configs first in Project Settings > Scores / Evaluation: numeric (with optional minimum and maximum), categorical (with labels) or boolean. See Score configs.

Deployments that hide score and evaluation settings also hide Annotate. If you do not see it, ask your ANTS administrator.

Review items in a case

Cases replace annotation queues. Each case collects traces, observations and sessions for review:

  1. On Investigations, select New case, give it a name and an optional description, and optionally assign users under Advanced Settings.
  2. Add items with Add to case on a trace, observation or session, or with Add to Case in the bulk actions of the Tracing and Sessions tables. Items already in a case show In case(s).
  3. Open the case and select Review case. For each item, add notes as comments and select Complete to move on. The case shows how many items are reviewed.

Cases are not scored: the case form has no score configs, and the review page has comments and Complete only. Score an item with Annotate on the item itself.

Items are added by hand. There is no rule that adds items to a case automatically.

Using annotation scores

  • Compare human scores with LLM-as-a-Judge scores on the same traces to check your evaluators.
  • Filter traces by an annotation score to find bad answers, then add them to a dataset.
  • Chart annotation scores on a custom dashboard.

API

Cases, case items and assignments are available through the public API under /api/public/annotation-queues. Score configs are optional there: scoreConfigIds can be an empty list. When Investigations is turned off for the organization, these routes answer 403. See API Reference.

Tips

  • Write clear guidance in the score config description, for example what a 1 and a 5 mean, and in the case description.
  • Start with a small case to check that reviewers read the criteria the same way.
  • Have two reviewers score the same items now and then to check agreement.

Next steps

© 2026 ANTS Platform, Inc.Docs v1.0 · Last updated June 2026