AI Violations
AI Violations lists every place where AI Guardrails found something: which agent, which category, which values, and in which trace. Rows come from both guardrail modes:
- Scheduled Detection results from the PII, PCI, PHI, Ethics & Bias and Safety scans, shown with source Platform.
- Scheduled Detection results from custom detectors, Secrets & Credentials and Source Code, shown with source Scheduled.
- Real-time Enforcement checks that found a violation, shown with source SDK (Python SDK and REST API) or Connector (Portkey connector and Budget AI gateway).
Enterprise feature. Open it from Governance → AI Violations. It appears when AI Violations is turned on in Organization Settings → Features. The list contains flagged prompt and response content, so it requires the PII data permission (pii:read) in Roles & Access; among the built-in roles only Owner and Security & Compliance have it. The detected values themselves are hidden from users who are not allowed to see trace input and output.
Members who own agents see the violations of their own agents only, including the flagged content, without the PII data permission ("You see violations of the agents you own only."). They cannot mark detections or open the Detector accuracy report. See Agent owners.
A banner at the top says what needs attention across all time: agents with violations and no guardrail (Add guardrails), agents with violations that are not blocked (Turn on blocking), and one agent causing half or more of all detections. Select a line to open the agents in a side panel with a next step, which appears only when you can open the AI Guardrails policy it leads to.
The page has two tabs: Detections (the default, described below) and Drift and suspend activity.
What each row shows
| Column | Description |
|---|---|
| Timestamp | When the trace or check happened |
| Name | The agent. For Platform rows it links to the trace. |
| PII, PCI, PHI, Ethics/Bias, Safety | Which categories were found |
| Detections | The labels that were found, for example email, credit card number, a harm category, secrets.aws_access_key, sourceCode.python or Custom: <label> for a custom detector |
| Detected At | When the detection ran |
| Source | Platform (Scheduled Detection of PII, PCI, PHI, Ethics & Bias and Safety), Scheduled (Scheduled Detection of custom detectors, Secrets & Credentials and Source Code), SDK (Real-time Enforcement through the Python SDK or the REST API) or Connector (Real-time Enforcement through the Portkey connector or the Budget AI gateway) |
| Result | For SDK and Connector rows: Pass (Audit Only), Warning, Blocked or Sanitized. For Scheduled rows: Warning or Pass, because a scheduled scan never blocks or changes anything. |
| Webhook | Whether a policy violation webhook went out for the detection: Sent, Pending or Failed, with the number of deliveries when there were several and the breakdown in a tooltip, or "-" when none was sent |
Filtering
- Filter: All Detections, PII, PCI, PHI, Ethics/Bias, Safety, Custom (custom detector matches), Secrets & Credentials, Source Code or Tools (tool calls that broke an agent's tool permissions: not approved, expired, not allowed for the user, outside an argument rule, carrying sensitive data or over the rate limit). Tool violations from the ANTS SDKs have source SDK, and those from the Budget AI gateway and the Portkey connector have source Connector. The result is Blocked when the agent's policy blocked the call and Pass when it was only recorded.
- Agent: all agents or one agent.
- Time range: Last 24 hours, Last 7 days (default), Last 14 days, Last 30 days, or All time.
You can also open the list already filtered from other views, for example from an agent's governance profile.
Custom detector rows
The Custom filter lists the checks and scheduled scans in which a custom detector matched. The Detections column shows each detector that matched as Custom: <label>, where the label is the detector's Label on detections, and Detection Details lists them under Custom detectors. Real-time matches are shown with source SDK or Connector, and scheduled matches with source Scheduled. A scheduled match has the result Warning for detectors set to Block, Redact or Warn, and Pass for Audit Only detectors, because a scheduled scan never blocks or changes anything.
The matched values are not stored, so the list shows which detector matched, not what it matched. The same label is in the guardrail API result, and in policy violation webhooks and SIEM push alerts.
Secrets and source code rows
The Secrets & Credentials filter lists the checks and scheduled scans in which a Secrets & Credentials rule matched, and the Source Code filter those in which Source Code found code. These categories have no column of their own: the Detections column shows the rule labels, such as secrets.aws_access_key or sourceCode.python. The matched secrets and code are not stored with the detection, so there is no value to highlight.
Scheduled rows for these categories have source Scheduled and the result Pass. Real-time rows have the result the policy's action produced, for example Sanitized for a secret masked by Redact or Blocked for source code under Block.
Rule-based card number and SSN findings (see Card number and SSN checks) are listed under PCI and PII with the labels pci.card_number, pci.test_pan and pii.us_ssn.
Violation details
Select a row to open Detection Details:
- Each detected PII, PCI or PHI value with its label and confidence score, highlighted where it appears in the input or the output. The values are masked: only their shape and last four letters or digits show, for example
•••• •••• •••• 1234, and values of four characters or fewer are fully masked. Select Show detected values to see them for that row. Each reveal is written to the audit log (resource typetrace, actionrevealDetectionValues) and needspii:read. The button is not shown when trace input and output are hidden for your organization or project. - Add to case, for rows that come from a trace, adds the trace to an investigation case.
- For Ethics & Bias and Safety, the category that was flagged, for example S10 Hate Speech, Gender Bias, or Jailbreak / Prompt Injection.
- For custom detector matches, the detectors that matched, under Custom detectors.
- For SDK, Connector and Scheduled rows, the result, the direction (input or output) and the risk level.
- For Platform rows, View full trace details opens the complete trace.
- When Detection review is on, Review detections, with buttons to mark each label.
Review detections
Analysts record whether a detection was right, so noisy detectors can be tuned from evidence. Review is off by default.
Turn on Detection review
An Owner or a Security & Compliance member turns on Detection review in Organization Settings → Features. While it is off, the review buttons and the Detector accuracy report are hidden, and marks cannot be saved. Organizations that leave it off see no change.
Mark a detection
Open the detection
In AI Violations, select a row that has PII, PCI or PHI detections. Detection Details opens.
Find Review detections
Under Review detections, each detected label is listed once, for example email or credit card number.
Mark the label
Select False positive or Real detection next to the label. The selected button stays highlighted. To clear a mark, select the highlighted button again.
A mark applies to the label on that row, not to each value: if a row found three email addresses, one mark covers email for that row. Selecting the other button changes the mark. Each mark records who set it, and every mark, change and clear is written to the audit log.
Review covers PII, PCI and PHI labels. Ethics & Bias, Safety, Secrets & Credentials, Source Code and custom detector results cannot be marked yet, and a mark does not suppress future detections.
Who can mark
Marking needs the PII data permission (pii:read), because the analyst must be able to see the flagged content to judge it. Among the built-in roles, Owner and Security & Compliance have it.
Detector accuracy report
When Detection review is on, the Detector accuracy card at the top of AI Violations shows, for each PII, PCI and PHI label:
| Column | Description |
|---|---|
| Detector | The detection label, for example email |
| Flagged traces | Traces in which Scheduled Detection found the label in the selected window |
| Reviewed | Marks set or changed in the window, False positive and Real detection together |
| False positives | Marks set to False positive in the window |
| False-positive rate | False positives divided by Reviewed. It shows "-" until someone has reviewed the label. |
Select 7d, 30d (default) or 90d to change the window. The report covers the whole project; the list's Filter, Agent and Time range do not change it.
The rate measures accuracy among the detections analysts reviewed, not across all traffic, so review a representative sample before acting on it. Flagged traces counts Scheduled Detection results only; marks you set on Real-time Enforcement rows are counted in Reviewed and False positives. Use the report to find labels that need a higher confidence threshold or a narrower label selection in AI Guardrails.
The report needs the AI Violations permission at Read. Because it is on the AI Violations page, viewers also need pii:read, so in practice it is available to Owner and Security & Compliance.
Scheduled detection
Each agent's policy in AI Guardrails sets its own schedule: Hourly, Daily or Weekly. Each run scans the agent's traces received since the previous run, for the categories and labels selected in that policy and its custom detectors. Change the schedule, labels or confidence threshold in AI Guardrails.
Incidents
Scheduled detections open incidents automatically when these integrations are set up:
- ServiceNow: when the ServiceNow integration is connected with incident creation turned on, PII, PCI, PHI, ethics and bias, and jailbreak detections open ServiceNow incidents.
- PagerDuty: when the PagerDuty integration is connected with incident creation turned on, and PagerDuty Alerting is on in the agent's policy, detections open incidents on the selected PagerDuty service with the selected severity.
Real-time Enforcement records appear in this list but do not open incidents. To send violations to a security tool instead, use policy violation webhooks or SIEM push.
Violations can contain the actual sensitive values that were detected. Limit the pii:read permission to the people who need it.
Drift and suspend activity
Select the Drift and suspend activity tab to see, in one list, the project's drift alerts and every time an agent was suspended or resumed.
- Choose All, Drift or Suspend and resume, and a window of 7d, 30d (default) or 90d.
- Each row shows When, Type (Drift, Suspended or Resumed), Agent, Details, Severity, Notified and By. Details is the alert's description, or for a suspension or resume the number of keys blocked or unblocked. Notified shows where a drift alert was sent and when, or Not sent. By is the person who suspended or resumed the agent.
- Up to 100 rows are shown, newest first.
- Automatic suspensions by behaviour checks are not listed here; find them in the audit log under action
autoSuspend.
The view needs the AI Violations permission at Read.
Next steps
- AI Guardrails: choose which categories run, and how.
- Drift Alerts: detection rates are compared with each agent's declared controls.
- Risk Dashboard: see risk across all agents.