Agent Trajectory Monitors
Agent Trajectory Monitors check which steps an agent takes, in what order, and how often. They alert when a required step is missing, a step happens too often or too rarely, steps occur in an unexpected order or combination, or a tool repeats the same call with the same result. For example, use them to catch a database write before a permission check. They work like Validation Monitors, but check individual steps (spans) within agent runs (traces).
Creating Agent Trajectory Monitors
Agent Trajectory monitors can be created by navigating to Add monitor → Agent trajectory.
Choose data
If your agent isn't in the Choose data → Agent list, Monte Carlo hasn't seen its traces yet, or the agent isn't in the domain selected for the monitor. For OpenTelemetry agents, check the setup. For a platform agent (Snowflake Cortex, Databricks), confirm the agent is listed on the Agent Observability settings page. For an agent whose traces land in a warehouse table, confirm that trace table is listed there.
- Agent: Select an agent to monitor. The agent appears here once Monte Carlo has detected its traces.
- Time Filter: Select a time window of last
hour,day, orweek. This monitor validates new traces that appear within this time window each time it runs.- Note: The time window cannot be greater than the schedule interval.
For repeat detection, use Traces scope. Entire conversations supports only occurs and does not occur conditions. Switching to it removes repeat and relationship conditions from the editor, so review the remaining conditions before saving.
Define alert conditions
Alert conditions determine when Monte Carlo should generate a new Alert based on the trajectory patterns you've configured.
Users can set alert conditions on workflow, task, and span_name. These can be selected from the span groups tree which appears when users clicks on Select span input field.
The following alert conditions are currently supported
- (workflow/task/span) occurs more than, less than, or exactly x times
- (workflow/task/span) occurs with (workflow/task/span)
- (workflow/task/span) occurs without (workflow/task/span)
- (workflow/task/span) occurs before (workflow/task/span)
- (workflow/task/span) occurs after (workflow/task/span)
- (workflow/task/span) does not occur before (workflow/task/span)
- (workflow/task/span) does not occur after (workflow/task/span)
- (workflow/task/span) repeats more than or exactly x times
Use repeats to detect consecutive calls to the same tool with matching input and output within one agent run. Use occurs to limit the total number of calls, regardless of their input or output.
- Note: Workflows can only be compared with other workflows, tasks with other tasks, and spans with other spans
Detect repeated tool calls
Repeat detection supports Snowflake, including Cortex agents; Databricks, including Genie; and ClickHouse.
For example, looking up five different orders can be legitimate work. Repeatedly looking up the same order and receiving the same answer may indicate a problem.
- Under Define alert conditions, choose Select span and select a workflow, task, or tool-call span. Repeats checks consecutive calls to the same tool within that selection. For an individual span identified as not being a tool call, the editor disables repeats with the message: "Repeats requires tool calls. Choose a tool call, task, or workflow".
- Select repeats. Its hint is same input and output, in a row.
- Choose more than or exactly, then enter the count. The count includes the first call: three matching calls in a row count as three.
- Finish the schedule and notification settings, then select Add. When editing an existing monitor, select Update.
Example: alert when lookup_order repeats more than two times.
| Condition | When it matches |
|---|---|
| repeats more than 2 times | The longest matching sequence has three or more calls. |
| repeats exactly 3 times | The longest matching sequence has three calls. A run with a longer matching sequence does not match, even if it also contains a separate three-call sequence. |
Use more than when you want to catch any sequence above an acceptable length. Its minimum count is 1, which alerts on two or more matching calls. The minimum count for exactly is 2.
Another tool call breaks the sequence. For example, lookup_order → send_email → lookup_order is not a repeat of lookup_order. Reasoning or planning steps between the calls do not break it. A change in the tool's input or output starts a new sequence.
Choose a threshold that fits the tool's expected behavior. A tool that polls for a job to finish may legitimately make the same call and receive the same answer several times. A repeat match identifies repeated work; it does not by itself prove that the agent is stuck.
Calls with missing, redacted, cut-off, or oversized input or output cannot match and break the sequence. The combined input and output limit is 16,384 characters, so repeated calls that return a large file can go undetected. See Details and edge cases below for matching rules and source-specific limits.
To manage the same rule in YAML, see the repeat-detection example in the Monitors-as-Code guide.
Define schedule
Select when the monitor should run. Agent Trajectory monitors run according to the schedule defined below.
- On a schedule: Input a regular, periodic schedule. Options for handling daylight savings are available in the advanced dropdown.
Send notifications
Select which audiences should receive notifications when an anomaly is detected.
Additional settings exist for setting the description of the monitor which will be set as the header for notifications sent to notification channels, also pre-setting a priority on any incidents generated by the monitor.
See here for more information on Audiences and Notifications. Learn more about setting Monitor tags.
Notes
Text in the Notes section will be included directly in notifications. The "Show notes tips" dropdown includes details on how to @mention an individual or team if you are sending notifications to Slack.
Notes support rich-text formatting, including bold, italic, underline, strike-through, lists, links, and code blocks. Rich-text channels display these styles, while text-only channels show a plain-text equivalent.
Monitor properties can be dynamically inserted into Notes through variables. Supported variables include Created by, Last updated at, Last updated by, Priority, Tags and—for Validation and Custom SQL monitors—Query Result.
Alerts and Notifications
Alerts
Agent Trajectory Monitors generate an Alert whenever traces breach the defined alert conditions.
Alerts from Trajectory Monitors include the following details:
- Number of traces that violated the conditions
- List of the invalid traces
- Historical graph of previous breaches
For a repeat condition, open Trajectory breaches → Matching traces. Each sampled row identifies a matching run and includes a repeat count column for the selected workflow, task, or span. For example, two rows with a repeat count of 3 mean that each of those runs had a longest matching sequence of three calls; they do not mean one sequence of six calls. The table is a sample and may contain fewer rows than the total number of matching runs.
Each demo run has a longest sequence of three matching lookup_order calls.
Notifications
Alert notifications are sent to your configured audiences and include:
- Description of the condition that was violated
- Number of traces that breached the alert condition
- Details about the most recent breach
Details and edge cases
How are inputs and outputs compared?
Matching ignores capitalization, repeated whitespace in text values, and JSON object key order. Array order and value types still matter: the number 1 differs from the text "1". A missing or blank value, null, or an empty array [] is excluded; an empty object {} is allowed. A redacted or cut-off field excludes the whole call. On ClickHouse and Databricks, values nested more than 64 levels deep are also excluded.
Which calls can break a repeat sequence?
Calls outside the selected agent or time window are not considered. Within that data, another tool call breaks the sequence even if it does not match the alert condition or an additional filter, or if it is nested inside the selected tool's call. An additional filter that excludes send_email does not turn lookup_order → send_email → lookup_order into a repeat. Late-arriving calls can change the result of a later check. The Monitors-as-Code guide explains the distinction between filters and agent_span_filters.
How are calls ordered?
Calls are ordered by their start time, then their span ID. Native Genie uses the saved statement order first. Calls with the same start time have a fixed order for counting, but that order does not prove that parallel calls were retries.
How do native Cortex and Genie calls differ?
Native Cortex excludes generated query IDs and durations from comparison. Native Genie excludes statement IDs and requires a complete captured result.
For native Cortex custom tools, matching uses the captured status and status description, not the full tool return value. Cortex server-skill calls without captured arguments cannot match. Cortex SQL text at the 8,000-character capture limit, when backed by a semantic model, and custom-tool status fields at the 4,000-character capture limit are excluded because they may have been cut off.
When Cortex or Genie data is read through a generic ClickHouse trace table instead, generated IDs remain part of the comparison and may prevent a match.
What limits apply to repeat queries?
Larger time windows, more calls, and larger input/output values increase query work. Start with the tool and time window you need. ClickHouse repeat queries have a 512 MiB memory limit; large runs or multiple repeat conditions can exceed it and fail the check.
For a custom ClickHouse connection, the read-only profile must allow the query settings max_block_size, max_threads, and max_memory_usage; a profile with readonly=1 rejects them, while readonly=2 allows settings within its configured constraints.
What happens when a run matches several conditions?
A run that matches several conditions still appears once in the sample. If a repeat-count cell shows NULL, that repeat condition did not match but another condition matched the run. NULL does not mean a repeat count of zero.
Updated 1 day ago