Agent Metric Monitor
Overview
Agent metric monitors track statistical metrics over an agent's spans β null rate, latency, token usage, and the like β and alert on ML-based or fixed thresholds. They use the same alert_conditions as the Metric Monitor; the difference is the source: an agent, not a table.
Reference scopeThis page covers MaC YAML configuration. For how agent monitors work, see Agent Monitors Overview. Alert conditions and the available metrics follow the Metric Monitor reference.
MaC key: agent_metric. Transforms are not supported here β for LLM-as-judge evaluation use the Agent Evaluation Monitor.
Quick Start
montecarlo:
agent_metric:
# Warehouse platform agent (e.g. Snowflake Cortex or Databricks)
- name: agent_prompt_null_rate
description: Null-rate of prompts on the support agent
agent: "ANALYTICS:AGENTS.support_agent" # <database>:<schema>.<name>
alert_conditions:
- metric: NULL_RATE
fields: [prompts]
operator: AUTO
schedule:
type: fixed
interval_minutes: 60
start_time: "2025-01-01T00:00:00+00:00"
aggregate_by: HOURConfiguration
string Β· required
The agent whose spans the monitor reads. Agent monitors name an agent instead of a table; the warehouse source is derived from it. The value takes one of two forms, told apart by its shape:
- Warehouse platform agents (e.g. Snowflake Cortex, Databricks) β a
<database>:<schema>.<name>reference, where<name>is the agent's display name when it has one (otherwise its underlying identifier). Notrace_table. - OpenTelemetry agents β the agent's bare
service_name. By default the OpenTelemetry trace store is resolved automatically; settrace_tableonly when the spans live in a specific warehouse table.
A value with a : and a dotted tail (db:schema.name) is read as a platform reference; a bare value is an OpenTelemetry service_name.
agent: "ANALYTICS:AGENTS.support_agent"string Β· optional
<database>:<schema>.<table> naming the warehouse trace table that holds an OpenTelemetry agent's spans. Required when the spans live in your warehouse, or when the warehouse holds more than one OpenTelemetry trace table. Omit it for platform agents and for agents resolved from the default OpenTelemetry trace store.
trace_table: "ingest:opentelemetry.traces"array of objects Β· required
Same shape as the Metric Monitor β a metric, optional fields, an operator (AUTO, GT, LT, range operators, β¦), and thresholds. See Available Metrics for metric names.
alert_conditions:
- metric: NULL_RATE
fields: [prompts]
operator: AUTOarray of objects Β· optional
Refines the monitor to specific spans. The agent is already set by agent, so it need not be repeated. Each entry sets one or more of workflow, task, span_name, each an object with a value.
agent_span_filters:
- workflow:
value: checkoutenum Β· optional
Accepted values: HOUR Β· DAY Β· WEEK Β· MONTH (uppercase)
Buckets metrics by the chosen interval.
aggregate_by: HOURboolean Β· optional Β· mutually exclusive
Raise the scoring grain from individual spans to whole traces (is_agent_trace_aggregation) or whole conversations (is_agent_conversation_aggregation). Setting both fails validation. Include the flag in the template to re-apply at the same grain β omitting it reverts to span grain.
is_agent_trace_aggregation: trueenum Β· optional
Accepted values: low Β· medium Β· high
Tunes how lax the ML-generated thresholds are for AUTO alert conditions.
sensitivity: mediumarray of strings Β· optional
Segment metrics by up to 5 fields (segment_fields) or one SQL expression (segment_sql). Use one or the other, or neither.
segment_fields:
- model_nameobject Β· optional
Same shape as the Metric Monitor β type (fixed, dynamic, manual), interval_minutes, start_time, timezone.
schedule:
type: fixed
interval_minutes: 60
start_time: "2025-01-01T00:00:00+00:00"The shared monitor envelope works the same as other monitor types:
namestringΒ· required β unique identifier in the namespace; renaming creates a new monitor.descriptionstringΒ· required β max 512 characters.warehousestringΒ· optional β UUID or name; overrides themontecarlo.ymldefault.connection_namestringΒ· optional β query engine to use within the warehouse.notesstringΒ· optional β shown in the UI, not in notifications.audiences/failure_audiencesarray of stringsΒ· optional β notification channels.priorityenumΒ· optional βP1βP5.tagsarray of objectsΒ· optional βname(required) +value(optional).data_quality_dimensionenumΒ· optional βACCURACYΒ·COMPLETENESSΒ·CONSISTENCYΒ·TIMELINESSΒ·UNIQUENESSΒ·VALIDITY.domainsarray of stringsΒ· optional (required on accounts created after January 2025) β domain for the monitor.
Deprecated fields
| Field | Use instead |
|---|---|
resource | warehouse |
domain_uuids | domains |
labels | audiences |
Examples
Platform agent metric (Snowflake Cortex / Databricks)
Null-rate metric on a warehouse platform agent, referenced by its <database>:<schema>.<name> identity. No trace_table β the source is derived from the agent.
montecarlo:
agent_metric:
- name: cortex_prompt_null_rate
description: Null-rate of prompts on the Cortex agent
agent: "ANALYTICS:AGENTS.support_cortex_agent"
alert_conditions:
- metric: NULL_RATE
fields: [prompts]
operator: AUTO
schedule:
type: fixed
interval_minutes: 60
start_time: "2025-01-01T00:00:00+00:00"
aggregate_by: HOUR
priority: P3
domains:
- my-domainOpenTelemetry agent with a span filter
Addressed by the agent's bare service_name; the default OpenTelemetry trace store is resolved automatically. A span filter narrows the monitor to one workflow.
montecarlo:
agent_metric:
- name: otel_checkout_latency
description: Latency of the checkout workflow
agent: my-otel-agent # the agent's service_name
agent_span_filters:
- workflow:
value: checkout
alert_conditions:
- metric: AVERAGE
fields: [duration_ms]
operator: AUTO
schedule:
type: fixed
interval_minutes: 60
start_time: "2025-01-01T00:00:00+00:00"
aggregate_by: HOURTrace-level aggregation
Score whole traces rather than individual spans by setting is_agent_trace_aggregation.
montecarlo:
agent_metric:
- name: trace_token_usage
description: Total token usage per trace
agent: "ANALYTICS:AGENTS.support_cortex_agent"
is_agent_trace_aggregation: true
alert_conditions:
- metric: SUM
fields: [total_tokens]
operator: AUTO
schedule:
type: fixed
interval_minutes: 60
start_time: "2025-01-01T00:00:00+00:00"
aggregate_by: HOURTroubleshooting
Agent source
- Omitting
agent. Every agent monitor requires anagent. A monitor without one fails validation. - Platform reference not found. A
<database>:<schema>.<name>reference must match an agent registered in the target account and warehouse. Cross-account: register the agent in the target account first. - Ambiguous platform reference. Export prefers an agent's display name, which is not unique β if two agents in the same schema share a display name, the reference is ambiguous and rejected. Use the agent's underlying endpoint identity instead.
- Span-filter agent disagrees with
agent. If you set anagentsub-field insideagent_span_filters, it must name the same agent as the top-levelagent. Omit it β the agent is already set.
Trace tables
- Setting
trace_tablefor a default-store agent. When the OpenTelemetry trace store is resolved automatically, naming it is redundant and rejected. Omittrace_table. trace_tablematches no table. The named table must exist in the target warehouse, resolved by its<database>:<schema>.<table>id.trace_tablenames a platform agent's table. Platform agents are authored by theiragentreference, not their trace table.- No OpenTelemetry trace store provisioned. A bare
service_namewith notrace_tablerequires the warehouse to have a provisioned OpenTelemetry trace store. If it doesn't, settrace_tableto name the warehouse table holding the spans.
Metric-specific
- Using
transforms. Transforms are not supported onagent_metric. Use the Agent Evaluation Monitor. - Setting both aggregation flags.
is_agent_trace_aggregationandis_agent_conversation_aggregationare mutually exclusive β set at most one. - Forgetting PUT semantics on updates. Updating a monitor replaces its full configuration β fields you omit revert to defaults, including the aggregation flags. Always specify the complete desired configuration.
