Pre-prod agent monitors allow you to run agent evaluations on a static golden set of prompts or conversations. These evaluations serve as repeatable tests executed against a live production or pre-production environment, enabling CI/CD pipeline gating and tracking agent performance across builds.

Why use pre-prod monitors?

When deploying LLM-powered agents, you need confidence that code changes don't degrade agent quality. Pre-prod monitors enable:

• Regression testing — Run the same golden prompts against each build to detect quality regressions before deployment
• CI/CD gating — Automatically fail pipelines when agent evaluation scores drop below thresholds
• Performance tracking — Track agent quality metrics across builds over time

How pre-prod monitors differ from standard agent monitors

Setup guide

1. Instrument your agent

First, set up the Monte Carlo OpenTelemetry SDK, if you already haven't.

When invoking your agent during CI builds, pass ci_build_id and optionally expected_output as metadata. These metadata attributes will be available in the attr_map of your agent's traces. When configuring the monitor (see below), you can filter by these attributes.

If using LangGraph Server, pass these values in the metadata field of your request:

POST /threads/{thread_id}/runs
{
  "assistant_id": "your-assistant",
  "input": {...},
  "metadata": {
    "ci_build_id": "build-127",
    "expected_output": "Expected response text"
  }
}

If using a custom API layer, use mc.create_span_with_attributes to propagate attributes to child spans. Each key becomes a montecarlo.<key> attribute. Use the keys below: Monte Carlo reads the expected output only from montecarlo.association_properties.expected_output.

import montecarlo_opentelemetry as mc

@app.post("/analyze")
def analyze_sync() -> dict[str, Any]:
    body: dict[str, Any] = app.current_event.json_body or {}
    attributes = {
        "association_properties.ci_build_id": body["ci_build_id"],
        "association_properties.expected_output": body.get("expected_output", ""),
    }

    with mc.create_span_with_attributes("analyze_sync", attributes):
        return _analyze_handler(body)

Alternatively, pass attributes via the config argument when invoking a LangGraph:

graph.ainvoke(
    input={...},
    config={"ci_build_id": "build-127", "expected_output": "..."}
)

2. Create a pre-prod agent monitor

Follow the standard agent monitor configuration steps with these key differences:

• Filter Spans By ci_build_id: Under "Filter spans by"
  • Select attr_map and select the montecarlo.association_properties.ci_build_id attribute
  • Set the value to {{ci_build_id}}. The {{ci_build_id}} syntax creates a runtime variable that you provide when triggering the monitor via the SDK.
• Group data: For "Group spans by" (or "Group traces by" / "Group conversations by"), select "none" instead of "ingestion time", since pre-prod evaluations should assess all traces for a given build rather than grouping by time. Only Agent Metric and Agent Evaluation monitors offer "none", so use one of those for pre-prod.
• Define Manual Schedule: Under "Define schedule", set "Run monitor" to "Manually" instead of a recurring schedule, since the monitor will be triggered programmatically from your CI pipeline.

3. Create your CI build script

Your CI script orchestrates the pre-prod evaluation flow:

1. Execute golden prompts — Invoke your agent with each prompt, passing ci_build_id and expected_output in the request
2. Wait for traces — Poll until all traces have arrived
3. Trigger the monitor — Call runMonitor with ci_build_id as a runtime variable
4. Poll for completion — Wait until the monitor execution finishes
5. Gate deployment — Check the breached status and fail or pass the pipeline accordingly

The Pycarlo SDK provides components for building your CI script. See the sample_agent_ci_build_script.py file for a complete reference implementation.