Reinforcement Loop

Reinforcement Loop turns production agent traces into evidence-backed findings, diagnosed causes, and fixes you can act on.

Bayes, Monte Carlo's AI teammate, drives the Reinforcement Loop: it finds what's breaking your agents — errors and all, plus the quieter failures an agent just works around instead of surfacing — then traces each one to its root cause in the agent's own setup and carries it through to a fix. Improving an agent stops meaning hours spent reading traces one conversation at a time — Bayes does it all for you.

Monitors and evals tell you something broke. Bayes tells you why and how to fix it, pulling from every one of your agent's Issue clusters by default — eval-flagged or not — so nothing falls through the cracks.

Findings show up as Issues, the first thing you see when you open an agent.

How it works

Bayes moves every finding through the same cycle: drawn from your agent's Issue clusters, diagnosed against the agent's own configuration and definitions, turned into a fix, and followed as new matching conversations arrive. All it takes is access to those definitions — connect them once, and Bayes takes it from there.

StageWhat Bayes does
DetectFinds recurring problems across your agent’s conversations, including quiet failures the agent works around, and keeps adding new instances while the problem is open
ConfirmChecks the problem against runtime evidence, your agent's configuration, and recently merged pull requests
DiagnosePins down why it's happening — in the agent's prompt, tools, or configuration, or in the data feeding it — with the affected conversations attached
FixProposes a fix, verifies it against the evidence, and hands it to you to open as a draft PR, send to your ticket management system, or pass to your coding agent.
VerifyOnce an issue is Fixed, keeps matching new conversations against it. If the problem comes back, the issue is flagged Regressed and reopened automatically
flowchart LR
    surface["Surfaces a recurring problem"] --> verify["Verifies it against evidence"]
    verify --> explain["Explains the cause"]
    explain --> fix["Produces a fix"]
    fix --> follow["Follows new matching conversations"]
    follow -->|"recurs"| surface

Findings are categorized by failure type and prioritized by severity.

Every finding is grounded in what real users experienced in production — not scores against a curated dataset, which is what surfaces the failure modes no one could have written a dataset for. Bayes interprets and explains that evidence; it doesn't invent a finding or fabricate the evidence behind it. Analysis runs against trace content under your existing data-sharing settings for agent traces — see Agent Trust overview for where trace content is stored and what Monte Carlo receives.

Connecting your agent's definitions

Connect your agent's definitions and Bayes names the exact cause and proposes a fix, instead of just flagging that something's wrong — it's the only thing you need to set up. Without it, a finding still tells you something broke, but the root cause and fix stay out of reach until you connect it.

What it reads depends on the agent:

Snowflake Cortex agents

Bayes reads the agent configuration, the semantic views, the Cortex Search services the agent's tools use, and its skills, so it can name exactly what to change. The Monte Carlo service account role must be able to describe those objects. See Required Snowflake permissions.

Databricks (AI/BI Genie, Agent Bricks, and custom Mosaic AI agents)

Bayes diagnoses these from the data already flowing into Unity Catalog — no extra grants required beyond each integration's own setup:

  • AI/BI Genie — the questions, generated SQL, and answers Monte Carlo's collector brings in.
  • Agent Bricks and custom Mosaic AI agents — their MLflow trace data.

See AI/BI Genie and Agent Bricks and mosaic agents.

Custom code-based agents

For agents you build yourself, Bayes diagnoses against the agent's code. Connecting the repository (GitHub, GitLab, or Azure DevOps) is what makes that possible.

Add the integration, then set the connection and repository in Reinforcement loop settings, from the issues list on your agent's page, to unlock:

  • Code-level diagnosis — the cause is identified in the code that produced the behavior, not inferred from the trace alone.
  • Merged-PR cross-check — findings are cross-referenced against merged pull requests, so problems you've already fixed stop being raised.
  • Draft pull requests — a finding's suggested change can be opened as a draft PR against your repository, for your team to review and merge.

Choosing what Bayes investigates

Issue clusters are Reinforcement Loop's input, and every cluster is enabled by default — built-in failure patterns and any cluster built from your eval monitors alike — so there's nothing to configure to get started:

  • Built-in failure patterns — has errors, long-running, many turns, out of scope requests, repeated tool calls, token-heavy.
  • Eval-based clusters — built from one of your eval monitors (like grounded response), whether or not that monitor alerted.

Want to narrow it down instead? Open Reinforcement loop settings from the issues list on your agent's page, or click Manage input clusters on any issue, and toggle clusters on or off. Use "Select all" to reset.

Each cluster's exact definition is one click away via View criteria (and View monitor for eval-based ones).

Reinforcement loop settings — choose which built-in and eval-based Issue clusters Bayes investigates.

Scheduling digests

Findings can be delivered to a team on a schedule. On the issues list on your agent's page, click Subscribe to digest (it shows Digests with a count once you have one) and add a digest with:

FieldWhat it sets
CadenceHow often the digest is delivered
AudienceAn existing audience with at least one email or Slack recipient
NameShown in the digest list, and used as the title of the delivered message

Digests are configured per agent, so agents owned by different teams can route to different audiences.

Working a finding

Bayes never changes a running agent. It analyzes traces and proposes changes — as a draft pull request, a ticket, or a prompt you can take into your AI tool of choice — and nothing lands without your team reviewing and merging it.

Reviewing a finding

Each finding includes:

  • A summary — a plain-language account of what happened.
  • Impact — what it cost: wrong answers, wasted queries, user complaints, and so on.
  • Priority and timeline — current priority, when it was first detected, and when it was last reported.
  • Status: Open, In progress or Fixed, with a link to the fix when there is one. See Tracking a finding to resolution.
  • Activity: the issue's history, including status changes, tickets, pull requests and regressions.
  • Affected conversations — every conversation the issue was diagnosed from, each with a turn-by-turn account of how it showed up there and links straight to the cited turns.
  • Feedback history — every rating and comment your team has left on it.

Where the evidence doesn't establish a cause, Bayes says so rather than asserting one.

An issue found from conversations alone still tells you what's wrong. Connect your agent's definitions (see Connecting your agent's definitions) and Bayes adds the root cause and a fix you can act on (see Acting on a finding).

Acting on a finding

Each finding carries a suggested change Bayes proposed when your agent's definitions are connected. Route a finding wherever your team already works:

  • Open in your coding agent — sends a ready-made prompt, with the summary, evidence, links, and suggested actions, straight to the AI tool your team already uses (or copy the prompt to paste it yourself). Connect the Monte Carlo MCP server once and your tool can query Monte Carlo directly as it works.
  • Send to your issue tracker — open a ticket in Jira, Linear, or ServiceNow, where the integration is configured.
  • Draft a pull request — with a repository connected, open the suggested change as a draft PR against your own codebase, for your team to review and merge.

Rating a finding

Rate any finding with a thumbs up or thumbs down, then add an optional comment — your rating trains Bayes. A thumbs down also asks for a reason — whether it's not a real problem, has the wrong root cause or priority, is a duplicate, or something else — with a place to add more detail where it helps.

You can also rate an individual conversation within a finding — flagging it as not belonging, or as one where the agent recovered on its own. Rate from the app or through MCP.

Not planning to act on a finding? Click Ignore next to the rating buttons and give a reason. Ignoring is separate from status: it takes the issue out of your open list without marking it fixed, and you can restore it at any time.

Tracking a finding to resolution

Every issue has a status, so your team can see what's being worked on, what's been fixed, and whether the fix held.

StatusWhat it means
OpenBayes found the issue and it hasn't been fixed. New issues start here.
In progressSomeone is working on a fix.
FixedA fix has shipped. Bayes keeps watching for the problem to come back.

Changing an issue's status

Use Track status in the issue's Details card:

  • From Open: Mark in progress or Mark as fixed
  • From In progress: Mark as fixed

When you mark an issue fixed, you can add a Link to the fix, such as the merged pull request or the ticket. It shows on the issue as View the fix.

Changed your mind? Click Undo in the confirmation that appears, or in the status banner on the issue. Undo stays available until something else changes the issue, like a new ticket, a new fix, or a regression.

Status changes Bayes makes for you

If you work a finding through a draft pull request or a ticket, the status follows along:

What happensNew status
A draft pull request Bayes opened is mergedFixed
A linked ticket is closed as done in Jira, Linear or ServiceNowFixed
A ticket is created for the issue, a new draft pull request is opened, or a linked ticket is reopenedIn progress

Change the status from Track status in the issue's Details card, and add a Link to the fix if you have one. Click Undo to reverse a change. Ticket updates need status sync set up for that integration.

When a fixed issue comes back

Bayes keeps watching new conversations after a fix, starting 24 hours later. If the problem shows up in two of them, Bayes reopens the issue, flags it Regressed, and highlights those conversations. If one isn't really the same problem, rate it Not the same issue.

Activity and filters

  • The Activity card shows the issue's history: status changes, tickets, pull requests and regressions.
  • Filter the issues list by Open, In progress, Fixed, Ignored or All. Regressed issues get a badge.
  • Fixed and ignored issues are left out of digests.

FAQ

Why has Bayes found no issues?

Bayes needs enough conversation volume to establish a pattern. Check that traces are arriving: open the agent and look at the Health or Activity card for trace and conversation volume.

Why do my findings have no proposed change?

A finding can still show you what's wrong even when Bayes can't read your agent's underlying setup — it just won't have a root cause or fix until it can. See Connecting your agent's definitions for what each platform needs.

Can I be notified of findings?

Yes — set up a scheduled digest to an email or Slack audience. See Scheduling digests.

How do I know if a fix worked?

Mark the issue Fixed, or let Bayes do it when the draft pull request merges or the linked ticket closes. Bayes keeps checking new conversations. If the problem shows up in two of them after the first 24 hours, the issue is reopened and flagged Regressed. See When a fixed issue comes back.

What's the difference between Fixed and Ignored?

Fixed means a change shipped, and Bayes watches for the problem to come back. Ignored means you've decided not to act on it, and the status stays as it was. Both take the issue out of your open list and out of digests.

How do I reopen a fixed issue?

Use Undo if you just marked it fixed. Otherwise the issue reopens on its own if the problem comes back, or moves to In progress if its linked ticket is reopened or a new ticket or draft pull request is created.



Did this page help you?