Monte Carlo's Agent Trust gives you self-improving Databricks Agent Bricks and Mosaic AI Agent Framework agents. Bayes, Monte Carlo's AI teammate, finds what's going wrong in real user conversations, traces each problem to its cause, and hands your team a fix to review. Turn on trace storage and Monte Carlo reads your MLflow traces through your existing Databricks SQL Warehouse connection. There's nothing to deploy.

What is Databricks Agent Bricks?

Databricks Agent Bricks is Databricks' managed platform for building and deploying AI agents — from ready-made templates like Knowledge Assistant to custom code-based agents. Teams can create agents that answer questions, search enterprise data, and orchestrate tools — all within their Databricks workspace. Agent runs are traced with MLflow, capturing conversation history, tool execution, LLM planning, and response generation.

Monte Carlo supports both kinds of Databricks agents, and labels them accordingly in its agent list:

• Knowledge Assistant (KA) — a managed Agent Bricks template.
• Custom agent (MLflow SDK) — your own Mosaic AI Agent Framework agents instrumented via MLflow (autolog or manual tracing).

Why connect Agent Bricks to Monte Carlo?

MLflow records what your agent did. Monte Carlo tells you what's going wrong and how to fix it.

• Finds failures in real conversations. Bayes reads every trace and surfaces recurring problems, including quiet ones that return a normal-looking response: retry loops, repeated tool calls, empty answers. You don't need to build a golden dataset first.
• Diagnoses from the traces you already have. Bayes works from the MLflow trace data already in Unity Catalog, with no extra grants beyond this setup.
• Hands you a fix, then checks that it worked. Each finding comes with a proposed change for your team to review. After it ships, Bayes watches new conversations to confirm the problem stopped.
• Separates agent problems from data problems. Monte Carlo already monitors your Databricks data, so a bad answer can be traced back to the tables behind it.

How it works

Databricks logs your agent's MLflow traces automatically, but they reach Unity Catalog only once you turn on trace storage. Monte Carlo then reads them from a trace table in Unity Catalog:

Monte Carlo reads this table with the principal already attached to your SQL Warehouse connection, so there are no extra credentials.

Set up

Prerequisite: an existing Databricks integration in Monte Carlo with a SQL Warehouse connection.

Step 1. Turn on trace storage

How you turn it on depends on the agent:

• Knowledge Assistant templates: enable trace syncing (Delta sync) on the template's MLflow experiment.
• Custom agents deployed with agents.deploy(): enable Production Monitoring on the agent's MLflow experiment.
• Custom code-based agents: set the experiment's trace location in your agent code. Requires MLflow 3.11+.

import os
import mlflow
from mlflow.entities.trace_location import UnityCatalog
 
os.environ["MLFLOW_TRACING_SQL_WAREHOUSE_ID"] = "<sql-warehouse-id>"  # SQL warehouse used for trace storage
mlflow.set_experiment(
    experiment_name="<your-experiment-name>",
    trace_location=UnityCatalog(
        catalog_name="<catalog>",
        schema_name="<schema>",
        table_prefix="<table_prefix>",
    ),
)

For full details, see Databricks' guides for Production Monitoring and MLflow Traces in Unity Catalog.

What happens before and after:

• Until trace storage is on, traces stay in the agent's MLflow experiment and never reach Unity Catalog, so Monte Carlo can't read them. Monte Carlo still discovers and lists the agent but flags it as having no trace table.
• Once it's on, traces from new agent runs land in the Unity Catalog table and appear in Monte Carlo within about 30 minutes.

Step 2. Group turns into conversations (custom agents)

Stamp each trace with a session ID: your application's conversation identifier, such as a chat thread ID, reused across every turn.

import mlflow
 
mlflow.update_current_trace(metadata={"mlflow.trace.session": "<your-conversation-id>"})

Without it, each turn appears as a standalone conversation.

Step 3. Tag workflows and tasks (custom agents, optional)

Tag MLflow spans with montecarlo.workflow and montecarlo.task to unlock facet filtering, segmentation, and monitor scoping in Monte Carlo.

• workflow names the high-level business flow.
• task names a sub-step within it.
• Tags apply only to the span they're set on. Child spans, including autolog-generated LLM spans, don't inherit them, so set them on each span you want to filter, segment, or monitor.

import mlflow
 
# Option A — set on the current span
with mlflow.start_span(name="retrieve_context") as span:
    span.set_attribute("montecarlo.workflow", "support-bot")
    span.set_attribute("montecarlo.task", "retrieve-context")
    ...
 
# Option B — set on a traced function via the decorator
@mlflow.trace(attributes={
    "montecarlo.workflow": "support-bot",
    "montecarlo.task": "final-answer",
})
def answer(question): ...

Tagging is fully optional. Untagged spans leave Workflow and Task empty in Monte Carlo.

Step 4. Grant permissions

Grant these to the principal attached to your SQL Warehouse connection. A dedicated, least-privilege service principal, granted exactly these per-agent permissions rather than broad workspace access, is the recommended connection identity.

• Read access: the trace table is the bare <table_prefix> for Knowledge Assistant templates or <table_prefix>_trace_unified for custom agents. If the principal can already read the catalog and schema where traces land (the common case), nothing extra is needed.
• Picker access: the principal only sees agents it can see in Databricks; by default, just the ones it created or was granted access to.

Step 5. Add the agent in Monte Carlo

1. Go to Settings → Agent Observability and click Add.
2. Set Agent Type to Platform Agent.
3. Select a Databricks SQL Warehouse: the warehouse where your trace table lives.
4. Select an Agent from the discovered list. If it isn't listed, check picker access in step 4.
5. Click Import.

Step 6. Schedule a Reinforcement Loop digest

Open the agent, click Subscribe to digest on its issues list, and pick a cadence and an audience with an email or Slack recipient. Your team gets each finding as it lands: what's breaking, why, and the proposed fix.

After about a day of traces, Bayes sets up monitoring on your agent across context, performance, behavior, and output. For a rule of your own, describe it to Bayes or build one yourself.