Monte Carlo's Agent Trust gives you self-improving AI/BI Genie spaces. Bayes, Monte Carlo's AI teammate, finds where Genie is getting answers wrong in real user conversations, traces each problem to its cause, and hands your team a fix to review. Monte Carlo installs and runs a small managed collector for you, so all you do is grant a few permissions and import the space.

What is AI/BI Genie?

AI/BI Genie is Databricks' natural-language analytics interface. Business users ask questions about their data in plain English, and Genie generates SQL, runs it against Unity Catalog, and returns an answer alongside the query it ran. Each Genie space is scoped to a set of tables. Unlike Agent Bricks, Genie keeps its conversation history behind an API rather than in a Unity Catalog table, which is why Monte Carlo uses a collector (explained below).

Why connect Genie spaces to Monte Carlo?

Genie answers your users' questions. Monte Carlo tells you which answers are going wrong and how to fix them.

• Finds failures in real conversations. Monte Carlo captures every turn: the question, the generated SQL, a sample of the results, and the final answer. Bayes surfaces recurring problems across them, including errors, repeated retries, and questions Genie couldn't answer or hallucinated. You don't need to build a golden dataset first.
• Shows what users ask that the space can't answer. Out-of-scope questions point to the table or permission the space is missing.
• Hands you a fix, then checks that it worked. Each finding comes with a proposed change for your team to review. After it ships, Bayes watches new conversations to confirm the problem stopped.
• Separates Genie problems from data problems. Monte Carlo already monitors the tables Genie queries, so you can tell a bad answer caused by Genie from one caused by bad data.

How it works

Genie stores conversations behind its Conversation API. Monte Carlo installs a collector job in your Databricks workspace and manages it for you; you never deploy, upgrade, or run it.

• About once an hour, the collector merges new conversations into a genie_traces table in a catalog and schema you choose.
• Monte Carlo reads that table through your SQL Warehouse connection, like any other trace table.
• One collector serves every Genie space registered to the same connection, catalog, and schema.
• Expect up to about an hour between a conversation in Genie and its trace in Monte Carlo.

Set up

Prerequisite: a Databricks SQL Warehouse connection in the same workspace as your Genie spaces. Each connection covers one workspace, so add one per workspace.

Step 1. Turn on Genie chat sharing

Do this first. It only applies to new conversations and can't be backfilled.

In Databricks, go to username menu (top right) → Previews → turn on Genie chat sharing.

Why: Genie keeps each answer private to the user who asked, even from space managers like the collector. With sharing on, new conversations are shared as Reviewable by space managers, so the collector can read their result sample and final answer. Without it, Monte Carlo sees only the question and SQL.

Users can still mark a conversation Private, and it stays question-only.

Step 2. Pick a catalog and schema for the trace table

Choose (or create) a catalog and schema in Unity Catalog where the collector will create genie_traces.

• Both must already exist before you import. Monte Carlo creates only the table, never the catalog or schema.
• A schema dedicated to Monte Carlo trace tables works best.

Step 3. Grant permissions

Grant these to the service principal your SQL Warehouse connection uses in Monte Carlo. That one principal usually handles all three jobs: installing the collector, collecting into the trace table (the collector runs as this principal), and reading the table. A dedicated, least-privilege service principal is recommended.

GRANT USE CATALOG ON CATALOG <catalog> TO `<service-principal>`;
GRANT USE SCHEMA ON SCHEMA <catalog>.<schema> TO `<service-principal>`;
GRANT CREATE TABLE ON SCHEMA <catalog>.<schema> TO `<service-principal>`;

Scope notes

• Folder access covers only /Shared/monte_carlo/. No workspace-wide grant is needed.

• Workspace access can't be scoped to a path; that's a Databricks limit. It's used only to create the collector job, one per catalog and schema. The service principal owns the job, so running and deleting it need nothing more.

• The collector runs on serverless compute, so no cluster entitlement is needed.

• Folder access and Workspace access are set in Databricks dialogs and entitlements, not with Unity Catalog GRANTs.

• The collector creates and owns genie_traces, so reading it needs no extra grant.
  Why CAN MANAGE on the space?

• Databricks only returns other users' conversations to principals with CAN MANAGE. There's no narrower read-all option.

• CAN MANAGE also lets the principal edit and delete the space.

• With only CAN VIEW, the collector runs without errors but sees only the service principal's own conversations, usually none.

Optional: pre-create genie_traces

Skip this unless governance requires the table to exist beforehand or be owned by another principal. Letting the collector create it is simpler and always gets the schema right.

If you do pre-create it, either transfer ownership to the service principal or grant SELECT and MODIFY on the table or its schema:

GRANT SELECT, MODIFY ON TABLE <catalog>.<schema>.genie_traces TO `<service-principal>`;

• Import doesn't check this grant. Without it, the first run fails with a permission error.
• The table must match the collector's schema. A column mismatch also fails the run.

Step 4. Add the Genie space in Monte Carlo

1. Go to Settings → Agent Observability and click Add.
2. Set Agent Type to Platform Agent.
3. Select a Databricks SQL Warehouse for Monte Carlo to read traces through.
4. Select an Agent: pick Databricks Genie, then your space. If it isn't listed, the service principal has no access; grant CAN MANAGE (step 3) and it appears.
5. Choose the trace table location: pick the catalog, then the schema, from step 2. Both lists are searchable.
6. Click Import. Import checks the space, catalog, and schema. If a grant is missing, it lists what to add; add it in Databricks and click Import again. Once Import succeeds, Monte Carlo installs the collector, and the space is collected on the collector's next run.

Step 5. Check the connection

Find your space in Settings → Agent Observability. Each space is listed by name. The Status column shows:

• Completed, with the date of the last run
• Awaiting this agent's first collection run, if the space was added since the collector last ran
• Installing, Pending, Permission needed, Install error, or Error, with a message saying what to fix

Once collected, turns are grouped into conversations automatically. Each turn shows the question, the generated SQL, a sample of the results, and the final answer, with failed turns flagged.

Step 6. Schedule a Reinforcement Loop digest

Open the agent, click Subscribe to digest on its issues list, and pick a cadence and an audience with an email or Slack recipient. Your team then gets each finding as it lands: what's breaking, why, and the proposed fix.

After about a day of traces, Bayes sets up monitoring on your agent across context, performance, behavior, and output. For a rule of your own, describe it to Bayes or build one yourself.

Troubleshooting

Status is Completed, but no conversations show up.
The service principal probably has CAN VIEW instead of CAN MANAGE on the space; the grant was likely lowered after setup. Re-grant CAN MANAGE. The next run picks up recent conversations within the collector's lookback window. Status stays healthy in this case, so check the trace table itself.

Turns show the question and SQL, but no answer.
Genie chat sharing is off, or the user marked the conversation Private. The collector gets PERMISSION_DENIED — "User … does not own conversation …". See step 1.

A run failed with a permission error.
If you pre-created genie_traces, check SELECT and MODIFY. This failure shows in the Status column. For any other revoked grant, add it back. The collector resumes on its next run; no need to re-import.

Removing a Genie space.
Deleting the agent removes the collector job and notebook once no other space uses that catalog and schema. Otherwise the collector keeps running for the others. Monte Carlo never drops genie_traces; delete it yourself if you don't need it.