Eval datasets

Bayes, Monte Carlo's AI teammate, turns your agent's production conversations into eval datasets. You can test every new version against the questions your users actually ask, without writing the questions yourself.

Bayes curates two sets for each agent from its scored conversations:

Every example is a real conversation: a question a user asked and the answer your agent gave.

Monte Carlo's evaluation monitors score your agent in production. Eval datasets carry those results back into development. Export a set to the tool where you test new versions (Snowflake, Databricks, Google ADK, or your own test suite) and check each version against it before it ships. This is often called offline evaluation.

Before you start

• The agent needs at least one conversation-level evaluation monitor that returns a number score (for example, 1–5).
• There must be conversations that have met the scoring criteria defined in Dataset settings.

How Bayes picks examples

• Which conversations count: single-turn conversations from the last 30 days that have at least one score. If a conversation is continued and becomes multi-turn, it leaves the set.
• Golden: every score is at or above the floor (default 4 out of 5).
• Improvement: any score is at or below the ceiling (default 3 out of 5). A conversation with one high score and one low score goes to Improvement.
• Variety: Bayes takes examples one intent cluster at a time, so the set covers every kind of question users ask, not just the most common. Improvement also rotates through issue clusters, so it covers every way the agent fails, like slow runs, tool loops and low scores.
• Always current: sets use each conversation's latest scores.
• To change which evaluations count, the floor and ceiling, or the size of each set, open Datasets → Settings.

Hand-pick examples

Found a great answer while debugging, or a failure you want every new version to beat? Pin it.

1. Open any single-turn conversation.
2. Select Pin and choose Golden or Improvement.

Pinned conversations:

• Are always included in the set you chose, whatever their scores.
• Are tagged Pinned, and you can filter to them on the review page.
• Count toward the set's size limit.
• Stay pinned until you unpin them or they're older than 30 days.

Review and export

1. Open the agent and go to Eval datasets.
2. Select Review & export on the Golden or Improvement set.
3. Open any example to see it exactly as it will be exported, along with its scores and the judge's reasoning.
4. Uncheck any examples you want to leave out.
5. Choose your format and select Export.

Export formats

Get the most out of your dataset

Your export starts from real production answers, so most of the work is already done. A quick pass in your testing tool turns it into a strong regression suite:

• Confirm your golden answers. They're your agent's best-scoring answers.
• Define the answer you want for Improvement examples. Each one includes the original low-scoring answer, so you can see exactly what a new version needs to beat.
• Pin dates for time-based questions. Turn "yesterday" or "last quarter" into a specific date so the question stays meaningful over time. See Snowflake's best practices.
• Expect fresh data. Tool results are recorded from the day the question was asked, and your eval runs against today's data, so a different answer can still be correct.
• Genie: when the agent ran several SQL statements, Genie marks the question for your review, so you can pick the SQL that best answers it. See curating an effective Genie space.
• Share with care. Examples keep the details from the original conversation, like user IDs or case numbers. Remove them if necessary before sharing the dataset more widely.

FAQ

Why is a set empty or small?
Only single-turn conversations with number scores qualify. Add or enable a conversation-level evaluation monitor.

Why did an example disappear?
A newer score moved it out, the conversation was continued into multi-turn, or it's older than 30 days.