Skip to main content

Running Evals and Datasets

A dataset is a reusable set of test cases for an agent. An eval is a run of one dataset's cases against the agent's published version, scored automatically.

Before you begin​

Running an eval requires the agent to be published. See Publishing an Agent. You can create datasets and cases before then.

Create a dataset​

  1. Open the Datasets tab.
  2. Select New dataset.
  3. Enter a Name.
  4. Optionally enter a Description.
  5. Select Create.

The new dataset appears in the list on the left and becomes selected.

Add a test case​

  1. With a dataset selected, select Add case.
  2. Under What is this case checking?, choose Final answer or Tool choice.
  3. Enter the prompt to send under Input the agent should receive.
  4. If you chose Tool choice, pick or type one or more tool names under Tool(s) the agent must call at least one of.
  5. If you chose Final answer, choose how to grade it under How should the answer be graded? — Contains this text, Equals this text exactly, Matches this regex, or Rubric — graded by an LLM judge — then enter the Expected value (or, for a rubric, the Grading instructions for the judge). Optionally check Case-sensitive.
  6. Select Add.

The case appears in the table with its input, type, and a plain-language description of the assertion.

A tool-choice case is checked by name only — it doesn't grade call order, so it won't punish a valid alternate approach. A rubric case is graded by a separate model call using the agent's own connection, so its result can vary slightly run to run.

Remove a dataset or case​

Select the delete icon next to a dataset or case. There's no confirmation dialog and no undo — deleting is immediate. There's also no rename or duplicate; to change a case, delete it and add a new one.

Run an eval​

  1. Open the Evals tab.
  2. Select New run.
  3. Choose a Dataset.
  4. Set Repetitions per case — how many times each case runs, useful since rubric grading can vary.
  5. Set Pass threshold (% of case results that must pass).
  6. Select Run.

A run always targets one dataset. To cover several datasets, start a run for each.

Read the results​

The results panel opens automatically when a run starts, and updates live while it's in progress. It shows the pass rate, and a table of every case with its pass/fail result, score, latency, and cost. Expand a row to see the agent's actual answer for that case.

Return to a previous run any time from the Evals tab's run table, filterable by dataset and by started date. A run's status is PENDING, RUNNING, COMPLETED, or FAILED.

How you know it worked​

The run's pass rate and per-case Pass/Fail results appear in the results panel once the run completes.

When it does not work​

SymptomCauseWhat to do
New run is disabledThe agent isn't published, or no datasets exist yetPublish the agent; create a dataset in the Datasets tab

Next steps​

  • Reviewing Logs — see the raw turn-by-turn record behind any conversation, not just eval runs.