> ## Documentation Index
> Fetch the complete documentation index at: https://hercules.app/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Agent evaluation runs

> Answer every test case with a version of your Hercules Agent, grade the replies with your scorers, and compare against last time to prove a change helped.

<Note>
  Agents is in private beta, so things may change. If you hit a problem, tell us via [Support &
  feedback](/docs/company/contact).
</Note>

An evaluation answers one question: did that change make my Agent better? It answers every question in your test cases, grades the replies with your scorers, and gives you a score you can compare.

<Warning>
  An evaluation doesn't run your Agent for real. It writes each reply from the version's
  instructions and model alone, in one pass, with no tools, no integrations, and no memory of
  earlier messages. That makes it a good way to measure your instructions, and the wrong way to
  check whether the Agent uses its tools correctly. For that, read a real run in [Run
  history](/docs/agents/run-history).
</Warning>

Go to **Agents** → open an Agent → **Evaluate** → **Evaluation runs** → **Start evaluation**.

<div className="screenshot fire">
  <img src="https://mintcdn.com/zeus-0f6dadbf/AUzCpLx0wjCzKFM0/images/agents/eval-runs-main.png?fit=max&auto=format&n=AUzCpLx0wjCzKFM0&q=85&s=890ba5c6441ddc442b66266b8197419f" alt="A completed evaluation run showing the per-scorer score and each case's ground truth beside the Agent's output" width="2123" height="1001" data-path="images/agents/eval-runs-main.png" />
</div>

### How do I run an evaluation?

<Steps>
  <Step title="Name it">
    Optional, but a name like `Tightened refusal tone` makes the list much easier to read later.
  </Step>

  <Step title="Pick the version to test">
    Whose instructions and model the replies are written from. The newest by default. See
    [Versions](/docs/agents/versions).
  </Step>

  <Step title="Pick your scorers">
    Choose up to five [scorers](/docs/agents/scorers) to grade the replies.
  </Step>

  <Step title="Choose the questions">
    Pick one or more [datasets](/docs/agents/test-cases), then narrow by tag if you want. Hercules tells
    you how many cases it can run with the scorers you picked.
  </Step>

  <Step title="Start it">Click **Run evaluation**.</Step>
</Steps>

### How do I prove a change helped?

Save a starting point, then compare against it.

<Steps>
  <Step title="Run an evaluation before you change anything">
    On the run's page, click **Set as baseline**. That's the score you'll compare everything
    against.
  </Step>

  <Step title="Make your change and publish it">See [Versions](/docs/agents/versions).</Step>

  <Step title="Run the same evaluation again">
    Open the new run and compare it against your baseline.
  </Step>
</Steps>

The comparison lines up each question against last time, so you can see exactly which answers got better and which got worse.

The report lists every question, the answer you expected, the reply the evaluation produced, and each scorer's grade. Filter it by All, Passed, Failed, or Errors.

<Tip>
  Because you pick which version answers, you can compare two versions without publishing either one
  in between.
</Tip>

### Why can't some of my cases run?

Because one of your scorers compares the reply against the answer you expected, and those cases don't have one. Cases without an expected answer get skipped. Add one on the [Test cases](/docs/agents/test-cases) page, or pick scorers that don't need it.

### What does an evaluation cost?

Two things cost credits: writing a reply to every question, and grading every reply. A big set of questions with several scorers isn't cheap, so filter down by tag while you're still tweaking things.

### Additional FAQ

<AccordionGroup>
  <Accordion title="Can I filter my evaluation runs?">
    Yes, by status: In progress, Complete, Failed, or Canceled.
  </Accordion>

  <Accordion title="What happens if I delete a run?">
    Its scores and results are gone. If it was your baseline, comparisons go back to having none. This
    can't be undone.
  </Accordion>

  <Accordion title="Is there a limit on questions per run?">
    Yes, 100. If you've picked more than that, Hercules tells you and you can't start. Narrow it down
    by tag, or pick fewer datasets.
  </Accordion>

  <Accordion title="Should I evaluate or use Auto-improve?">
    Both, for different jobs. An evaluation measures a change you already made.
    [Auto-improve](/docs/agents/auto-improve) reads your run history and suggests changes to make.
  </Accordion>
</AccordionGroup>