A test case is one question you want your Agent to handle well, and optionally the answer you expect back. Collect enough of them and you can measure whether a change made your Agent better or worse.
Go to Agent → open an Agent → Evaluate → Test cases.
What's a dataset?
A dataset is a group of test cases. The Test cases page opens on your datasets, one row each. Click one to see its cases in a spreadsheet-style grid.
Tags label a case within its dataset, so the two do different jobs. A dataset is the set you run, and a tag narrows it down. An evaluation picks one or more datasets, then filters by tag if you want.
How do I add test cases?
Four ways:
| Way | How |
|---|---|
| From a real run | Open a run in Run history and add it to your test cases |
| By hand | New test case, then write the question and the answer you expect |
| Generate | ... → Generate test cases. Hercules writes them for you, and you can give it a topic |
| Import | Upload a spreadsheet (CSV) or JSON file, up to 200 rows at a time |
Adding from real runs is usually best. Those are the questions your Agent actually gets asked.
What's in a test case?
| Field | What it is |
|---|---|
| Input | What gets sent to the Agent |
| Expected output | The answer you want back, used by most scorers |
| Tags | Labels for grouping cases within the dataset |
| Metadata | Any extra details you want to keep with the case |
How do I import a spreadsheet?
Open the import dialog
Open a dataset, then ... → Import.Upload the file
A CSV with a header row, or a JSON file.Match up the columns
Say which column is the Input, the Expected output, Tags, or Metadata, and skip the ones you don't need. Hercules guesses from your column names first.
Do I need to fill in the expected output?
Only if your scorer needs one. Some scorers compare the Agent's reply against the answer you expected, and those skip any case that doesn't have one. Scorers that judge the reply on its own grade every case.
When you start an evaluation, Hercules tells you how many of your cases it can actually run.
What are tags for?
Narrowing a dataset down. Tag cases by topic (billing, refunds) or by risk (edge-case), then run an evaluation against only those.
Additional FAQ
Can I export my test cases?
Yes. Export everything, or only the rows you've ticked, as CSV or JSON. The file you get back can be imported again.
How many test cases should I have?
Enough to cover what you care about, including the things that have gone wrong before. A small set of real, hard questions beats a big set of made-up ones. One evaluation covers at most 100 cases, so split bigger sets across datasets.
Does generating test cases cost credits?
Yes, generating uses AI and costs credits. See Costs and limits.
Can I edit a case after importing it?
Yes. Edit a cell in the grid, or click a row to open it and change anything, including its tags.