Agents is in private beta, so things may change. If you hit a problem, tell us via Support &
feedback.
How does a scorer grade?
An AI does the grading. You describe what a good answer looks like in plain English, and it reads each reply and grades it against what you wrote.What can I start from?
How do I write my own?
Describe what earns a pass, in plain English, and be specific.Answer accuracy criteria
Pick the model under What model should we use?. It offers the same tiers your Agent runs on: Lite, Balanced, Expert, and Expert - Fast. Balanced is the usual choice.
Can I try a scorer before saving it?
Yes. There’s a preview panel next to what you’re writing. Pick a dataset, choose some cases, and click Run on N cases. You get a pass rate and a grade for each one, so you can tighten your wording before saving.Can a scorer grade real runs?
Yes, as long as it doesn’t need an expected answer to compare against. Under Advanced, answer Run this scorer on production runs? and type what share of runs to grade, as any whole percentage from 1 to 100. Grades show up beside the run in Run history, and add up on Analytics. A scorer that compares against the expected answer can’t grade real runs, because a real run doesn’t have one.Additional FAQ
Do scorers have versions?
Do scorers have versions?
Yes. Each one keeps its own history, so you can see exactly how your wording changed.
How many scorers can I use in one evaluation?
How many scorers can I use in one evaluation?
Up to five, and you can’t add more once you hit it. Pick the ones that answer the question you’re
asking.
Do scorers cost credits?
Do scorers cost credits?
Yes. Grading uses AI, including every time it grades a real run. See Costs and
limits.
My scorer disagrees with me. What now?
My scorer disagrees with me. What now?
Read the reasoning it gives on each case. If the reasoning is fair but your wording was vague,
tighten the wording. It only knows what you told it.