Hercules posts each run's durable events into your app's Convex deployment, so your database holds every message and the run's result, even when no one has the page open. @usehercules/agents receives and applies them for you once you call support.registerRoutes(http) (see Convex backend). Read on to see what it does, or to receive the events yourself.
Where are events delivered?
POST https://<deployment>.convex.site/hercules/agents/events- The deployment is the one whose app key started the run. An app key that is not tied to one deployment delivers to your app's production deployment. A run started with an Organization API key is not delivered anywhere
- If that deployment is no longer your app's, or such a key's app has no production deployment, the run's events are not delivered. They are never sent to another deployment
- The path is part of the API, so it stays the same within a version
Authorization: Bearer <token>carries a token signed by your app's Hercules Auth. See How do I check the token?Hercules-Versionis the version the run started with- The body is
{ "events": [...] }: durable events, plusmessage.appendedwhile it is still available - Delivered events always carry the full tool details: the names of
herculestools, and theresultof your app tools.expose_tool_detailschanges only what a run token reads
What does my app need?
- Hercules Auth. Hercules signs each delivery with your app's own Hercules Auth, so the app needs Hercules Auth 2.0, where each app has its own sign-in setup. Without it nothing can be signed, so Hercules does not start a run with the app's key: the request is a 409
hercules_auth_required. See Start a run - The events route. In a Convex app,
support.registerRoutes(http)inconvex/http.ts
Without the route, replies still reach your app's database, but only through the package's fallback check, minutes late. The next message in a thread fails with thread_busy until then.
In what order do events arrive?
- A thread's events arrive in order, with one call in flight per thread at a time
- One call carries at most 100 events or 1 MiB
- A
page_actionand a run's final event are sent at once. Other events wait up to 5 seconds, to go together - Delivery is at least once. After a timeout or a crash, the same events can arrive again, so apply each
(run_id, seq)once - At most 8 calls are in flight to one deployment at a time. When more threads are waiting, the ones with the fewest failures go first, then the ones that have waited longest
How should my endpoint answer?
| Answer | What Hercules does |
|---|---|
2xx | Counts the events as applied and moves on |
429 or 503 with Retry-After | Waits as told, at least 1 second and at most 10 minutes |
408, any other 5xx, no answer in 10 seconds, a network error | Retries after about 1 second, doubling up to 10 minutes, with jitter. The thread stalls 24 hours after its first failure |
413 | Halves the batch and sends it again, and keeps the smaller size until the thread is caught up. A single event that is still too large is held like a 400 |
400, 422, any other 4xx, any 3xx | Holds that thread's delivery until your app's next deploy to that deployment |
401, 403, or 404 | Stops every delivery to that deployment, and asks again with one call at a time |
- Answer
2xxonly after the events are written - Answer
401or403only for a token you checked and found wrong. If you cannot check it just now, for example because your issuer's keys cannot be fetched, answer503withRetry-After - A
429or503without aRetry-AfterHercules can read is retried like any other5xx - A thread that gets nothing but
413backs off like any other failing thread, and stalls after 24 hours
What happens while my endpoint keeps failing?
A call that gets no answer, a 408, a 429, or a 5xx is retried, thread by thread:
- The 24 hours count from the thread's first failure, whatever kinds of failure follow it. They start over when the thread's events are delivered, or when your app deploys
- A stalled thread goes again when any delivery to the same deployment succeeds, or on your app's next deploy
- When five different threads of one deployment fail this way in a row, Hercules pauses every delivery to that deployment for 16 seconds, then sends one call. Each call that fails doubles the pause, up to 10 minutes, or makes it the
Retry-Afterwhen that is longer. The first call that gets a2xxends the pause
What stops every delivery to a deployment?
A 401 or a 403, a 404, or an app that has no Hercules Auth anymore. Each is your app's answer for every thread, not for one, so all of the deployment's deliveries stop.
- One call at a time asks again: after 30 seconds, then 2 minutes, then 10 minutes, then every hour. Each carries a newly made token
- A
2xxopens the deployment again. So does an answer that refuses only that call's events, such as a400. The threads that waited then go out, at most 8 calls at once - A call that gets no answer or a
5xxleaves the deployment stopped, now for that reason. Whenever the reason changes, the waits start again from the first step for the new one: 16 seconds here, or 30 seconds for a401,403, or404 - Nothing here needs a deploy to recover. Add the route or fix the sign-in setup, and the next call finds out
What does a deploy change?
- Publishing your app counts for its production deployment. A build that Hercules pushes successfully from the App Builder counts for the deployment it pushed to
- After either one, that deployment's failed threads start over: the held ones, the stalled ones, and the ones waiting between retries. If the deployment was stopped, its next call is due at once
- Code that reaches a deployment any other way does not count, such as a file saved while Convex watches your code
When does Hercules stop trying?
A thread whose oldest undelivered event has waited 7 days, and that has no run going, is dropped from delivery. The events are not lost: they stay with their runs, where your app reads them with GET /v1/agents/runs/{run_id}/events.
How do I know a delivery is failing?
Not from the console yet. Failing deliveries are visible to Hercules staff only, so nothing tells you about one. Your app can always pull instead: read a run with GET /v1/agents/runs/{run_id} and its events with GET /v1/agents/runs/{run_id}/events. @usehercules/agents does this for you for any run that hears nothing for 3 minutes.
How do I check the token?
Every call Hercules makes to your app (event deliveries and app tool calls) carries an RS256 JWT signed by your app's Hercules Auth. For an event delivery, check that:
- The signature matches a key in your issuer's OpenID discovery document
issis your app's Hercules Auth URL, theHERCULES_OIDC_AUTHORITYenvironment variableaudis exactly your events URL, on your deployment's ownhttps://<deployment>.convex.siteaddress. Hercules signs for that address even when your deployment also has a custom domainexphas not passedhercules_principalishercules
Do not refuse a token because you have seen its jti before. Hercules reuses one delivery token for up to 5 minutes, so several calls carry the same one.
| Claim | Value |
|---|---|
iss | Your app's Hercules Auth URL |
aud | The exact URL being called: your MCP server's, or your events endpoint's |
sub | The end user's subject on a user token. Otherwise agent:<agent_id> |
hercules_principal | user (an Agent calling a tool for a signed-in user), agent (an Agent calling a tool for no end user), or hercules (Hercules delivering events) |
hercules_actor | agent:<agent_id>, on user tokens |
hercules_inspect | true on the tokens Hercules uses to list your tools without calling them |
iat, exp, jti | When it was issued, when it expires (10 minutes later), and the token's id. A delivery token is used for more than one call |
A token is good only at the URL in its aud, so a delivery token cannot call your app tools, and neither works on your app's other functions. Claims are only ever added, so ignore the ones you don't know.
@usehercules/agents exports the check it uses:
import { bearerTokenOf, createTokenVerifier, deploymentOrigins } from "@usehercules/agents";
// Build it once: it caches your issuer's keys for five minutes.
const verifier = createTokenVerifier({
issuer: process.env.HERCULES_OIDC_AUTHORITY!,
// The deployment's own .convex.site address, which Hercules signs for.
audiences: deploymentOrigins(process.env.CONVEX_SITE_URL, process.env.CONVEX_CLOUD_URL).map(
(origin) => `${origin}/hercules/agents/events`,
),
});
/** Returns null for a delivery from Hercules, and otherwise the answer to send back. */
export async function refuseUnlessHercules(request: Request): Promise<Response | null> {
const result = await verifier.verify(bearerTokenOf(request));
if (!result.ok) {
// The keys could not be fetched. That says nothing about the token, so ask for a retry.
if (result.reason === "keys_unavailable") {
return new Response(null, { status: 503, headers: { "Retry-After": "5" } });
}
return new Response(null, { status: 401 });
}
if (result.token.claims.hercules_principal !== "hercules") {
return new Response(null, { status: 403 });
}
return null;
}deploymentOrigins takes the address from CONVEX_SITE_URL, and from CONVEX_CLOUD_URL when CONVEX_SITE_URL is a custom domain. A deployment that overrides both with custom domains has no .convex.site address to accept, so every delivery is refused with 401. Leave CONVEX_CLOUD_URL as Convex sets it.
What does the package do with a delivery?
The Convex component in @usehercules/agents:
- Checks the token. It answers
401for one that is not valid here, and403for one that is not a delivery token, and only for a token it could check - Answers
503withRetry-Afterwhile it cannot fetch your issuer's keys, so a problem reaching Hercules Auth never reads as your app refusing deliveries - Applies each batch in one mutation, and skips any
(run_id, seq)it has already applied - Accepts event types it does not know, and skips entries it cannot read, so an event Hercules adds later never stops a thread's delivery
- Writes the thread's messages, runs, and tool calls to its tables in your database
- Merges a message's
message.appendedtext when itsmessage.completedarrives - Splits messages larger than about 512 KB across several documents
- Answers
413for a batch of several events it could not apply, so Hercules sends half. For one event it could not apply, it answers500, which Hercules retries for 24 hours before that thread stalls. It answers400only for a body that is not{ "events": [...] }at all, so no thread is held over one event - Calls your
onRunFinishedfunction after it writes the run's final event
If a run that is still going gets no delivery for 3 minutes, the component reads the run and its events from the API itself, and checks again, less often the longer the run goes on. See What does the component store?
Additional FAQ
Do I have to receive events?
No, but your app then knows about a run only from the API. Without delivery, read the run with
GET /v1/agents/runs/{run_id} and its events or messages with the routes in the API
reference.
What if my app is down for a while?
Hercules retries each thread for 24 hours from its first failure, then the thread stalls. A stalled thread goes again when any delivery to the same deployment succeeds, or when your app deploys. After 7 days, a thread with no run going is dropped from delivery. Nothing is lost: the events stay with their runs, and your app can read them from the API.
Do I have to deploy to get deliveries going again?
Only for a thread held after a 400, a 422, or another refusal of its events: that one waits
for your app's next deploy to that deployment. A deployment stopped by a 401, 403, 404, or
missing Hercules Auth is asked again on a timer, so it recovers by itself once it answers. A
deploy only makes that next call come at once.
Does delivery work with a custom domain?
Yes, as long as CONVEX_CLOUD_URL is still the one Convex set. Hercules always calls
https://<deployment>.convex.site, and signs its tokens for that address. A deployment that
overrides both CONVEX_SITE_URL and CONVEX_CLOUD_URL with custom domains refuses every
delivery.