Hercules

DevelopersEvent delivery

Developers

Agent event delivery

Hercules posts each run's durable events into your app's Convex deployment, so your database holds every message and the run's result, even when no one has the page open. @usehercules/agents receives and applies them for you once you call support.registerRoutes(http) (see Convex backend). Read on to see what it does, or to receive the events yourself.

Where are events delivered?

Endpoint
POST https://<deployment>.convex.site/hercules/agents/events
  • The deployment is the one whose app key started the run. An app key that is not tied to one deployment delivers to your app's production deployment. A run started with an Organization API key is not delivered anywhere
  • If that deployment is no longer your app's, or such a key's app has no production deployment, the run's events are not delivered. They are never sent to another deployment
  • The path is part of the API, so it stays the same within a version
  • Authorization: Bearer <token> carries a token signed by your app's Hercules Auth. See How do I check the token?
  • Hercules-Version is the version the run started with
  • The body is { "events": [...] }: durable events, plus message.appended while it is still available
  • Delivered events always carry the full tool details: the names of hercules tools, and the result of your app tools. expose_tool_details changes only what a run token reads

What does my app need?

  • Hercules Auth. Hercules signs each delivery with your app's own Hercules Auth, so the app needs Hercules Auth 2.0, where each app has its own sign-in setup. Without it nothing can be signed, so Hercules does not start a run with the app's key: the request is a 409 hercules_auth_required. See Start a run
  • The events route. In a Convex app, support.registerRoutes(http) in convex/http.ts

Without the route, replies still reach your app's database, but only through the package's fallback check, minutes late. The next message in a thread fails with thread_busy until then.

In what order do events arrive?

  • A thread's events arrive in order, with one call in flight per thread at a time
  • One call carries at most 100 events or 1 MiB
  • A page_action and a run's final event are sent at once. Other events wait up to 5 seconds, to go together
  • Delivery is at least once. After a timeout or a crash, the same events can arrive again, so apply each (run_id, seq) once
  • At most 8 calls are in flight to one deployment at a time. When more threads are waiting, the ones with the fewest failures go first, then the ones that have waited longest

How should my endpoint answer?

AnswerWhat Hercules does
2xxCounts the events as applied and moves on
429 or 503 with Retry-AfterWaits as told, at least 1 second and at most 10 minutes
408, any other 5xx, no answer in 10 seconds, a network errorRetries after about 1 second, doubling up to 10 minutes, with jitter. The thread stalls 24 hours after its first failure
413Halves the batch and sends it again, and keeps the smaller size until the thread is caught up. A single event that is still too large is held like a 400
400, 422, any other 4xx, any 3xxHolds that thread's delivery until your app's next deploy to that deployment
401, 403, or 404Stops every delivery to that deployment, and asks again with one call at a time
  • Answer 2xx only after the events are written
  • Answer 401 or 403 only for a token you checked and found wrong. If you cannot check it just now, for example because your issuer's keys cannot be fetched, answer 503 with Retry-After
  • A 429 or 503 without a Retry-After Hercules can read is retried like any other 5xx
  • A thread that gets nothing but 413 backs off like any other failing thread, and stalls after 24 hours

What happens while my endpoint keeps failing?

A call that gets no answer, a 408, a 429, or a 5xx is retried, thread by thread:

  • The 24 hours count from the thread's first failure, whatever kinds of failure follow it. They start over when the thread's events are delivered, or when your app deploys
  • A stalled thread goes again when any delivery to the same deployment succeeds, or on your app's next deploy
  • When five different threads of one deployment fail this way in a row, Hercules pauses every delivery to that deployment for 16 seconds, then sends one call. Each call that fails doubles the pause, up to 10 minutes, or makes it the Retry-After when that is longer. The first call that gets a 2xx ends the pause

What stops every delivery to a deployment?

A 401 or a 403, a 404, or an app that has no Hercules Auth anymore. Each is your app's answer for every thread, not for one, so all of the deployment's deliveries stop.

  • One call at a time asks again: after 30 seconds, then 2 minutes, then 10 minutes, then every hour. Each carries a newly made token
  • A 2xx opens the deployment again. So does an answer that refuses only that call's events, such as a 400. The threads that waited then go out, at most 8 calls at once
  • A call that gets no answer or a 5xx leaves the deployment stopped, now for that reason. Whenever the reason changes, the waits start again from the first step for the new one: 16 seconds here, or 30 seconds for a 401, 403, or 404
  • Nothing here needs a deploy to recover. Add the route or fix the sign-in setup, and the next call finds out

What does a deploy change?

  • Publishing your app counts for its production deployment. A build that Hercules pushes successfully from the App Builder counts for the deployment it pushed to
  • After either one, that deployment's failed threads start over: the held ones, the stalled ones, and the ones waiting between retries. If the deployment was stopped, its next call is due at once
  • Code that reaches a deployment any other way does not count, such as a file saved while Convex watches your code

When does Hercules stop trying?

A thread whose oldest undelivered event has waited 7 days, and that has no run going, is dropped from delivery. The events are not lost: they stay with their runs, where your app reads them with GET /v1/agents/runs/{run_id}/events.

How do I know a delivery is failing?

Not from the console yet. Failing deliveries are visible to Hercules staff only, so nothing tells you about one. Your app can always pull instead: read a run with GET /v1/agents/runs/{run_id} and its events with GET /v1/agents/runs/{run_id}/events. @usehercules/agents does this for you for any run that hears nothing for 3 minutes.

How do I check the token?

Every call Hercules makes to your app (event deliveries and app tool calls) carries an RS256 JWT signed by your app's Hercules Auth. For an event delivery, check that:

  • The signature matches a key in your issuer's OpenID discovery document
  • iss is your app's Hercules Auth URL, the HERCULES_OIDC_AUTHORITY environment variable
  • aud is exactly your events URL, on your deployment's own https://<deployment>.convex.site address. Hercules signs for that address even when your deployment also has a custom domain
  • exp has not passed
  • hercules_principal is hercules

Do not refuse a token because you have seen its jti before. Hercules reuses one delivery token for up to 5 minutes, so several calls carry the same one.

ClaimValue
issYour app's Hercules Auth URL
audThe exact URL being called: your MCP server's, or your events endpoint's
subThe end user's subject on a user token. Otherwise agent:<agent_id>
hercules_principaluser (an Agent calling a tool for a signed-in user), agent (an Agent calling a tool for no end user), or hercules (Hercules delivering events)
hercules_actoragent:<agent_id>, on user tokens
hercules_inspecttrue on the tokens Hercules uses to list your tools without calling them
iat, exp, jtiWhen it was issued, when it expires (10 minutes later), and the token's id. A delivery token is used for more than one call

A token is good only at the URL in its aud, so a delivery token cannot call your app tools, and neither works on your app's other functions. Claims are only ever added, so ignore the ones you don't know.

@usehercules/agents exports the check it uses:

Check a delivery token
import { bearerTokenOf, createTokenVerifier, deploymentOrigins } from "@usehercules/agents";

// Build it once: it caches your issuer's keys for five minutes.
const verifier = createTokenVerifier({
  issuer: process.env.HERCULES_OIDC_AUTHORITY!,
  // The deployment's own .convex.site address, which Hercules signs for.
  audiences: deploymentOrigins(process.env.CONVEX_SITE_URL, process.env.CONVEX_CLOUD_URL).map(
    (origin) => `${origin}/hercules/agents/events`,
  ),
});

/** Returns null for a delivery from Hercules, and otherwise the answer to send back. */
export async function refuseUnlessHercules(request: Request): Promise<Response | null> {
  const result = await verifier.verify(bearerTokenOf(request));
  if (!result.ok) {
    // The keys could not be fetched. That says nothing about the token, so ask for a retry.
    if (result.reason === "keys_unavailable") {
      return new Response(null, { status: 503, headers: { "Retry-After": "5" } });
    }
    return new Response(null, { status: 401 });
  }
  if (result.token.claims.hercules_principal !== "hercules") {
    return new Response(null, { status: 403 });
  }
  return null;
}

deploymentOrigins takes the address from CONVEX_SITE_URL, and from CONVEX_CLOUD_URL when CONVEX_SITE_URL is a custom domain. A deployment that overrides both with custom domains has no .convex.site address to accept, so every delivery is refused with 401. Leave CONVEX_CLOUD_URL as Convex sets it.

What does the package do with a delivery?

The Convex component in @usehercules/agents:

  • Checks the token. It answers 401 for one that is not valid here, and 403 for one that is not a delivery token, and only for a token it could check
  • Answers 503 with Retry-After while it cannot fetch your issuer's keys, so a problem reaching Hercules Auth never reads as your app refusing deliveries
  • Applies each batch in one mutation, and skips any (run_id, seq) it has already applied
  • Accepts event types it does not know, and skips entries it cannot read, so an event Hercules adds later never stops a thread's delivery
  • Writes the thread's messages, runs, and tool calls to its tables in your database
  • Merges a message's message.appended text when its message.completed arrives
  • Splits messages larger than about 512 KB across several documents
  • Answers 413 for a batch of several events it could not apply, so Hercules sends half. For one event it could not apply, it answers 500, which Hercules retries for 24 hours before that thread stalls. It answers 400 only for a body that is not { "events": [...] } at all, so no thread is held over one event
  • Calls your onRunFinished function after it writes the run's final event

If a run that is still going gets no delivery for 3 minutes, the component reads the run and its events from the API itself, and checks again, less often the longer the run goes on. See What does the component store?

Additional FAQ

Do I have to receive events?

No, but your app then knows about a run only from the API. Without delivery, read the run with GET /v1/agents/runs/{run_id} and its events or messages with the routes in the API reference.

What if my app is down for a while?

Hercules retries each thread for 24 hours from its first failure, then the thread stalls. A stalled thread goes again when any delivery to the same deployment succeeds, or when your app deploys. After 7 days, a thread with no run going is dropped from delivery. Nothing is lost: the events stay with their runs, and your app can read them from the API.

Do I have to deploy to get deliveries going again?

Only for a thread held after a 400, a 422, or another refusal of its events: that one waits for your app's next deploy to that deployment. A deployment stopped by a 401, 403, 404, or missing Hercules Auth is asked again on a timer, so it recovers by itself once it answers. A deploy only makes that next call come at once.

Does delivery work with a custom domain?

Yes, as long as CONVEX_CLOUD_URL is still the one Convex set. Hercules always calls https://<deployment>.convex.site, and signs its tokens for that address. A deployment that overrides both CONVEX_SITE_URL and CONVEX_CLOUD_URL with custom domains refuses every delivery.