Replay a failed webhook event

Each verified webhook delivery is a row in the provider event inbox in PostgreSQL. It waits there while a provider API or the handler is unavailable. A proven temporary failure retries with a backoff of 2 seconds up to 5 minutes, without a limit. An unknown exception retries up to hephaestus.integration.consumer.poison.max-attempts (default 10), then becomes a dead letter. A parsing failure becomes a dead letter on its first attempt.

Each workspace handles the events of one resource, such as a repository or a Slack channel, one at a time and in arrival order. A later event of that resource waits until the first one is done or is a dead letter. Events of other resources and other workspaces do not wait. GitHub App installation events have no workspace yet. They wait in one order for the whole instance.

A dead letter keeps its payload and headers in PostgreSQL. The integration.consumer.poison counter counts new dead letters. HephaestusConsumerRetryStalled alerts when retries continue for 15 minutes. HephaestusProviderEventsWaiting alerts when the oldest due event of a provider waits more than 5 minutes.

For a repeated non-parse failure, inspect error.root_type and the sanitized stack frames. These diagnostics appear on the first attempt and when the event becomes a dead letter. Fix the cause, then replay the dead letter.

A server that stops while it handles an event loses its lease. After 2 minutes, another server takes the event back and counts the attempt. An event that stops every server that handles it becomes a dead letter after its attempts.

Inspect the records

Use an authenticated API client with a workspace administrator's access token. Request GET /workspaces/{workspaceSlug}/integrations/dead-letters.

The response is a cursor page: content holds up to 100 records in ID order. Pass its nextCursor as cursor to read the next page. The last page has no nextCursor.

Each record contains its ID, provider, error class, arrival time, failure time, last replay time and requesting account ID. The API does not return payloads or routing credentials.

Installation lifecycle events can arrive before a workspace exists. Only an instance administrator can list these records, with GET /admin/integration/dead-letters. An installation replay requires a recent sign-in because it can change provider access. Its completion and refused sign-in checks appear in the authentication audit trail. This endpoint excludes every workspace-owned record. An instance administrator uses the workspace endpoint for workspace-owned records.

Replay a record

1. Fix the parser or restore the required connection.
2. Request POST /workspaces/{workspaceSlug}/integrations/dead-letters/{id}/replays. For an unassigned installation event, use POST /admin/integration/dead-letters/{id}/replays instead.
3. Check replayedAt in the response. The event is queued again as its first attempt, at the head of its resource's order.

Check that it worked

Request the list again. A replayed event that succeeds is no longer in the list. An event that fails again comes back with its replayedAt.

A record that is not a dead letter of that workspace returns HTTP 404. This includes a record that you already replayed, and a record of another workspace. If an installation replay cannot be written to the audit trail, the request returns HTTP 503 and queues nothing.

A replay is an explicit late event. It does not rewind later events. The handler skips the steps that an earlier attempt committed. Handlers must tolerate duplicate and stale events. A replay does not supply a new provider revision.

Dead letters stay for hephaestus.integration.consumer.dead-retention (default 30d) after they fail. Handled events stay for hephaestus.integration.consumer.replay-window (default 3d), so a provider's redelivery in that time is not handled twice. Deleting a workspace deletes its events. Include PostgreSQL in the backup procedure and restrict database access to operators who can read provider data.