# Webhook ingestion operations The server receives webhooks. It stores each verified delivery in the provider event inbox, a PostgreSQL table, before it answers. This page explains what to monitor, configure, and do after a failure. [Ingest durability](/next/contributor/ingest-durability.md#the-database-inbox) explains the design. GitHub counts a delivery as failed when the receiver answers with an error or takes longer than 10 seconds. It never retries a failed delivery on its own, but it keeps deliveries from the past three days and can [redeliver](https://docs.github.com/en/webhooks/testing-and-troubleshooting-webhooks/redelivering-webhooks) them. Hephaestus asks GitHub to redeliver failed deliveries of its GitHub App automatically. See [Failed GitHub deliveries](#failed-github-deliveries). Hephaestus does not redeliver GitLab, Slack and Outline deliveries that the receiver did not accept. The reconciliation sync recovers their effect over `SYNC_TIMEFRAME_DAYS` days. ## What the receiver stores The receiver verifies a delivery, then stores one event for each workspace whose subscription matches it, in one transaction. It reads the subscriptions in that transaction. A workspace that starts to monitor a repository therefore gets every event that the receiver stores after the change commits. A GitHub App installation event also gets one event with no workspace, because it can create the workspace. The receiver answers `202` only after the transaction commits. If PostgreSQL does not commit within 2 seconds, the receiver answers `503`. Each statement of the transaction stops after 1.5 seconds. The receiver thus answers inside Slack's 3-second window, also while the database pool is exhausted. A transaction that still commits after a `503` is harmless: the provider's retry or redelivery adds no event. * A delivery that arrives again with the same identifier adds no event. GitHub uses `X-GitHub-Delivery`, and Slack its `event_id`. GitLab uses a hash of the body, the event UUID and the event kind, for each connection. * Each `503` has a log record with `event.name=webhook.publish.failed` and `webhook.publish.reason=store-failed`. * What each provider does after a `503` is in [Which providers deliver again](#which-providers-deliver-again). Each server claims due events from the inbox. A claim leases one event for 2 minutes, and the handling server renews the lease every 30 seconds. A workspace handles the events of one resource, such as a repository or a Slack channel, one at a time and in arrival order. [Replay a failed webhook event](/next/admin/webhook-replay.md) has the retries, the dead letters and their replay. ## Failed GitHub deliveries When PostgreSQL is slow or down, the receiver cannot store a delivery in time, and GitHub records the delivery as failed. Every 10 minutes, the server reads the GitHub App's deliveries of the past two hours from the [GitHub App webhook API](https://docs.github.com/en/rest/apps/webhooks). For each event with no successful delivery, it asks GitHub to redeliver the latest attempt once the next redelivery is due. This is the procedure that GitHub [recommends](https://docs.github.com/en/webhooks/using-webhooks/automatically-redelivering-failed-deliveries-for-a-github-app-webhook). * Runs never overlap, also across server replicas: each run holds a lock for 9 minutes. A run sends requests for at most 5 minutes, and at most 5 a second. It stops after GitHub refuses 5 requests in a row. Due redeliveries that do not fit wait for the next run, the oldest events first. * If the newest 10 redeliveries all failed, the receiver is not taking deliveries. A run then sends only 3 redeliveries, to probe it, and counts `outcome=paused`. The other events keep their attempts until a probe succeeds. * If GitHub answers `403` or `429` to a request, the run stops, and the rest wait for the next run. * An event whose latest answer was a client error (`4xx`) is not redelivered. The receiver refused it on purpose, for example for a bad signature or an oversized payload. * An event gets at most six attempts: the original and five redeliveries. The redeliveries are due 0, 10, 20, 40 and 80 minutes after the original delivery. The gaps double, so the attempts outlast a database stall of up to 80 minutes. A stall that still lasts gets few requests. * A redelivery keeps its `X-GitHub-Delivery` identifier. The inbox stores a delivery once for each workspace and identifier. A redelivery of a stored delivery thus adds no event. The inbox keeps a handled event for 3 days, as long as GitHub keeps a delivery. `hephaestus.integration.consumer.replay-window` sets this time. * A slow database can still store a delivery after GitHub stopped waiting for it. Its redelivery then adds no event. * Silent mode does not stop redelivery. A redelivery sends nothing to people. It only asks GitHub to send the event to the receiver again. The job needs the GitHub App credentials and runs in the server role. An event that failed all six attempts is counted in `github.webhook.unrecovered`. For an outage longer than that, redeliver older failed deliveries in the GitHub App settings under **Advanced** → **Recent Deliveries**, within three days. After that, only the reconciliation sync recovers their effect. ## Which providers deliver again | Provider | Delivers a failed event again | What else recovers it | | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------ | | GitHub | Hephaestus asks GitHub to redeliver, on the schedule above. | Manual redelivery for three days, then reconciliation. | | GitLab | No. GitLab does not retry a failed webhook. It disables a hook after 4 failures in a row, for 1 minute up to 24 hours, and for good after 40. See [GitLab webhooks](https://docs.gitlab.com/user/project/integrations/webhooks/). | Reconciliation over `SYNC_TIMEFRAME_DAYS` days. | | Slack | Yes. Slack retries 3 times: at once, after 1 minute and after 5 minutes. With [Delayed Events](https://docs.slack.dev/apis/events-api/) turned on in the Slack app, it also retries every hour for 24 hours. The receiver answers Slack within 2 seconds and stores a retry once, by its `event_id`. | Reconciliation of monitored channels. | | Outline | No. Outline waits 5 seconds and does not retry. It disables a subscription when 80 % of its deliveries in 24 hours fail. | Reconciliation of the Outline connection. | ## What to alert on The server exports these Micrometer meters on its private management listener. [Scrape metrics](/next/admin/monitoring.md#scrape-metrics) has the port and the scrape configuration. [Alert rules](/next/admin/alert-rules.md) gives the shipped rules that read them. | Metric | Type | What it means | | ----------------------------------------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `webhook.publish` | counter | Deliveries that the receiver stored (`outcome=success`) or could not store (`outcome=failure`), by `stream`, which names the provider. A failure is a delivery that the provider records as failed. | | `integration.consumer.backlog` | gauge | Events in the inbox, by `provider` and `state`: `QUEUED`, `LEASED` or `DEAD`. | | `integration.consumer.oldest.due.seconds` | gauge (s) | Seconds since the oldest due event of a `provider` became due. 0 when none is due. | | `integration.consumer.backlog.sampler.failures` | counter | Passes that could not read the backlog. The gauges keep their last values. | | `integration.consumer.retry` | counter | Attempts that failed and wait for a retry, by `kind`. | | `integration.consumer.poison` | counter | Events that became dead letters, by `kind`. | | `integration.message.dropped` | counter | Attempts stopped by the write fence, by provider kind and reason: `erased-during-processing` or `lease-lost`. Erased events do not retry. | | `integration.consumer.inbox.retention.failures` | counter | Retention passes that failed. | | `github.webhook.redelivery` | counter | Redelivery requests that GitHub accepted (`requested`) or refused (`refused`). Runs that could not read the deliveries (`unavailable`), only probed a failing receiver (`paused`), or stopped listing at a bound (`truncated`). | | `github.webhook.unrecovered` | gauge | GitHub events of the past two hours that failed all six attempts. Above zero means events that Hephaestus did not receive. | 1. Alert when `integration.consumer.oldest.due.seconds` stays above a few minutes. No server claims, or a handler holds a resource's order. The events of a suspended workspace are not claimed but count here, so a long suspension also shows. 2. Alert when `webhook.publish` counts failures. The database refuses deliveries. 3. Read the `DEAD` backlog after a `HephaestusConsumerPoison` alert, and replay the dead letters after the fix. An `erased-during-processing` drop is the expected result of erasing an event while its handler runs. Do not replay it. The payload and its steps are gone. For repeated `lease-lost` drops without a restart, inspect lease renewal failures and database latency. ## How long the inbox keeps an event | Event | Kept for | Setting | | ---------------- | -------------------------------------------- | --------------------------------------------------------------------------------------------------- | | Handled | 3 days after its handling | `hephaestus.integration.consumer.replay-window` (`HEPHAESTUS_INTEGRATION_CONSUMER_REPLAY_WINDOW`) | | Dead letter | 30 days after its failure | `hephaestus.integration.consumer.dead-retention` (`HEPHAESTUS_INTEGRATION_CONSUMER_DEAD_RETENTION`) | | Queued or leased | Until it is handled or becomes a dead letter | — | One server deletes the ended events every hour. The handled events cover the three days in which GitHub can redeliver, so a late redelivery is not handled twice. Deleting a workspace deletes its events. Erasing a person deletes the events that name them. An event that a server handles at that moment stays, and its retention removes it. ## Payload size The receiver accepts webhook deliveries up to 25 MiB, matching GitHub's delivery cap (`hephaestus.webhook.http.max-payload-bytes`). The inbox stores the body as it arrived. The receiver holds each delivery in its heap until the inbox stores it, and a body costs several times its size. So it bounds what large deliveries hold at once, from its heap. A delivery of 256 KiB or less does not count, and it never waits. A larger one that does not fit waits for 1.5 seconds. Then the receiver answers `503` with `Retry-After`, and counts `webhook_rejected_total{reason="busy"}`. To hold more at once, raise `server.resources`. A delivery's body must arrive within 30 seconds, or the receiver answers `408` and counts `webhook_rejected_total{reason="too-slow"}`. No provider waits that long for an answer. ## Known limits * **A very large sync window does not finish.** The daily sync of a GitHub repository reads only the work that changed since its last complete run. It reads at most 10,000 pull requests and 25,000 issues. If more changed, for example after a long outage, the sync of that repository stops with an error. The next run reads the same window again, so it can stop again. The repository's resource sync table shows the error ([Read sync errors](/next/admin/integrations.md#read-sync-errors)). ## Where these settings live The server reads all of them ([Server runtime](/next/admin/runtime-roles.md)). `gate:env` checks the rendered configuration.