# Webhook ingestion operations Hephaestus receives webhooks in a separate container. It buffers all deliveries on NATS JetStream before anything else reads them. This page explains what to monitor, configure, and do after a failure. [ADR 0008](https://github.com/hephaestus-build/Hephaestus/blob/main/docs/decisions/0008-webhook-runtime-role.md) explains the design. GitHub counts a delivery as failed when the receiver answers with an error or takes longer than 10 seconds. It never retries a failed delivery on its own, but it keeps deliveries from the past three days and can [redeliver](https://docs.github.com/en/webhooks/testing-and-troubleshooting-webhooks/redelivering-webhooks) them. Hephaestus asks GitHub to redeliver failed deliveries of its GitHub App automatically. See [Failed GitHub deliveries](#failed-github-deliveries). Hephaestus does not redeliver GitLab, Slack and Outline deliveries that the receiver did not accept. The reconciliation sync recovers their effect over `MONITORING_TIMEFRAME` days. ## Durability and availability The supported Compose topology runs one NATS server and every webhook stream has one replica (R1). The bundled broker sets `sync_interval: always`, so it calls `fsync` for each message before acknowledging it. An acknowledged webhook or integration-sync event therefore survives host power loss or an unclean shutdown, provided the storage stack honors `fsync`. The [NATS durability documentation](https://docs.nats.io/nats-concepts/jetstream#syncing-data-to-disk) defines this guarantee. [Jepsen's NATS 2.12.1 analysis](https://jepsen.io/analyses/nats-2.12.1) shows the acknowledged-write loss possible with periodic syncing. Waiting for durable storage costs write throughput and latency. R1 remains unavailable while its only server is down, and cannot survive loss or corruption of its only JetStream volume. A clustered profile with three servers and R3 streams can remain available after one server fails, as described by [NATS replicated storage](https://docs.nats.io/nats-concepts/jetstream/replication). That profile may use a longer sync interval after an assessment of its failure model and latency budget. Keep `always` to protect acknowledgments from a correlated, unclean shutdown of the whole cluster. ## While the broker is slow The receiver answers `202` only after JetStream acknowledges the delivery. If the broker does not acknowledge in time, the receiver answers `503`. It never accepts a delivery that it has not stored. * The receiver waits up to `hephaestus.webhook.publish.deadline` (8 seconds by default) for the acknowledgment. The deadline ends before the 10-second response window of GitHub and GitLab. After that window, the provider has already failed the delivery, and another attempt only holds a request thread. Outline stops waiting after 5 seconds: a delivery stored later is kept, but Outline records a failure. Slack deliveries wait 2 seconds, inside Slack's 3-second window. * Within the deadline, the receiver publishes again in two cases. The client stopped waiting for an acknowledgment, which it does after 5 to 10 seconds. Or the broker refused at once, for example with no responders or a dropped connection. In that case, the receiver first waits a short backoff. Each attempt carries the same `Nats-Msg-Id`. If an earlier attempt was stored late, the broker answers the next one as a duplicate, and the receiver answers `202`. * A stream that refuses the message (for example, a full stream) is not retried. * What each provider does after a `503` is in [Which providers deliver again](#which-providers-deliver-again). * Each `503` has a log record with `event.name=webhook.publish.failed`. `webhook.publish.reason` names the cause and `webhook.publish.attempts` the attempts. The causes are `timeout`, `no_responders`, `connection`, `connection_closed`, `stream_full`, `rejected` and `interrupted`. * The receiver's health check reads only the state of the NATS connection. It makes no request to the broker, so it stays fast while the broker is slow. The stream gauges below report a broker that does not answer. Slow storage under the broker is the usual cause. The bundled broker calls `fsync` for each message, so every publish waits for the disk. Give the JetStream volume storage that is not shared with a busy database or container runtime. See [Troubleshooting](/admin/troubleshooting.md). ## Failed GitHub deliveries When the broker is slow or down, the receiver cannot store a delivery in time, and GitHub records the delivery as failed. Every 10 minutes, the server reads the GitHub App's deliveries of the past two hours from the [GitHub App webhook API](https://docs.github.com/en/rest/apps/webhooks). For each event with no successful delivery, it asks GitHub to redeliver the latest attempt once the next redelivery is due. This is the procedure that GitHub [recommends](https://docs.github.com/en/webhooks/using-webhooks/automatically-redelivering-failed-deliveries-for-a-github-app-webhook). * Runs never overlap, also across server replicas: each run holds a lock for 9 minutes. A run sends requests for at most 5 minutes, and at most 5 a second. It stops after GitHub refuses 5 requests in a row. Due redeliveries that do not fit wait for the next run, the oldest events first. * If the newest 10 redeliveries all failed, the receiver is not taking deliveries. A run then sends only 3 redeliveries, to probe it, and counts `outcome=paused`. The other events keep their attempts until a probe succeeds. * If GitHub answers `403` or `429` to a request, the run stops, and the rest wait for the next run. * An event whose latest answer was a client error (`4xx`) is not redelivered. The receiver refused it on purpose, for example for a bad signature or an oversized payload. * An event gets at most six attempts: the original and five redeliveries. The redeliveries are due 0, 10, 20, 40 and 80 minutes after the original delivery. The gaps double, so the attempts outlast a broker stall of up to 80 minutes. A stall that still lasts gets few requests. * A redelivery keeps its `X-GitHub-Delivery` identifier. The stream discards a copy of a delivery that it stored within `hephaestus.webhook.stream.duplicate-window`. The default is 3 hours, longer than the 2-hour look-back of the redelivery job. An existing stream gets this window at the receiver's first start, with no step. A manual redelivery within the window of a delivery that the stream stored is answered `202` and not processed again. * A slow broker can still store a delivery after GitHub stopped waiting for it. If its redelivery comes after the duplicate window, the event is processed twice. A late copy can restate an older reviewer list until the next sync reads the list again. [Reviewer lists are dated snapshots](/contributor/sync-lifecycle.md#reviewer-lists-are-dated-snapshots) explains this. * Silent mode does not stop redelivery. A redelivery sends nothing to people. It only asks GitHub to send the event to the receiver again. The job needs the GitHub App credentials and runs in the server role. An event that failed all six attempts is counted in `github.webhook.unrecovered`. For an outage longer than that, redeliver older failed deliveries in the GitHub App settings under **Advanced** → **Recent Deliveries**, within three days. After that, only the reconciliation sync recovers their effect. ## Which providers deliver again | Provider | Delivers a failed event again | What else recovers it | | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------ | | GitHub | Hephaestus asks GitHub to redeliver, on the schedule above. | Manual redelivery for three days, then reconciliation. | | GitLab | No. GitLab does not retry a failed webhook. It disables a hook after 4 failures in a row, for 1 minute up to 24 hours, and for good after 40. See [GitLab webhooks](https://docs.gitlab.com/user/project/integrations/webhooks/). | Reconciliation over `MONITORING_TIMEFRAME` days. | | Slack | Yes. Slack retries 3 times: at once, after 1 minute and after 5 minutes. With [Delayed Events](https://docs.slack.dev/apis/events-api/) turned on in the Slack app, it also retries every hour for 24 hours. The receiver answers Slack within 2 seconds and stores a retry once, by its `event_id`. | Reconciliation of monitored channels. | | Outline | No. Outline waits 5 seconds and does not retry. It disables a subscription when 80 % of its deliveries in 24 hours fail. | Reconciliation of the Outline connection. | ## Key-value buckets that the receiver reads The receiver has no database. The server writes two NATS key-value buckets from PostgreSQL, at start and every minute: | Bucket | Holds | Without it | | ----------------------- | --------------------------------------------------------- | --------------------------------------------------- | | `outline-routes` | The route id of each active Outline connection. | Outline deliveries get `503`. | | `slack-channel-consent` | The Slack team and channel ids of each monitored channel. | Slack deliveries get `503`, and Slack retries them. | The buckets hold no secret and no content. The server recreates a lost bucket within one minute of its start. Do not delete them while the server is down: Outline does not retry a refused delivery. When at least 80 % of at least 10 deliveries in 24 hours failed, Outline disables the subscription and emails the Outline admin. The server registers it again at the next full sync. Two server meters show the state of Outline subscriptions: * `outline.webhook.subscription.unrouted` counts the active Outline connections that wait for a subscription on a current route. The server retries every 5 minutes. * `outline.webhook.subscription.deletion.failed` counts upstream deletions that failed. Outline disables such a subscription after its deliveries are refused. ## What to alert on The receiver exports these Micrometer meters on its private management listener. See [Scrape metrics](/admin/observability.md#scrape-metrics) for the port and scrape configuration. [Alert rules](/admin/observability.md#alert-rules) gives the shipped rules that read these meters. The meters below are tagged `stream` with one of `github`, `gitlab`, `slack`, `outline`. | Metric | Type | What it means | | ------------------------------------------------ | ----------- | ----------------------------------------------------------------------------------------------------------------------- | | `webhook.stream.consumer.pending` | gauge | Messages not yet delivered, summed over your deployment's consumers. A message two consumers match counts twice. | | `webhook.stream.consumer.ack.pending` | gauge | Messages delivered and not yet acknowledged, including ones waiting for redelivery, summed the same way. | | `webhook.stream.consumers.without.pull.requests` | gauge | Your deployment's consumers with no pull request waiting when polled. A healthy consumer can read zero between fetches. | | `webhook.stream.poll.age` | gauge (s) | Seconds since the gauges on this page were last refreshed. `NaN` means never. | | `webhook.stream.oldest.message.age` | gauge (s) | **Your effective retention.** The age of the oldest message the stream still holds. | | `webhook.stream.bytes` | gauge | Bytes stored. | | `webhook.stream.bytes.limit` | gauge | The bound in force on the live stream. | | `webhook.stream.bytes.utilization` | gauge (0-1) | `bytes` over `bytes.limit`. | | `webhook.stream.messages` | gauge | Messages stored. | | `webhook.stream.consumers` | gauge | Durable consumers on the stream. | These meters describe the publishes to the stream, the deliveries that never reached it, and their recovery. Only `webhook.publish` is tagged by `stream`: | Metric | Type | Role | What it means | | ---------------------------- | ------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `webhook.publish` | counter | webhook | Publishes the broker acknowledged (`outcome=success`) or did not acknowledge in time (`outcome=failure`), by `stream`. A failure is a delivery the provider records as failed. | | `github.webhook.redelivery` | counter | server | Redelivery requests that GitHub accepted (`requested`) or refused (`refused`). Runs that could not read the deliveries (`unavailable`), only probed a failing receiver (`paused`), or stopped listing at a bound (`truncated`). | | `github.webhook.unrecovered` | gauge | server | GitHub events of the past two hours that failed all six attempts. Above zero means events that Hephaestus did not receive. | **No metric counts deliveries that the stream removed.** A delivery is lost when the stream removes it before its required consumer acknowledges it. The gauges describe what the stream holds now. Zero pending means nothing waits, not that nothing was lost. Stream positions cannot establish loss either. Each consumer reads only its own subjects on a shared stream. A caught-up consumer therefore trails the stream's oldest message whenever other subjects are busier. The broker can announce a delivered but unacknowledged message that it deleted as a consumer advisory. Hephaestus does not subscribe to that advisory. 1. Alert if `webhook.stream.consumer.pending` or `webhook.stream.consumer.ack.pending` stays above zero or grows for several monitor intervals. 2. If `webhook.stream.bytes.utilization` is also near 1, investigate the risk. 3. Alert on `webhook.stream.poll.age` at a few times `hephaestus.webhook.stream.monitor-interval` (default 60s). 4. Alert when `github.webhook.unrecovered` is above zero, or when `github.webhook.redelivery` keeps counting `unavailable` or `refused`. Redeliver those events by hand, or let reconciliation recover their effect. 5. Alert when `webhook.publish` failures continue. Redelivery covers a short stall of the broker, not one that lasts. At its bound, the stream removes its oldest message on the next publish, regardless of which consumer still needs it. `webhook.stream.oldest.message.age` shows how much history remains, not how long a pending message has left. `webhook.stream.consumers.without.pull.requests` above zero with a growing backlog indicates a consumer from which nothing fetches. Alone, that value is not an alert. A receiver that cannot reach the broker holds all gauges at their last values. Those stale values look like current values. Provider reconciliation restores current source state within each integration's own limits, not every event. See [choosing the broker recovery strategy](/admin/production-operations-runbook.md#choose-the-broker-recovery-strategy-before-starting-consumers). Alert when `webhook.stream.bytes.utilization` stays above 0.5. In steady state the age bound holds the stream, well below its byte bound. A stream at its byte bound deletes its oldest message on every publish. Lower `HEPHAESTUS_WEBHOOK_STREAM_MAX_AGE` or raise the byte bound. `webhook.stream.consumers` climbing and never falling means durables are accumulating on a shared broker. See [Consumers that outlive their deployment](#consumers-that-outlive-their-deployment). ## How long a delivery is actually kept Two environment files ship, with different numbers: * `docker/.env.example` (the reference deployment) keeps the application defaults below: `1GB` per stream, `10GB` for GitHub, and a broker cap of `NATS_JS_MAX_FILE_BYTES=17179869184` (16 GiB). * `docker/self-host/.env.example` (one small host) sets `512MB` per stream, `6GB` for GitHub, `NATS_JS_MAX_MEM=1G` and `NATS_JS_MAX_FILE_BYTES=10737418240` (10 GiB). At the 7-day age, its GitHub bound fits about 460 MB of deliveries a day. The four configured stream bounds must fit within the broker's file-storage cap. Do not size that cap above the volume's available space. Each stream carries two bounds: * `HEPHAESTUS_WEBHOOK_STREAM_MAX_AGE` (default `7d`) holds the stream in steady state. Consumers acknowledge a delivery within seconds. The age decides only how long a deployment that is offline can resume from the stream. * `HEPHAESTUS_WEBHOOK_STREAM_MAX_BYTES` (default `1GB`, and `HEPHAESTUS_WEBHOOK_STREAM_MAX_BYTES_GITHUB` at `10GB`) is the disk-safety bound behind it. At this bound, the broker deletes the oldest message on every publish. Size the age so that your daily GitHub volume times the age stays below half of the GitHub byte bound. The defaults keep a busy instance (750 MB a day) at about 5 GB. Lower the age as your volume grows. `webhook.stream.oldest.message.age` shows the retention that your deployment gets. ### Upgrading a stream that is already full An existing stream keeps its earlier limits until startup applies the new ones. Startup withholds a shorter age, because it deletes messages that the stream holds. The webhook receiver then logs `Stream github limit change withheld because it would delete stored messages` at every start. To apply the new age once: 1. Check that `webhook.stream.consumer.pending` and `webhook.stream.consumer.ack.pending` are near zero. The messages that the new age deletes were then delivered already. 2. Set `HEPHAESTUS_WEBHOOK_STREAM_ALLOW_DESTRUCTIVE_LIMIT_UPDATES=true` and restart the webhook receiver. On the Helm chart, set `webhook.stream.allowDestructiveLimitUpdates` instead. The broker deletes the old messages before it answers. How long that takes on a large stream on a slow disk is unmeasured. With several webhook receivers, the others keep publishing while one applies the limit. If the log reports a timeout, raise `HEPHAESTUS_WEBHOOK_STREAM_LIMIT_UPDATE_TIMEOUT` (chart: `webhook.stream.limitUpdateTimeout`) and restart again. 3. Look for `Reconciled JetStream stream limits and DELETED stored messages` in the log. 4. Set the flag back to `false`. Scheduled sync (`MONITORING_SYNC_CRON`, hourly by default in production) fetches the last `MONITORING_TIMEFRAME` days from the provider API again. Other means can recover a delivery removed inside that window. Nothing can recover one removed outside it. 1. The age bound removes messages in steady state. A deployment that was offline gets back what it missed inside `MONITORING_TIMEFRAME` days through reconciliation. 2. The byte bound only protects the disk. Keep it above twice your daily GitHub volume times the age, so the age binds first. By default, streams with personal message content, `slack` and `outline`, have a 72-hour age bound. Their `hephaestus.webhook.stream.max-age-by-stream` entries take precedence over the shared age. PostgreSQL is the system of record. An erasure request cannot reach inside a broker stream. These streams are transport buffers. The `small` chart profile selects its own [stream limits](/admin/kubernetes.md#storage-and-retention), including both per-stream age overrides. ## The storage budget, and why it is not just a number `NATS_JS_MAX_FILE_BYTES` is 16 GiB in `docker/.env.example` and without an override, and 10 GiB in `docker/self-host/.env.example`. It sets the broker's JetStream budget and the total against which the receiver checks per-stream bounds. **The receiver refuses to start if the four bounds sum above that budget:** ``` Webhook stream bounds total 21474836480 bytes, over the 17179869184-byte broker storage budget (hephaestus.webhook.stream.storage-budget). Lower hephaestus.webhook.stream.max-bytes[-by-stream], or raise the budget and the JetStream volume with it. ``` Keep the budget **below the free space on the broker's volume**. The two failure modes are not alike: | The broker fills… | What happens | | ------------------------ | ---------------------------------------------------------------------------- | | its own JetStream budget | refuses new messages, keeps running, recovers by itself when messages expire | | the **filesystem** | cannot write its own metadata, and stays wedged even after space is freed | The second one needs a human. That is what the arithmetic above exists to make unreachable — for webhook stream growth. `max_file` governs the whole JetStream store, so anything else sharing that volume can still fill it. ## Changing a bound on a stream that already exists Startup reconciles an existing stream's **limits** with your configuration. It never rewrites subjects, retention mode, storage type or discard policy. | Situation | Startup does | | --------------------------------------------------------------------- | --------------------------------------------------------------- | | The change cannot delete anything | applies it, logs at INFO | | The change would delete stored messages | **withholds it**, logs at ERROR with exactly what it would cost | | …and `HEPHAESTUS_WEBHOOK_STREAM_ALLOW_DESTRUCTIVE_LIMIT_UPDATES=true` | applies it, logs what was deleted | | The change would leave the stream with no bound at all | **withholds it**, logs at ERROR — no flag overrides this | | Stream state cannot be read | withholds everything. Nothing can be proven safe | | The live stream's shape has drifted | withholds everything, logs at ERROR | A withheld change looks like this: ``` Stream github limit change withheld because it would delete stored messages: [maxBytes -1 -> 10737418240 (32300000000 bytes stored, 21562581760 would be deleted)] — set hephaestus.webhook.stream.allow-destructive-limit-updates=true to apply it ``` CAUTION: Destructive limit changes A destructive change deletes stored messages. A flag left on gives all future deploys permission to delete messages. Enable it only for one deliberate change. Unset it after you confirm the deletion. To apply a withheld change deliberately: 1. Run: ``` nats stream report ``` 2. Decide whether the size is legitimate for your traffic. 3. If it is, raise `HEPHAESTUS_WEBHOOK_STREAM_MAX_BYTES_GITHUB` (or `..._MAX_BYTES`) instead. 4. Keep the four bounds under `NATS_JS_MAX_FILE_BYTES`. 5. If the size is not legitimate, set `HEPHAESTUS_WEBHOOK_STREAM_ALLOW_DESTRUCTIVE_LIMIT_UPDATES=true`. 6. If the size is not legitimate, restart the receiver **once**. 7. After that destructive restart, confirm the `DELETED stored messages` log line. 8. After that confirmation, unset the flag again. The broker deletes excess messages **before** it replies. On a large stream, the call therefore takes as long as the deletion. `HEPHAESTUS_WEBHOOK_STREAM_LIMIT_UPDATE_TIMEOUT` (default `5m`) specifies startup's wait time. 1. If deletion takes longer, raise the timeout. If the call still times out, startup reports the stream's actual limits. It does not assume the update failed. The broker may have applied a timed-out update: ``` Failed to reconcile JetStream stream limits: name=github changed=[maxBytes -1 -> 10737418240] — live configuration is now maxAge=PT4320H maxBytes=10737418240 maxMessages=-1 ``` That line says the change **did** occur. 1. Before retrying, confirm with: ``` nats stream info github ``` If instead you see a shape-drift error, the stream is not what this deployment expects and no bound is written onto it at all: ``` Stream github has drifted from the shape this deployment expects: [discardPolicy=New (expected Old)] — limits left unreconciled ``` 1. Before applying the bound, repair the shape with: ``` nats stream edit github ``` `discardPolicy=New` makes the broker reject incoming publishes at the bound instead of removing old messages. A full stream therefore causes a total ingestion outage. ## Recovering a wedged broker A wedged broker has these symptoms: * `webhook.publish` counts `outcome=failure`, and providers receive `503`. * `webhook.stream.poll.age` grows or shows `NaN`. * The NATS container logs write errors. * `docker compose --env-file .env --env-file release-lock.env ps` shows `nats-server` unhealthy but running. Docker does not act on a failed healthcheck. The container therefore remains in that state indefinitely. 1. **Confirm it is the disk.** On the broker host: ``` df -h # free space on the JetStream volume docker compose --env-file .env --env-file release-lock.env logs --tail=100 nats-server ``` `no space left on device` on a JetStream write identifies this failure. If space is free, the fault is different. Check the network path instead. Check with: ``` nats server check connection ``` 2. **Free space.** Remove other content from that volume. JetStream will *not* recover alone after it fails to write its metadata. The errors therefore continue. 3. **Restart the broker**, which is the step that clears the wedge: ``` docker compose --env-file .env --env-file release-lock.env restart nats-server ``` 4. **Restart the receiver** so it re-establishes its connection and re-reconciles the streams: ``` docker compose --env-file .env --env-file release-lock.env restart webhook-server ``` 5. Before another incident, set `NATS_JS_MAX_FILE_BYTES` below the volume's actual free space. 6. Bring the per-stream bounds under it. Startup deliberately refuses to run if they do not fit. 7. **Account for what was lost.** Deliveries that arrived while the broker could not write were dropped at the edge. The provider saw a 503 or a timeout. Hephaestus redelivers failed GitHub App deliveries of the past two hours. Redeliver older GitHub deliveries in the App settings, within three days. 8. Under **Administration → Integrations → GitHub/GitLab**, select **Sync now** on the workspace connection. It fetches what falls inside `MONITORING_TIMEFRAME` again. Anything older than that window is gone. ## Payload size The receiver accepts webhook deliveries up to 25 MiB, matching GitHub's delivery cap. `NATS_MAX_PAYLOAD_BYTES` (default `26214400`) sets NATS' `max_payload`. NATS' own default is only 1 MB. Without sufficient capacity, each delivery in the gap would pass verification and acceptance but fail at publish. 1. Configure the broker to carry at least 25 MiB. The receiver reads the broker's live limit at startup and logs at ERROR if the two disagree: ``` The receiver accepts webhooks up to 26214400 bytes but this NATS server caps a message at 1048576 ``` ## Consumers that outlive their deployment Only JetStream removes a durable consumer. A stack that is *deleted*, rather than shut down, never removes its consumers. On a shared broker, each former stack therefore leaves a generation of durables whose backlogs never drain. `HEPHAESTUS_INTEGRATION_CONSUMER_INACTIVE_THRESHOLD` (default `30d`) lets JetStream remove them. It measures time with **nothing connected**, not time without traffic. A running deployment holds continuous requests against its consumers. Those requests continuously reset the clock, even with silent queues. Only a deployment that no longer exists ages out. Values between `0s` and `1h` fail at startup. A threshold that short expires a consumer during an ordinary restart, which the setting must prevent. A removed consumer is recreated at new messages only. It skips messages that arrived while the deployment was gone. 1. On disposable stacks that share a broker, set a much shorter value (`72h`). This includes temporary test stacks. 2. If an offline deployment must resume after the threshold exactly where it stopped, set `0s` to disable removal. `0s` preserves the consumer's *position*, not its *messages*. Messages still expire under the stream's retention. A deployment offline past the stream's byte or age bound returns to a cursor that points at messages no longer present. Those messages are not delivered. ## Who reaches the broker The Compose files publish the broker's client port 4222 on the host's loopback address only. A process on that host, such as an operator's tool, connects to `127.0.0.1:4222`. A stack on the same host connects over the Docker network `shared-network` to `nats-server:4222`. The single-host stack publishes no broker port. Reach its broker from a container on its network, or with `docker exec`. No other host reaches the broker. ## Broker image security scope The bundled broker uses standalone JetStream with user/password authentication and plain NATS with the host port bound to loopback. Its image assessment for CVE-2026-111943 applies to these shipped launches. The certificate-verification code is present but its TLS, HTTPS resolver, OCSP and remote-cluster paths are not enabled. A custom broker configuration that enables certificate verification is outside that assessment. Use a fixed vendor image or reassess the advisory before enabling those features. This also applies to an external broker reached through a `tls://` application connection. ## Making a change actually reach the broker **Compose does not recreate NATS when only its config changes.** The broker configuration is supplied by a Compose `configs:` block. A content change does not change the container spec. CAUTION: Restart loses active deliveries Recreation preserves stored messages because JetStream data lives in the volume. GitHub deliveries in progress during the restart are redelivered within an hour. See [Failed GitHub deliveries](#failed-github-deliveries). For the other providers, reconciliation covers `MONITORING_TIMEFRAME` days afterward. 1. After a configuration change, recreate the broker from the directory where the stack is deployed. 2. For the self-host stack, run from `docker/self-host`: ``` docker compose --env-file .env --env-file release-lock.env up -d --force-recreate nats-server docker compose --env-file .env --env-file release-lock.env exec nats-server cat /etc/nats/nats-server.conf ``` 3. For the multi-file Compose stack: ``` dc() { docker compose --env-file .env --env-file release-lock.env -p hephaestus-core -f compose.core.yaml "$@"; } dc up -d --force-recreate nats-server dc exec nats-server cat /etc/nats/nats-server.conf # confirm the value you set ``` ## Where these settings live Stream bounds are read by beans that only exist on the **webhook** runtime role. Setting them on the `application-server` container configures nothing at all — see [Runtime Roles](/admin/runtime-roles.md). The reference Compose files put them on `webhook-server`, and a build-time check fails if they drift back.