# Pull-Based Deployment A host polls a **channel** naming a release or commit, verifies its signature, and applies the verified Helm chart or digest-pinned Compose stacks. Deployment needs no inbound administration connection to the host. ## What the host trusts For a chart release deployment, the host verifies three signatures: | Artifact | Signed by | Answers | | ----------------------------------------------- | ---------------------- | --------------------------------------------------------------- | | The channel (`channels/.json`) | the promotion workflow | may this host move, and who said so | | The joint lock (`release-.chart.json`) | the release workflow | which source, chart digest, and image digests form this release | | The chart at its OCI digest | the release workflow | whether the chart package came from that workflow | Compose deployments verify the channel and the schema-1 release lock (`release-.json`). The host rejects a new channel commit that does not descend from its last accepted channel commit. It also rejects lower release versions and older commits unless `allowRollback` is set. These checks do not order manual transitions between releases and commits. Choose those targets deliberately. Each environment acts only on its own signed request: its channel file and signature bundle together. A promotion of another environment also adds a commit to `deploy-state`. If the host already applied its own unchanged request, it verifies the signature again. It records the new commit as accepted. It restarts nothing. If its own request is not applied yet, for example after a failed apply, the host tries it again. Each promotion signs again, so a promotion of the target that already runs applies again. A frozen promotion of the target that already runs is the exception: it restarts nothing, unless it also asks for **refresh-database-image**. A frozen promotion that does not ask for it also replaces a request for that target that the host has not applied yet. `applied.json` records the last-applied request as `channelIdentity`. A record without `channelIdentity` was written by an earlier release. For such a record, only its accepted commit counts as the applied request. The first unchanged run records the identity. If another environment was promoted first, the host applies its own request once more. ## The channel ``` { "release": "v1.2.3", "allowRollback": false, "freeze": false } ``` | Field | Effect | | --------------- | ---------------------------------------------------------------------------------------------- | | `release` | the `vX.Y.Z` release to apply | | `allowRollback` | lets the host apply an older release or commit. It does not restore the database or any volume | | `freeze` | keeps a following environment on this target: automatic promotion leaves the channel alone | Staging can follow successful builds of `main` instead of releases. Its channel names a full `commit` SHA instead of `release`, plus an `images` map of digest-pinned references and a `refreshDatabaseImage` flag (see [What restarts the database](#what-restarts-the-database)). The promotion workflow verifies first-party image provenance against that source commit before signing the channel. Upstream image pins come from that commit's inventory. Automatic promotions check the latest channel while holding the promotion lock. A frozen channel, an already-promoted commit, or an older completed build leaves the channel unchanged, including when switching from a release to a commit. An unreadable channel or failed history comparison stops promotion rather than clearing a freeze. Establish the initial channel with a manual promotion before enabling continuous staging. A host applies a frozen channel like any other: `freeze` matters only to automatic promotion. Thus, one promotion moves Staging to a release or commit and keeps it there. Only Staging follows `main`, so only Staging can be frozen. A host that runs tooling from before this rule ignores a frozen channel. On such a host, a frozen promotion to a new target fails its version check. Promote that host once without **freeze** first, to a commit that includes this rule. An automatic promotion that would advance the channel checks the most recent successful CI/CD run of a push to `main` in the same repository. If that run's commit strictly descends from the requested commit, the promotion moves the channel to that commit instead. The run's title still names the requested commit. The channel and the deployment record name the promoted one. If the lookup fails, the requested commit continues through the existing promotion checks. Manual promotions never do this lookup. ## Requirements 1. For Compose stacks, install Docker Engine and Docker Compose in the [tested versions](/admin/install.md#tested-versions). For the chart, see [Follow a channel with the chart](#follow-a-channel-with-the-chart). 2. Install `git` and Cosign in the [tested version](/admin/install.md#tested-versions). 3. Install the Node.js version pinned in `package.json#devEngines.runtime`. 4. Verify downloaded tools against their published checksums. 5. Allow outbound HTTPS to the host destinations in [Network egress](/admin/network-egress.md#host). ## Install The initial clone is the first tooling that the host runs. After its first applied release, the host uses that release's reconciler and verifier through `/var/lib/hephaestus/tooling`. The reconciler moves the link only after it verifies and applies a release. A release never replaces the tooling that verifies it. 1. Install the trusted reconciler checkout: ``` sudo install -d -m 0750 /var/lib/hephaestus sudo git clone --branch main --single-branch \ https://github.com/hephaestus-build/Hephaestus.git /var/lib/hephaestus/checkout sudo ln -sfn /var/lib/hephaestus/checkout /var/lib/hephaestus/tooling ``` 2. Copy `deploy/reconcile/reconcile.env.example` to `/etc/hephaestus/reconcile.env`. 3. Set its ownership to `root:root` and mode to `0600`. 4. Set the channel, the stacks this host owns, and the promotion identity. 5. Put one `.env` per stack in `/etc/hephaestus`, with mode `0600`. These files hold deployment secrets and never leave the host. 6. Install the units: ``` sudo install -m 644 \ /var/lib/hephaestus/checkout/deploy/reconcile/hephaestus-reconcile.service \ /var/lib/hephaestus/checkout/deploy/reconcile/hephaestus-reconcile.timer \ /etc/systemd/system/ sudo systemctl daemon-reload sudo systemctl enable --now hephaestus-reconcile.timer ``` The units are copies because systemd needs them on the root filesystem at boot. The reconciler keeps both at the host's release. It reloads systemd when it rewrites a unit. This is therefore the last manual installation of the units. 7. Watch the first run with: ``` journalctl -fu hephaestus-reconcile.service ``` ### Moving an existing host A host with an existing Hephaestus database keeps the secrets used to write that database. Pull-based deployment does not change them. Operators write the per-stack files in `/etc/hephaestus` (`app.env`, `core.env`) manually here. `docker/self-host/setup.sh` and its guard belong to the Compose installer. 1. Before the first reconcile, copy these settings from the previous deployment: | Setting | What it must be | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `HEPHAESTUS_SECURITY_ENCRYPTION_KEY` | The previous deployment's value. It encrypted the server's stored values other than integration credentials. | | `HEPHAESTUS_SECURITY_CREDENTIAL_ENCRYPTION_KEY` | The previous deployment's value, or the master key if the installation never set it separately. It encrypted integration credentials. A different key makes them unreadable without a warning. | | `HEPHAESTUS_SECURITY_CREDENTIAL_ROTATION_ENABLED`, `HEPHAESTUS_SECURITY_CREDENTIAL_ENCRYPTION_KEY_VERSION`, `HEPHAESTUS_SECURITY_PRIOR_CREDENTIAL_ENCRYPTION_KEY`, `HEPHAESTUS_SECURITY_PRIOR_CREDENTIAL_ENCRYPTION_KEY_VERSION` | The previous deployment's values, if it rotated credential keys. Each stored credential names the key version that wrote it. A credential under the prior key remains readable only while that key is configured. | | `WEBHOOK_SECRET` | what the providers were registered to sign with | | `HEPHAESTUS_AUTH_STATE_COOKIE_KEY` | optional. A new one only invalidates sign-ins that are in flight at the moment of the switch | A host with its own edge keeps certificates in the `proxy_letsencrypt` volume. In a release worktree, the old bind-mounted `acme.json` is under `docker/letsencrypt/`. The reconciler never prunes a worktree that still holds that file. CAUTION: Preserve the active ACME store An existing `acme.json` in the volume is Traefik's active store. An older backup would discard certificates currently in service. The command below refuses that replacement. Inspect an existing store deliberately before replacing it. Before reconciling a host that used the old bind mount: 2. Pause the timer. 3. Stop its existing Traefik container so the ACME store cannot change during the copy. 4. Back up `acme.json` first. 5. Before the new proxy starts, copy it into the volume. Otherwise, Traefik issues the certificates again. 6. On the Docker Engine host, use the volume's actual mountpoint, not an assumed Docker data root: ``` sudo docker volume create proxy_letsencrypt volume_path=$(sudo docker volume inspect --format '{{ .Mountpoint }}' proxy_letsencrypt) if sudo test -s "$volume_path/acme.json"; then echo "acme.json is already in proxy_letsencrypt — inspect it before replacing it" >&2 else sudo install -m 600 /path/to/previous/acme.json "$volume_path/acme.json" fi ``` 7. Keep the backup until the new proxy successfully serves the existing certificates. 8. Then re-enable reconciliation. For rollback to a release whose proxy still uses the bind mount: 1. Stop Traefik. 2. Copy the current ACME store back. The old release tree alone does not keep that copy current. The pull-based stacks use the bundled database with fixed credentials. No database password therefore needs transfer. On the Compose installer, `docker/self-host/setup.sh` refuses to generate an encryption key while `hephaestus_postgresql-data` exists. It stops rather than guesses if Docker cannot list volumes. ### Upgrading a host installed before the tooling link A host installed before `/var/lib/hephaestus/tooling` existed runs the reconciler directly from its checkout. That checkout did not update. The commands below advance it manually once. Afterward, the reconciler maintains the link and units. CAUTION: Do not change a checkout during reconciliation A oneshot service remains `activating` while it runs. Stopping the timer does not stop an active tick. The commands first stop the timer and wait for an active tick to finish. Do not change the checkout underneath a tick. 1. Run this block: ``` sudo systemctl stop hephaestus-reconcile.timer while [ "$(systemctl show --property=ActiveState --value hephaestus-reconcile.service)" = activating ]; do sleep 5; done sudo git -C /var/lib/hephaestus/checkout fetch --quiet origin main sudo git -C /var/lib/hephaestus/checkout reset --quiet --hard origin/main sudo ln -sfn /var/lib/hephaestus/checkout /var/lib/hephaestus/tooling sudo install -m 0644 \ /var/lib/hephaestus/checkout/deploy/reconcile/hephaestus-reconcile.service \ /var/lib/hephaestus/checkout/deploy/reconcile/hephaestus-reconcile.timer \ /etc/systemd/system/ sudo systemctl daemon-reload sudo systemctl start hephaestus-reconcile.service sudo systemctl start hephaestus-reconcile.timer sudo readlink /var/lib/hephaestus/tooling ``` 2. Run the **Release / Promote** workflow for the release that the host already runs. The host cannot complete an old record that lacks the release commit from files on disk. Those files cannot distinguish the accepted tree from one left by a failed promotion. The host therefore keeps the checkout's tooling and reports `hephaestus_deploy_tooling_pending 1`. The next apply records the commit and adopts that release's tooling. After a tick adopts the tooling, the block's last line names the running release's worktree. The service starts only through the link. 3. If the link is missing or points to a removed directory, recreate it with the `ln` line above. If a recorded release's worktree under `releases/` was deleted, the reconciler rebuilds it before any other action. It uses the accepted commit, not the tag's current target. If it cannot rebuild the worktree, such as when the checkout lost that commit, the tick stops with the reason. `HEPHAESTUS_STACKS` selects the host's managed Compose projects. 4. If another proxy supplies ingress, omit `proxy` and use `core app`. ## Follow a channel with the chart A host can apply its channel with the [Helm chart](/admin/kubernetes.md) instead of Compose stacks. A release channel verifies the signed joint lock, its evidence manifest, and the chart signature. It pulls the chart by its OCI digest and checks the reported digest, metadata, and image pins. The installed verifier performs these checks before any candidate tooling runs. It does not download a schema-1 lock or apply the source chart for a release channel. A commit channel applies the source chart with the image pins from its signed channel. A commit chart has no release version. To switch it to a release channel, pause reconciliation and apply the verified target through Database maintenance first. Then resume the timer. The next tick reads the target chart's release version. Prepare the verified target source tree first, as [Database maintenance](/admin/kubernetes.md#database-maintenance) describes. Keep writers stopped until the target migration and server succeed. Use the target source commit from its verified signed lock as ``: ``` sudo install -d -m 0750 /var/lib/hephaestus/chart sudo node /var/lib/hephaestus/tooling/scripts/prepare-release-chart.ts \ /var/lib/hephaestus/chart helm \ /var/lib/hephaestus/releases//security/release-images.json helm upgrade --install /var/lib/hephaestus/chart/hephaestus-.tgz \ --namespace \ --values /var/lib/hephaestus/releases//deploy/chart/hephaestus/profiles/.yaml \ --values /etc/hephaestus/chart.yaml \ --values /var/lib/hephaestus/chart/.images.yaml \ --set server.replicas=1 --set worker.replicas=0 \ --wait=watcher --wait-for-jobs --timeout s ``` Install the 0.95.1 reconciler before promoting 1.0. The reconciler units now live in `deploy/reconcile/`, independent of the Compose files. To update an existing 0.95.0 reconciler, stop its timer and wait for the active tick to finish, as [Upgrading a host](#upgrading-a-host-installed-before-the-tooling-link) describes. Then run these commands. The installed 0.95.0 verifier checks the 0.95.1 lock before the host runs the new tooling. This bootstrap uses the 0.95.1 lock only. The 1.0 chart path needs no schema-1 asset. ``` sudo install -d -m 0750 /var/lib/hephaestus/tooling-update sudo node /var/lib/hephaestus/tooling/scripts/prepare-release-lock.ts \ v0.95.1 /var/lib/hephaestus/tooling-update/lock.env tooling_commit=$(sudo sed -n 's/^HEPHAESTUS_RELEASE_COMMIT=//p' /var/lib/hephaestus/tooling-update/lock.env) sudo git -C /var/lib/hephaestus/checkout fetch --quiet origin tag v0.95.1 if [ "$(sudo git -C /var/lib/hephaestus/checkout rev-parse 'v0.95.1^{commit}')" = "$tooling_commit" ]; then sudo git -C /var/lib/hephaestus/checkout worktree add --detach \ /var/lib/hephaestus/tooling-update/v0.95.1 "$tooling_commit" sudo install -m 0644 \ /var/lib/hephaestus/tooling-update/v0.95.1/deploy/reconcile/hephaestus-reconcile.service \ /var/lib/hephaestus/tooling-update/v0.95.1/deploy/reconcile/hephaestus-reconcile.timer \ /etc/systemd/system/ sudo systemctl daemon-reload sudo ln -sfn /var/lib/hephaestus/tooling-update/v0.95.1 /var/lib/hephaestus/tooling sudo systemctl start hephaestus-reconcile.timer else echo "v0.95.1 does not match the verified lock" >&2 fi ``` After an apply, the reconciler completes both unit copies and reloads systemd before it moves its tooling link. If that step stops, the next tick completes it before reading another channel. ### Requirements for the chart 1. A cluster and the platform items that [Kubernetes](/admin/kubernetes.md#supported-clusters) lists. 2. The namespaces, the Secrets and the values that [Kubernetes § Install](/admin/kubernetes.md#install) describes. 3. `kubectl` on `PATH`, with access to read Pods and Jobs in the release and sandbox namespaces. 4. Helm 4.3.1. Install the pinned binary and verify its checksum. On `amd64`: ``` curl -fsSLO https://get.helm.sh/helm-v4.3.1-linux-amd64.tar.gz echo "0403a88ae20913d11b10a2467b50087721fea7581736da6b9d3992f5357fe54b helm-v4.3.1-linux-amd64.tar.gz" \ | sha256sum --check tar -xzf helm-v4.3.1-linux-amd64.tar.gz linux-amd64/helm sudo install -m 755 linux-amd64/helm /usr/local/bin/helm ``` On `arm64`, use `linux-arm64` and the checksum `8cf8f66ecbc252ac8f7f06cca68aba88508ddfde3a6a34feb1678ad1694b4f88`. ### Settings for the chart Set these values in `/etc/hephaestus/reconcile.env`: | Setting | Default | Value | | ---------------------------- | ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `HEPHAESTUS_STACKS` | None | `chart`. It takes no other stack. | | `HEPHAESTUS_CHART_RELEASE` | `hephaestus` | The Helm release name. | | `HEPHAESTUS_CHART_NAMESPACE` | `hephaestus` | The release namespace. | | `HEPHAESTUS_CHART_PROFILE` | `small` | One or more values files in the chart's `profiles/`, without `.yaml`, separated by spaces or commas. Each profile applies over the one before it, for example `"small small-16gib"`. The reconciler refuses a profile that the release does not have. | | `HEPHAESTUS_HELM` | `helm` | The path of the Helm binary, for example `/usr/local/bin/helm`. | | `KUBECONFIG` | None | The cluster's kubeconfig. On k3s, `/etc/rancher/k3s/k3s.yaml`. | | `HEPHAESTUS_WAIT_TIMEOUT` | `1200` on a chart host | The Helm timeout in seconds. It must be longer than the 10 minutes that a new Pod waits for the schema. | Put your values from [Kubernetes § Install](/admin/kubernetes.md#install) in `/etc/hephaestus/chart.yaml`. Set its ownership to `root:root` and its mode to `0600`. Do not set `images` or `release` in it. The reconciler writes the verified pins to `/var/lib/hephaestus/chart/.images.yaml`. That file is the last values file, so it overrides any image in your file. A commit apply runs this command: ``` helm upgrade --install /deploy/chart/hephaestus \ --namespace \ --values /deploy/chart/hephaestus/profiles/.yaml \ --values /deploy/chart/hephaestus/profiles/.yaml \ --values /etc/hephaestus/chart.yaml \ --values /var/lib/hephaestus/chart/.images.yaml \ --rollback-on-failure --wait=watcher --wait-for-jobs --timeout s ``` A release apply uses `/var/lib/hephaestus/chart/hephaestus-.tgz` as the chart argument. It uses that release's profiles and pinned values. The chart and image downloads have finite time limits. Helm status and history reads have a two-minute limit. The upgrade and recovery commands have deadlines derived from `HEPHAESTUS_WAIT_TIMEOUT`. The unit stops the whole tick after two hours. If you increase the Helm timeout, set `TimeoutStartSec` above three times that timeout plus 40 minutes. Use a systemd drop-in for the override. The revision's Job migrates the database. If the migration or a Pod fails, Helm returns the workloads to their previous revision. Helm does not roll back the database. The tick then fails, the metrics show the failure, and the applied record does not move. An upgrade that stops before it ends, for example at a reboot or at the unit's timeout, leaves the Helm release in a `pending-*` status. Helm then refuses every upgrade. For an upgrade within the same major version, the next tick finds that status. It rolls back to the newest revision whose status is `deployed` or `superseded`, then upgrades. To do it by hand, read `helm history --namespace ` and run `helm rollback --namespace --wait`. A release that Helm never finished, such as a first install that stopped, has no such revision. The tick then fails and names `helm uninstall --namespace `. The uninstall keeps the data volumes and Secrets ([Kubernetes § Uninstall](/admin/kubernetes.md#uninstall)). After you run it, the next tick installs the release again. ### Stopped major upgrades For 0.95.x → 1.0, follow [Database maintenance](/admin/kubernetes.md#database-maintenance) and [Upgrade to 1.0](/admin/upgrade-to-1-0.md). Pause the timer before phase 1. Save `/var/lib/hephaestus/applied.json` as `/var/lib/hephaestus/applied-before-cutover.json` before you change the release. Record the stopped phase-1 Helm revision from `helm history --namespace `. Verify the backups, record the worker count, and use the verified source chart to stop the server and workers. Wait until their Pods and every sandbox Pod and Job are removed. Then promote the target release and restart the timer for phase 2. The reconciler checks both its applied record and the installed chart version. For a major transition, it requires zero server and worker replicas in the installed values and no remaining writer workloads. It then applies the verified target chart with one server and zero workers. If Helm finished before the applied record was written, the next tick checks the target identity and image pins. A matching deployed target completes its record without running the migration hooks again. This also applies after you restore the worker count. It omits `--rollback-on-failure` because Helm cannot restore the old schema. A pending cutover stops reconciliation and requires database recovery. It never rolls old workloads onto a migrated database. After the server is ready, pause the timer and restore the recorded worker count with the verified target chart, as phase 4 describes. Keep the desired worker count in `/etc/hephaestus/chart.yaml`, then resume the timer. A new promotion of the same target can also restore that count after the cutover succeeds. The reconciler removes no image on a chart host. The kubelet's image garbage collection removes unused images. ### Recover a stopped cutover Before you promote 1.0, set `HEPHAESTUS_WAIT_TIMEOUT` above the expected migration time. Set the systemd deadline as [Settings for the chart](#settings-for-the-chart) describes. A Helm timeout stops its wait. It does not stop a migration Job. If a stopped cutover fails or remains pending: 1. Stop the timer: `sudo systemctl stop hephaestus-reconcile.timer`. 2. Wait for the active reconciler service to finish. 3. Read `helm history --namespace ` and record the failed revision. 4. Inspect `kubectl get job -migration- --namespace `. Wait for its Complete or Failed condition before you restore data. After its 24-hour TTL, an absent Job is finished only if no migration Pod remains. If you must cancel it, delete that Job and wait for its Pods to disappear first. 5. Restore the pre-upgrade database as [If migration fails](/admin/upgrade.md#if-migration-fails) describes. Keep every database writer stopped during the restore. 6. Run `helm rollback --namespace --wait`. Use the recorded source revision with zero server and worker replicas. 7. To retry the target, resume the timer. To return to the source release, promote its signed release with rollback enabled, then resume the timer. 8. After the selected release succeeds, restore the worker count as phase 4 describes. A signed rollback request does not permit an automatic major release downgrade. Before a commit rollback across a schema boundary, restore the database through the same manual procedure. If the completed target already has an applied record, restore `applied-before-cutover.json` to `applied.json` after both restores succeed. Then promote the source release and resume the timer. A commit channel must descend from the recorded release source, unless its signed request permits rollback. ### Switch a host between Compose and the chart To move a Compose host to the chart: 1. Disable the timer: `sudo systemctl disable --now hephaestus-reconcile.timer`. 2. Install the chart at the commit that the host runs, as [Kubernetes § Install](/admin/kubernetes.md#install) describes, and move the data. 3. Stop the Compose stacks with `docker compose stop`. Keep their volumes until the chart has run for one release cycle. 4. Set `HEPHAESTUS_STACKS=chart` and the chart settings, and write `/etc/hephaestus/chart.yaml`. 5. Run the reconciler once from a tree that has the chart stack. Each tick first adopts the tooling of the release that the host applied. A host that applied a release from before the chart stack therefore runs tooling that refuses `HEPHAESTUS_STACKS=chart`. Run the reconciler of a newer `main` commit by hand, with the settings of `hephaestus-reconcile.service`: ``` COMMIT= sudo git -C /var/lib/hephaestus/checkout fetch origin +refs/heads/main:refs/remotes/origin/main sudo git -C /var/lib/hephaestus/checkout worktree add --detach /var/lib/hephaestus/switch "$COMMIT" sudo systemd-run --unit=hephaestus-reconcile-switch --wait --collect -p Type=oneshot \ -p EnvironmentFile=/etc/hephaestus/reconcile.env \ -p StateDirectory=hephaestus -p StateDirectoryMode=0750 -p TimeoutStartSec=30min \ -p Environment=HELM_CACHE_HOME=/var/lib/hephaestus/helm/cache \ -p Environment=HELM_CONFIG_HOME=/var/lib/hephaestus/helm/config \ -p Environment=HELM_DATA_HOME=/var/lib/hephaestus/helm/data \ -p NoNewPrivileges=yes -p ProtectSystem=strict -p ProtectHome=yes -p PrivateTmp=yes \ -p ReadWritePaths=-/var/lib/node_exporter -p ReadWritePaths=/etc/systemd/system \ -p "RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6" -p RestrictSUIDSGID=yes \ -p RestrictRealtime=yes -p LockPersonality=yes \ -p SystemCallFilter=@system-service -p SystemCallErrorNumber=EPERM \ /usr/bin/env node /var/lib/hephaestus/switch/scripts/reconcile-deployment.ts sudo git -C /var/lib/hephaestus/checkout worktree remove /var/lib/hephaestus/switch ``` The run verifies the channel and the lock, applies the release that the channel names with Helm, and records it. It then adopts the tooling and the unit of that release. 6. Enable the timer: `sudo systemctl enable --now hephaestus-reconcile.timer`. A tick does nothing while the channel names the release that the host runs. The next promotion runs Helm. To move the host back to Compose: 1. Disable the timer. 2. Stop the cluster. On k3s, run `sudo systemctl stop k3s`. 3. Set `HEPHAESTUS_STACKS` to the Compose stacks that the host had, for example `"core app"`. 4. Start each stack, for example `sudo docker compose --project-name app start`, then enable the timer. Data that the chart wrote after the switch is not in the Compose database. ## Operating it | You want to | Do this | | ------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Move an environment | Run the **Release / Promote** workflow: pick the environment and release or commit | | Roll back | Pause reconciliation, then follow [Rollback](/admin/upgrade.md#rollback). After the database matches the older release, promote that release with **allow-rollback** | | Keep Staging on a release or commit | Promote it with **freeze** | | Let Staging follow `main` again | Promote a commit of `main` with **freeze** cleared. The next successful build moves it on. Promoting the commit that runs applies it again | | Reapply Compose configuration | Promote the release or commit the host already runs | | Move a following host onto the commit's database image | Promote the commit with **refresh-database-image** | | Prevent new reconciliation runs | Use the timer-disable command below | | See what a host runs | Use the applied-record command below | | See what was promoted to an environment, and when | Open the environment's deployment history on GitHub | | Keep a release worktree while you work in it | Lock it. See [What it keeps on disk](#what-it-keeps-on-disk) | Timer-disable command: ``` sudo systemctl disable --now hephaestus-reconcile.timer ``` Applied-record command: ``` cat /var/lib/hephaestus/applied.json ``` CAUTION: Pause reconciliation before manual changes An active reconciliation can conflict with manual changes. Disable the timer before editing. Confirm that the service no longer runs with: ``` systemctl status hephaestus-reconcile.service ``` A disabled timer does not stop an active reconciliation. Re-enable the timer after recovery. A new promotion reapplies Compose configuration, not arbitrary container-file changes. The reconciler refuses a release checkout with modified tracked files. ### How an apply stops the worker An apply stops the worker before it replaces a service that the worker depends on. After the signed lock and every rendered Compose configuration of the new release pass their checks, the reconciler pulls the images of each stack with `docker compose pull --policy missing`. A pull starts no container, so the running worker continues during the download. Then the reconciler runs the [database migration](/admin/runtime-roles.md#database-migrations) of the new release, while every role of the previous release still runs. A PostgreSQL that runs is left alone: the `up` that replaces the roles applies a changed PostgreSQL service. If the migration fails, the reconciler stops nothing, applies no stack and records no release. Then the reconciler stops the running worker. It uses the `app` configuration and retained lock of the release that the host applied last. If a pull fails, the reconciler stops nothing, applies no stack and records no release. This also stops the worker when the new release removes it. It waits until the stop is complete and then applies the stacks. A host applies a release with the reconciler of the release that it applied last. The first apply of the release that adds the migration service therefore uses a reconciler without the migration step. That apply stops the previous roles before Compose runs the migration. Compose waits up to the `stop_grace_period` that the running worker was started with while the worker drains, then stops the container. A review that is still running at that time stops with the worker. The database stays available during the stop. The `app` stack's normal `up` starts the worker of the new release, if that release has one. There is nothing to stop on a first install, or when the applied release has no `application-worker` service. A host without the `app` stack skips the stop. When the host already applied its own request and that request is unchanged, the worker does not stop, even if another environment is promoted. If the stop fails, the reconciler applies no stack and records no release. The next tick tries again. The reconciler also applies nothing when it cannot identify the worker that runs. The applied record must identify its source commit. The retained lock must be present and name that same commit. Verify the revision that the host runs before you let it apply another release. The reconciler runs the tooling that the host already adopted. Thus, the apply that brings this step runs without it, and later applies include it. ### What restarts the database An apply recreates a container whose image or Compose definition changed. PostgreSQL recreation drops all database connections until the application services stop and return. Each default-branch commit rebuilds the PostgreSQL image to include Debian security updates. A host that follows commits therefore receives a new digest on each apply. Applying each digest would restart the database several times a day. The reconciler instead keeps the current image and applies the rest of the channel. On such a host, the database restarts in these cases: * The commit changes `docker/postgres/` or the workflows that build the image. * The host applies the first commit of a later day. That day's security updates arrive within a day of the push that built them. * The `postgres` service in `docker/compose.app.yaml` changes. * The host has no usable image record. This includes a missing applied commit, missing lock, or checkout unable to resolve the two commits. * You promote with **refresh-database-image**. That takes the commit's image once. The next automatic promotion carries it forward again. A release is applied exactly as its signed lock names it, so a release that rebuilt the image restarts the database. ### What it keeps on disk Each applied release or commit gets a Git worktree under `/var/lib/hephaestus/releases/` and a lock under `/var/lib/hephaestus/release-locks/`. After a successful apply, the reconciler removes unnecessary worktrees with `git worktree remove`. It then removes the locks of releases with no worktree. It keeps these worktrees: * The release just applied. * The previous running release, so rollback starts from an existing worktree. Promoting the running release again also keeps this previous release. * The worktree named by `/var/lib/hephaestus/tooling`. * A worktree locked with `git worktree lock` for manual work. * A worktree with files absent from its release, including modified, untracked, or ignored files. This includes `acme.json` written to `docker/letsencrypt/` by an older proxy. The journal names each worktree the reconciler removes, and says why it keeps a locked worktree or one that holds files. Pruning deletes worktrees and lock files only. It never touches a Docker volume, so the certificates in `proxy_letsencrypt` and the database stay as they are. 1. To keep a worktree during use, lock it. 2. When you finish, unlock it: ``` sudo git -C /var/lib/hephaestus/checkout worktree lock --reason "incident review" \ /var/lib/hephaestus/releases/ sudo git -C /var/lib/hephaestus/checkout worktree unlock /var/lib/hephaestus/releases/ ``` A worktree kept because of its files remains until you remove it. 3. Save its files first. 4. Use: ``` sudo git -C /var/lib/hephaestus/checkout worktree remove --force ``` A rollback to a pruned release creates its worktree again at the commit the host accepts for it. The proxy then finds its certificates in the volume, so nothing is issued again. This is not true for a release whose proxy still used the bind mount. See [Moving an existing host](#moving-an-existing-host). If pruning fails, the pass fails after the new release is running, recorded and its tooling adopted. The journal gives the reason. The next apply prunes everything that is still obsolete. Systemd runs one pass at a time. This prevents another pass from using a worktree that pruning removes. 1. To run a pass manually, use: ``` sudo systemctl start hephaestus-reconcile.service ``` 2. Do not start the reconciler directly. ### Images and free space After it prunes the worktrees, the reconciler removes the images that neither the release just applied nor the previous release names. It considers only the repositories that these two releases name, and only images without a tag. A release image is pulled by digest and has no tag, so a tagged image of another workload stays. Docker refuses to remove an image that a container uses, running or stopped. The journal names each image that is removed or kept. A rollback to the previous release therefore needs no download. A rollback to an older release downloads its images again. If a step after the apply fails, the next apply removes the images. Before it pulls, the reconciler checks the free space on the filesystems that hold Docker's data and the containerd image store. It checks only when the release needs a pull, so a rollback whose images are on the host is never refused. Below `HEPHAESTUS_MIN_FREE_GIB`, it first removes the images that neither the running, the previous nor the new release names. If the space is still too small, it refuses the apply, stops nothing and records nothing. The journal states the free space and the threshold, and the next tick tries again. [Sizing § Disk](/admin/sizing.md#disk) gives the default threshold and the disk budget. #### Recover a host that is already full A host runs the reconciler of the release it last applied. A reconciler from before the image cleanup removes no images. When such a host is full, its pull fails with `no space left on device`, and it cannot pull the release that brings the cleanup. If you set `HEPHAESTUS_METRICS_FILE`, a file without `hephaestus_deploy_free_bytes` shows a reconciler from before the cleanup. Without the file, you can still follow these steps: they are safe on a newer reconciler too. Free the space once by hand: 1. Disable the timer with `sudo systemctl disable --now hephaestus-reconcile.timer`. 2. Wait for an active tick to finish: ``` while [ "$(systemctl show --property=ActiveState --value hephaestus-reconcile.service)" = activating ]; do sleep 5; done ``` 3. In `/var/lib/hephaestus/applied.json`, read `release` and, if it is there, `previous`. 4. Open `/var/lib/hephaestus/release-locks/.env` for each of them. If a lock is missing, stop. Enable the timer again as step 11 shows. Do not guess which images to keep. 5. For each `HEPHAESTUS_IMAGE_*` value in the locks, read its image ID with `docker image inspect --format '{{.Id}}' `. If Docker reports no such image, the host does not hold it. Skip it. These are the kept IDs. The agent image and the carried PostgreSQL image are among them. 6. List the images on the host with `docker image ls --all --digests --no-trunc`. Docker Engine 29 with the `overlay2` storage driver lists an image that has no tag only with `--all`. A release image is pulled by digest, so it has no tag. 7. Find the images of the locks' repositories whose `TAG` is `` and whose `IMAGE ID` is not a kept ID. Leave an image that has a tag: a release image never has one. A `` repository is a layer, not an image of these repositories. 8. Remove each of them with `docker image rm @`. Do not force the removal. Docker refuses only an image that a container uses, and the agent image has no container between reviews. 9. Find Docker's data directory with `docker info --format '{{.DockerRootDir}}'`. 10. Check the free space with `df -h` on that directory and on `/var/lib/containerd`, if it exists. Both need at least the free space that [Sizing § Disk](/admin/sizing.md#disk) sets as the default threshold. If they do not, free more space or extend the disk, and check again. 11. Enable the timer again with `sudo systemctl enable --now hephaestus-reconcile.timer`. The next tick applies a pending release. From then on, the reconciler keeps the images itself. ## Watching it 1. Set `HEPHAESTUS_METRICS_FILE` to a `.prom` file in an existing node\_exporter textfile collector directory. After deployment succeeds, each run writes the metrics below. A run that only adopts the applied release's tooling leaves them unchanged: ``` hephaestus_deploy_info{channel="staging",release="v1.2.3",channel_commit="…"} 1 hephaestus_deploy_reconcile_success 1 hephaestus_deploy_reconcile_timestamp_seconds … hephaestus_deploy_tooling_pending 0 hephaestus_deploy_last_success_timestamp_seconds … hephaestus_deploy_free_bytes … ``` Every run that writes metrics, including a failed run, reports `hephaestus_deploy_free_bytes`, the least free space on the filesystems that hold Docker's data. Warn when it falls below twice `HEPHAESTUS_MIN_FREE_GIB`, before the reconciler refuses an apply. The group `hephaestus-pull-deploy` of the shipped [alert rules](/admin/observability.md#alert-rules) alerts on these metrics. It covers a failing reconciler, a reconciler that finishes no run, absent metrics, and tooling that no apply recorded. `hephaestus_deploy_last_success_timestamp_seconds` records the last successful **apply**. An unchanged channel does not advance it, so its age is not a reconciliation-liveness check. A promotion waits for the public webapp to report the requested version. This checks the webapp version, not the readiness of every service. Use host metrics and logs for the Compose result. A promotion also records a GitHub deployment of the release or commit that it promotes. The record names who requested the promotion. Its final state tells whether the public webapp reported the version. Its time is when the promotion ran, not when a host applied it. Host metrics remain the record of what each host runs. A skipped automatic promotion records no deployment. [Release Management § Deployment history](/contributor/release-management.md#deployment-history) lists the recorded fields. The promotion job summary explicitly names this boundary, including when a version check fails. It does not fetch host metrics. The textfile collector and authenticated monitoring access belong to the host, not the public application. A matching frontend version does not establish server, worker, webhook, database, NATS, or integration readiness. 1. On **each** participating host, confirm a fresh successful reconciliation. 2. Confirm the expected release/channel commit. 3. Inspect Compose health. 4. Inspect service logs. 5. Do not expose management endpoints to replace operator verification with the public version check. ## What it will not do **It does not restore a database.** A failed Compose apply or stopped major upgrade may leave some services updated. Schema migrations are forward-only. Older images may be incompatible with the migrated database. Promote a fix, or follow [Rollback](/admin/upgrade.md#rollback) if database recovery is needed. **It does not manage anything it does not own.** The reconciler touches only the Compose projects named in `HEPHAESTUS_STACKS`, or the one Helm release of a chart host. Containers belonging to anything else on the host are left alone.