Skip to main content

Pull-Based Deployment

A host polls a channel naming a release or commit, verifies its signature, and applies the verified Helm chart or digest-pinned Compose stacks. Deployment needs no inbound administration connection to the host.

What the host trusts​

For a chart release deployment, the host verifies three signatures:

ArtifactSigned byAnswers
The channel (channels/<env>.json)the promotion workflowmay this host move, and who said so
The joint lock (release-<version>.chart.json)the release workflowwhich source, chart digest, and image digests form this release
The chart at its OCI digestthe release workflowwhether the chart package came from that workflow

Compose deployments verify the channel and the schema-1 release lock (release-<version>.json).

The host rejects a new channel commit that does not descend from its last accepted channel commit. It also rejects lower release versions and older commits unless allowRollback is set. These checks do not order manual transitions between releases and commits. Choose those targets deliberately.

Each environment acts only on its own signed request: its channel file and signature bundle together. A promotion of another environment also adds a commit to deploy-state. If the host already applied its own unchanged request, it verifies the signature again. It records the new commit as accepted. It restarts nothing.

If its own request is not applied yet, for example after a failed apply, the host tries it again. Each promotion signs again, so a promotion of the target that already runs applies again. A frozen promotion of the target that already runs is the exception: it restarts nothing, unless it also asks for refresh-database-image. A frozen promotion that does not ask for it also replaces a request for that target that the host has not applied yet.

applied.json records the last-applied request as channelIdentity. A record without channelIdentity was written by an earlier release. For such a record, only its accepted commit counts as the applied request. The first unchanged run records the identity. If another environment was promoted first, the host applies its own request once more.

The channel​

{ "release": "v1.2.3", "allowRollback": false, "freeze": false }
FieldEffect
releasethe vX.Y.Z release to apply
allowRollbacklets the host apply an older release or commit. It does not restore the database or any volume
freezekeeps a following environment on this target: automatic promotion leaves the channel alone

Staging can follow successful builds of main instead of releases. Its channel names a full commit SHA instead of release, plus an images map of digest-pinned references and a refreshDatabaseImage flag (see What restarts the database). The promotion workflow verifies first-party image provenance against that source commit before signing the channel. Upstream image pins come from that commit's inventory.

Automatic promotions check the latest channel while holding the promotion lock. A frozen channel, an already-promoted commit, or an older completed build leaves the channel unchanged, including when switching from a release to a commit. An unreadable channel or failed history comparison stops promotion rather than clearing a freeze. Establish the initial channel with a manual promotion before enabling continuous staging.

A host applies a frozen channel like any other: freeze matters only to automatic promotion. Thus, one promotion moves Staging to a release or commit and keeps it there. Only Staging follows main, so only Staging can be frozen. A host that runs tooling from before this rule ignores a frozen channel. On such a host, a frozen promotion to a new target fails its version check. Promote that host once without freeze first, to a commit that includes this rule.

An automatic promotion that would advance the channel checks the most recent successful CI/CD run of a push to main in the same repository. If that run's commit strictly descends from the requested commit, the promotion moves the channel to that commit instead. The run's title still names the requested commit. The channel and the deployment record name the promoted one. If the lookup fails, the requested commit continues through the existing promotion checks. Manual promotions never do this lookup.

Requirements​

  1. For Compose stacks, install Docker Engine and Docker Compose in the tested versions. For the chart, see Follow a channel with the chart.
  2. Install git and Cosign in the tested version.
  3. Install the Node.js version pinned in package.json#devEngines.runtime.
  4. Verify downloaded tools against their published checksums.
  5. Allow outbound HTTPS to the host destinations in Network egress.

Install​

The initial clone is the first tooling that the host runs. After its first applied release, the host uses that release's reconciler and verifier through /var/lib/hephaestus/tooling. The reconciler moves the link only after it verifies and applies a release. A release never replaces the tooling that verifies it.

  1. Install the trusted reconciler checkout:

    sudo install -d -m 0750 /var/lib/hephaestus
    sudo git clone --branch main --single-branch \
      https://github.com/hephaestus-build/Hephaestus.git /var/lib/hephaestus/checkout
    sudo ln -sfn /var/lib/hephaestus/checkout /var/lib/hephaestus/tooling

  2. Copy deploy/reconcile/reconcile.env.example to /etc/hephaestus/reconcile.env.

  3. Set its ownership to root:root and mode to 0600.

  4. Set the channel, the stacks this host owns, and the promotion identity.

  5. Put one <stack>.env per stack in /etc/hephaestus, with mode 0600. These files hold deployment secrets and never leave the host.

  6. Install the units:

    sudo install -m 644 \
      /var/lib/hephaestus/checkout/deploy/reconcile/hephaestus-reconcile.service \
      /var/lib/hephaestus/checkout/deploy/reconcile/hephaestus-reconcile.timer \
      /etc/systemd/system/
    sudo systemctl daemon-reload
    sudo systemctl enable --now hephaestus-reconcile.timer

    The units are copies because systemd needs them on the root filesystem at boot. The reconciler keeps both at the host's release. It reloads systemd when it rewrites a unit. This is therefore the last manual installation of the units.

  7. Watch the first run with:

    journalctl -fu hephaestus-reconcile.service

Moving an existing host​

A host with an existing Hephaestus database keeps the secrets used to write that database. Pull-based deployment does not change them. Operators write the per-stack files in /etc/hephaestus (app.env, core.env) manually here. docker/self-host/setup.sh and its guard belong to the Compose installer.

  1. Before the first reconcile, copy these settings from the previous deployment:

    SettingWhat it must be
    HEPHAESTUS_SECURITY_ENCRYPTION_KEYThe previous deployment's value. It encrypted the server's stored values other than integration credentials.
    HEPHAESTUS_SECURITY_CREDENTIAL_ENCRYPTION_KEYThe previous deployment's value, or the master key if the installation never set it separately. It encrypted integration credentials. A different key makes them unreadable without a warning.
    HEPHAESTUS_SECURITY_CREDENTIAL_ROTATION_ENABLED, HEPHAESTUS_SECURITY_CREDENTIAL_ENCRYPTION_KEY_VERSION, HEPHAESTUS_SECURITY_PRIOR_CREDENTIAL_ENCRYPTION_KEY, HEPHAESTUS_SECURITY_PRIOR_CREDENTIAL_ENCRYPTION_KEY_VERSIONThe previous deployment's values, if it rotated credential keys. Each stored credential names the key version that wrote it. A credential under the prior key remains readable only while that key is configured.
    WEBHOOK_SECRETwhat the providers were registered to sign with
    HEPHAESTUS_AUTH_STATE_COOKIE_KEYoptional. A new one only invalidates sign-ins that are in flight at the moment of the switch

    A host with its own edge keeps certificates in the proxy_letsencrypt volume. In a release worktree, the old bind-mounted acme.json is under docker/letsencrypt/. The reconciler never prunes a worktree that still holds that file.

    CAUTION: Preserve the active ACME store

    An existing acme.json in the volume is Traefik's active store. An older backup would discard certificates currently in service. The command below refuses that replacement. Inspect an existing store deliberately before replacing it.

    Before reconciling a host that used the old bind mount:

  2. Pause the timer.

  3. Stop its existing Traefik container so the ACME store cannot change during the copy.

  4. Back up acme.json first.

  5. Before the new proxy starts, copy it into the volume. Otherwise, Traefik issues the certificates again.

  6. On the Docker Engine host, use the volume's actual mountpoint, not an assumed Docker data root:

    sudo docker volume create proxy_letsencrypt
    volume_path=$(sudo docker volume inspect --format '{{ .Mountpoint }}' proxy_letsencrypt)
    if sudo test -s "$volume_path/acme.json"; then
      echo "acme.json is already in proxy_letsencrypt — inspect it before replacing it" >&2
    else
      sudo install -m 600 /path/to/previous/acme.json "$volume_path/acme.json"
    fi

  7. Keep the backup until the new proxy successfully serves the existing certificates.

  8. Then re-enable reconciliation.

For rollback to a release whose proxy still uses the bind mount:

  1. Stop Traefik.
  2. Copy the current ACME store back.

The old release tree alone does not keep that copy current. The pull-based stacks use the bundled database with fixed credentials. No database password therefore needs transfer. On the Compose installer, docker/self-host/setup.sh refuses to generate an encryption key while hephaestus_postgresql-data exists. It stops rather than guesses if Docker cannot list volumes.

A host installed before /var/lib/hephaestus/tooling existed runs the reconciler directly from its checkout. That checkout did not update. The commands below advance it manually once. Afterward, the reconciler maintains the link and units.

CAUTION: Do not change a checkout during reconciliation

A oneshot service remains activating while it runs. Stopping the timer does not stop an active tick. The commands first stop the timer and wait for an active tick to finish. Do not change the checkout underneath a tick.

  1. Run this block:

    sudo systemctl stop hephaestus-reconcile.timer
    while [ "$(systemctl show --property=ActiveState --value hephaestus-reconcile.service)" = activating ]; do sleep 5; done
    sudo git -C /var/lib/hephaestus/checkout fetch --quiet origin main
    sudo git -C /var/lib/hephaestus/checkout reset --quiet --hard origin/main
    sudo ln -sfn /var/lib/hephaestus/checkout /var/lib/hephaestus/tooling
    sudo install -m 0644 \
      /var/lib/hephaestus/checkout/deploy/reconcile/hephaestus-reconcile.service \
      /var/lib/hephaestus/checkout/deploy/reconcile/hephaestus-reconcile.timer \
      /etc/systemd/system/
    sudo systemctl daemon-reload
    sudo systemctl start hephaestus-reconcile.service
    sudo systemctl start hephaestus-reconcile.timer
    sudo readlink /var/lib/hephaestus/tooling

  2. Run the Release / Promote workflow for the release that the host already runs.

    The host cannot complete an old record that lacks the release commit from files on disk. Those files cannot distinguish the accepted tree from one left by a failed promotion. The host therefore keeps the checkout's tooling and reports hephaestus_deploy_tooling_pending 1. The next apply records the commit and adopts that release's tooling.

    After a tick adopts the tooling, the block's last line names the running release's worktree. The service starts only through the link.

  3. If the link is missing or points to a removed directory, recreate it with the ln line above.

    If a recorded release's worktree under releases/ was deleted, the reconciler rebuilds it before any other action. It uses the accepted commit, not the tag's current target. If it cannot rebuild the worktree, such as when the checkout lost that commit, the tick stops with the reason.

    HEPHAESTUS_STACKS selects the host's managed Compose projects.

  4. If another proxy supplies ingress, omit proxy and use core app.

Follow a channel with the chart​

A host can apply its channel with the Helm chart instead of Compose stacks. A release channel verifies the signed joint lock, its evidence manifest, and the chart signature. It pulls the chart by its OCI digest and checks the reported digest, metadata, and image pins. The installed verifier performs these checks before any candidate tooling runs. It does not download a schema-1 lock or apply the source chart for a release channel. A commit channel applies the source chart with the image pins from its signed channel.

A commit chart has no release version. To switch it to a release channel, pause reconciliation and apply the verified target through Database maintenance first. Then resume the timer. The next tick reads the target chart's release version.

Prepare the verified target source tree first, as Database maintenance describes. Keep writers stopped until the target migration and server succeed. Use the target source commit from its verified signed lock as <commit>:

sudo install -d -m 0750 /var/lib/hephaestus/chart
sudo node /var/lib/hephaestus/tooling/scripts/prepare-release-chart.ts \
  <tag> <commit> /var/lib/hephaestus/chart helm \
  /var/lib/hephaestus/releases/<tag>/security/release-images.json
helm upgrade --install <release> /var/lib/hephaestus/chart/hephaestus-<version>.tgz \
  --namespace <namespace> \
  --values /var/lib/hephaestus/releases/<tag>/deploy/chart/hephaestus/profiles/<profile>.yaml \
  --values /etc/hephaestus/chart.yaml \
  --values /var/lib/hephaestus/chart/<tag>.images.yaml \
  --set server.replicas=1 --set worker.replicas=0 \
  --wait=watcher --wait-for-jobs --timeout <timeout>s

Install the 0.95.1 reconciler before promoting 1.0. The reconciler units now live in deploy/reconcile/, independent of the Compose files. To update an existing 0.95.0 reconciler, stop its timer and wait for the active tick to finish, as Upgrading a host describes. Then run these commands.

The installed 0.95.0 verifier checks the 0.95.1 lock before the host runs the new tooling. This bootstrap uses the 0.95.1 lock only. The 1.0 chart path needs no schema-1 asset.

sudo install -d -m 0750 /var/lib/hephaestus/tooling-update
sudo node /var/lib/hephaestus/tooling/scripts/prepare-release-lock.ts \
  v0.95.1 /var/lib/hephaestus/tooling-update/lock.env
tooling_commit=$(sudo sed -n 's/^HEPHAESTUS_RELEASE_COMMIT=//p' /var/lib/hephaestus/tooling-update/lock.env)
sudo git -C /var/lib/hephaestus/checkout fetch --quiet origin tag v0.95.1
if [ "$(sudo git -C /var/lib/hephaestus/checkout rev-parse 'v0.95.1^{commit}')" = "$tooling_commit" ]; then
sudo git -C /var/lib/hephaestus/checkout worktree add --detach \
  /var/lib/hephaestus/tooling-update/v0.95.1 "$tooling_commit"
sudo install -m 0644 \
  /var/lib/hephaestus/tooling-update/v0.95.1/deploy/reconcile/hephaestus-reconcile.service \
  /var/lib/hephaestus/tooling-update/v0.95.1/deploy/reconcile/hephaestus-reconcile.timer \
  /etc/systemd/system/
sudo systemctl daemon-reload
sudo ln -sfn /var/lib/hephaestus/tooling-update/v0.95.1 /var/lib/hephaestus/tooling
sudo systemctl start hephaestus-reconcile.timer
else
  echo "v0.95.1 does not match the verified lock" >&2
fi

After an apply, the reconciler completes both unit copies and reloads systemd before it moves its tooling link. If that step stops, the next tick completes it before reading another channel.

Requirements for the chart​

  1. A cluster and the platform items that Kubernetes lists.

  2. The namespaces, the Secrets and the values that Kubernetes § Install describes.

  3. kubectl on PATH, with access to read Pods and Jobs in the release and sandbox namespaces.

  4. Helm 4.3.1. Install the pinned binary and verify its checksum. On amd64:

    curl -fsSLO https://get.helm.sh/helm-v4.3.1-linux-amd64.tar.gz
    echo "0403a88ae20913d11b10a2467b50087721fea7581736da6b9d3992f5357fe54b  helm-v4.3.1-linux-amd64.tar.gz" \
      | sha256sum --check
    tar -xzf helm-v4.3.1-linux-amd64.tar.gz linux-amd64/helm
    sudo install -m 755 linux-amd64/helm /usr/local/bin/helm

    On arm64, use linux-arm64 and the checksum 8cf8f66ecbc252ac8f7f06cca68aba88508ddfde3a6a34feb1678ad1694b4f88.

Settings for the chart​

Set these values in /etc/hephaestus/reconcile.env:

SettingDefaultValue
HEPHAESTUS_STACKSNonechart. It takes no other stack.
HEPHAESTUS_CHART_RELEASEhephaestusThe Helm release name.
HEPHAESTUS_CHART_NAMESPACEhephaestusThe release namespace.
HEPHAESTUS_CHART_PROFILEsmallOne or more values files in the chart's profiles/, without .yaml, separated by spaces or commas. Each profile applies over the one before it, for example "small small-16gib". The reconciler refuses a profile that the release does not have.
HEPHAESTUS_HELMhelmThe path of the Helm binary, for example /usr/local/bin/helm.
KUBECONFIGNoneThe cluster's kubeconfig. On k3s, /etc/rancher/k3s/k3s.yaml.
HEPHAESTUS_WAIT_TIMEOUT1200 on a chart hostThe Helm timeout in seconds. It must be longer than the 10 minutes that a new Pod waits for the schema.

Put your values from Kubernetes § Install in /etc/hephaestus/chart.yaml. Set its ownership to root:root and its mode to 0600. Do not set images or release in it. The reconciler writes the verified pins to /var/lib/hephaestus/chart/<target>.images.yaml. That file is the last values file, so it overrides any image in your file.

A commit apply runs this command:

helm upgrade --install <release> <commit tree>/deploy/chart/hephaestus \
  --namespace <namespace> \
  --values <commit tree>/deploy/chart/hephaestus/profiles/<first profile>.yaml \
  --values <commit tree>/deploy/chart/hephaestus/profiles/<next profile>.yaml \
  --values /etc/hephaestus/chart.yaml \
  --values /var/lib/hephaestus/chart/<commit>.images.yaml \
  --rollback-on-failure --wait=watcher --wait-for-jobs --timeout <timeout>s

A release apply uses /var/lib/hephaestus/chart/hephaestus-<version>.tgz as the chart argument. It uses that release's profiles and pinned values. The chart and image downloads have finite time limits. Helm status and history reads have a two-minute limit. The upgrade and recovery commands have deadlines derived from HEPHAESTUS_WAIT_TIMEOUT. The unit stops the whole tick after two hours.

If you increase the Helm timeout, set TimeoutStartSec above three times that timeout plus 40 minutes. Use a systemd drop-in for the override.

The revision's Job migrates the database. If the migration or a Pod fails, Helm returns the workloads to their previous revision. Helm does not roll back the database. The tick then fails, the metrics show the failure, and the applied record does not move.

An upgrade that stops before it ends, for example at a reboot or at the unit's timeout, leaves the Helm release in a pending-* status. Helm then refuses every upgrade. For an upgrade within the same major version, the next tick finds that status. It rolls back to the newest revision whose status is deployed or superseded, then upgrades. To do it by hand, read helm history <release> --namespace <namespace> and run helm rollback <release> <revision> --namespace <namespace> --wait.

A release that Helm never finished, such as a first install that stopped, has no such revision. The tick then fails and names helm uninstall <release> --namespace <namespace>. The uninstall keeps the data volumes and Secrets (Kubernetes § Uninstall). After you run it, the next tick installs the release again.

Stopped major upgrades​

For 0.95.x → 1.0, follow Database maintenance and Upgrade to 1.0. Pause the timer before phase 1. Save /var/lib/hephaestus/applied.json as /var/lib/hephaestus/applied-before-cutover.json before you change the release. Record the stopped phase-1 Helm revision from helm history <release> --namespace <namespace>.

Verify the backups, record the worker count, and use the verified source chart to stop the server and workers. Wait until their Pods and every sandbox Pod and Job are removed. Then promote the target release and restart the timer for phase 2.

The reconciler checks both its applied record and the installed chart version. For a major transition, it requires zero server and worker replicas in the installed values and no remaining writer workloads. It then applies the verified target chart with one server and zero workers. If Helm finished before the applied record was written, the next tick checks the target identity and image pins. A matching deployed target completes its record without running the migration hooks again. This also applies after you restore the worker count.

It omits --rollback-on-failure because Helm cannot restore the old schema. A pending cutover stops reconciliation and requires database recovery. It never rolls old workloads onto a migrated database.

After the server is ready, pause the timer and restore the recorded worker count with the verified target chart, as phase 4 describes. Keep the desired worker count in /etc/hephaestus/chart.yaml, then resume the timer. A new promotion of the same target can also restore that count after the cutover succeeds.

The reconciler removes no image on a chart host. The kubelet's image garbage collection removes unused images.

Recover a stopped cutover​

Before you promote 1.0, set HEPHAESTUS_WAIT_TIMEOUT above the expected migration time. Set the systemd deadline as Settings for the chart describes. A Helm timeout stops its wait. It does not stop a migration Job.

If a stopped cutover fails or remains pending:

  1. Stop the timer: sudo systemctl stop hephaestus-reconcile.timer.
  2. Wait for the active reconciler service to finish.
  3. Read helm history <release> --namespace <namespace> and record the failed revision.
  4. Inspect kubectl get job <release>-migration-<revision> --namespace <namespace>. Wait for its Complete or Failed condition before you restore data. After its 24-hour TTL, an absent Job is finished only if no migration Pod remains. If you must cancel it, delete that Job and wait for its Pods to disappear first.
  5. Restore the pre-upgrade database as If migration fails describes. Keep every database writer stopped during the restore.
  6. Run helm rollback <release> <phase-1-revision> --namespace <namespace> --wait. Use the recorded source revision with zero server and worker replicas.
  7. To retry the target, resume the timer. To return to the source release, promote its signed release with rollback enabled, then resume the timer.
  8. After the selected release succeeds, restore the worker count as phase 4 describes.

A signed rollback request does not permit an automatic major release downgrade. Before a commit rollback across a schema boundary, restore the database through the same manual procedure. If the completed target already has an applied record, restore applied-before-cutover.json to applied.json after both restores succeed. Then promote the source release and resume the timer. A commit channel must descend from the recorded release source, unless its signed request permits rollback.

Switch a host between Compose and the chart​

To move a Compose host to the chart:

  1. Disable the timer: sudo systemctl disable --now hephaestus-reconcile.timer.

  2. Install the chart at the commit that the host runs, as Kubernetes § Install describes, and move the data.

  3. Stop the Compose stacks with docker compose stop. Keep their volumes until the chart has run for one release cycle.

  4. Set HEPHAESTUS_STACKS=chart and the chart settings, and write /etc/hephaestus/chart.yaml.

  5. Run the reconciler once from a tree that has the chart stack. Each tick first adopts the tooling of the release that the host applied. A host that applied a release from before the chart stack therefore runs tooling that refuses HEPHAESTUS_STACKS=chart. Run the reconciler of a newer main commit by hand, with the settings of hephaestus-reconcile.service:

    COMMIT=<a main commit with the chart stack>
    sudo git -C /var/lib/hephaestus/checkout fetch origin +refs/heads/main:refs/remotes/origin/main
    sudo git -C /var/lib/hephaestus/checkout worktree add --detach /var/lib/hephaestus/switch "$COMMIT"
    sudo systemd-run --unit=hephaestus-reconcile-switch --wait --collect -p Type=oneshot \
      -p EnvironmentFile=/etc/hephaestus/reconcile.env \
      -p StateDirectory=hephaestus -p StateDirectoryMode=0750 -p TimeoutStartSec=30min \
      -p Environment=HELM_CACHE_HOME=/var/lib/hephaestus/helm/cache \
      -p Environment=HELM_CONFIG_HOME=/var/lib/hephaestus/helm/config \
      -p Environment=HELM_DATA_HOME=/var/lib/hephaestus/helm/data \
      -p NoNewPrivileges=yes -p ProtectSystem=strict -p ProtectHome=yes -p PrivateTmp=yes \
      -p ReadWritePaths=-/var/lib/node_exporter -p ReadWritePaths=/etc/systemd/system \
      -p "RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6" -p RestrictSUIDSGID=yes \
      -p RestrictRealtime=yes -p LockPersonality=yes \
      -p SystemCallFilter=@system-service -p SystemCallErrorNumber=EPERM \
      /usr/bin/env node /var/lib/hephaestus/switch/scripts/reconcile-deployment.ts
    sudo git -C /var/lib/hephaestus/checkout worktree remove /var/lib/hephaestus/switch

    The run verifies the channel and the lock, applies the release that the channel names with Helm, and records it. It then adopts the tooling and the unit of that release.

  6. Enable the timer: sudo systemctl enable --now hephaestus-reconcile.timer. A tick does nothing while the channel names the release that the host runs. The next promotion runs Helm.

To move the host back to Compose:

  1. Disable the timer.
  2. Stop the cluster. On k3s, run sudo systemctl stop k3s.
  3. Set HEPHAESTUS_STACKS to the Compose stacks that the host had, for example "core app".
  4. Start each stack, for example sudo docker compose --project-name app start, then enable the timer.

Data that the chart wrote after the switch is not in the Compose database.

Operating it​

You want toDo this
Move an environmentRun the Release / Promote workflow: pick the environment and release or commit
Roll backPause reconciliation, then follow Rollback. After the database matches the older release, promote that release with allow-rollback
Keep Staging on a release or commitPromote it with freeze
Let Staging follow main againPromote a commit of main with freeze cleared. The next successful build moves it on. Promoting the commit that runs applies it again
Reapply Compose configurationPromote the release or commit the host already runs
Move a following host onto the commit's database imagePromote the commit with refresh-database-image
Prevent new reconciliation runsUse the timer-disable command below
See what a host runsUse the applied-record command below
See what was promoted to an environment, and whenOpen the environment's deployment history on GitHub
Keep a release worktree while you work in itLock it. See What it keeps on disk

Timer-disable command:

sudo systemctl disable --now hephaestus-reconcile.timer

Applied-record command:

cat /var/lib/hephaestus/applied.json
CAUTION: Pause reconciliation before manual changes

An active reconciliation can conflict with manual changes. Disable the timer before editing. Confirm that the service no longer runs with:

systemctl status hephaestus-reconcile.service

A disabled timer does not stop an active reconciliation. Re-enable the timer after recovery.

A new promotion reapplies Compose configuration, not arbitrary container-file changes. The reconciler refuses a release checkout with modified tracked files.

How an apply stops the worker​

An apply stops the worker before it replaces a service that the worker depends on. After the signed lock and every rendered Compose configuration of the new release pass their checks, the reconciler pulls the images of each stack with docker compose pull --policy missing. A pull starts no container, so the running worker continues during the download.

Then the reconciler runs the database migration of the new release, while every role of the previous release still runs. A PostgreSQL that runs is left alone: the up that replaces the roles applies a changed PostgreSQL service. If the migration fails, the reconciler stops nothing, applies no stack and records no release. Then the reconciler stops the running worker.

It uses the app configuration and retained lock of the release that the host applied last. If a pull fails, the reconciler stops nothing, applies no stack and records no release. This also stops the worker when the new release removes it. It waits until the stop is complete and then applies the stacks.

A host applies a release with the reconciler of the release that it applied last. The first apply of the release that adds the migration service therefore uses a reconciler without the migration step. That apply stops the previous roles before Compose runs the migration.

Compose waits up to the stop_grace_period that the running worker was started with while the worker drains, then stops the container. A review that is still running at that time stops with the worker.

The database stays available during the stop. The app stack's normal up starts the worker of the new release, if that release has one.

There is nothing to stop on a first install, or when the applied release has no application-worker service. A host without the app stack skips the stop. When the host already applied its own request and that request is unchanged, the worker does not stop, even if another environment is promoted. If the stop fails, the reconciler applies no stack and records no release. The next tick tries again.

The reconciler also applies nothing when it cannot identify the worker that runs. The applied record must identify its source commit. The retained lock must be present and name that same commit. Verify the revision that the host runs before you let it apply another release.

The reconciler runs the tooling that the host already adopted. Thus, the apply that brings this step runs without it, and later applies include it.

What restarts the database​

An apply recreates a container whose image or Compose definition changed. PostgreSQL recreation drops all database connections until the application services stop and return. Each default-branch commit rebuilds the PostgreSQL image to include Debian security updates. A host that follows commits therefore receives a new digest on each apply. Applying each digest would restart the database several times a day. The reconciler instead keeps the current image and applies the rest of the channel.

On such a host, the database restarts in these cases:

  • The commit changes docker/postgres/ or the workflows that build the image.
  • The host applies the first commit of a later day. That day's security updates arrive within a day of the push that built them.
  • The postgres service in docker/compose.app.yaml changes.
  • The host has no usable image record. This includes a missing applied commit, missing lock, or checkout unable to resolve the two commits.
  • You promote with refresh-database-image. That takes the commit's image once. The next automatic promotion carries it forward again.

A release is applied exactly as its signed lock names it, so a release that rebuilt the image restarts the database.

What it keeps on disk​

Each applied release or commit gets a Git worktree under /var/lib/hephaestus/releases/ and a lock under /var/lib/hephaestus/release-locks/. After a successful apply, the reconciler removes unnecessary worktrees with git worktree remove. It then removes the locks of releases with no worktree. It keeps these worktrees:

  • The release just applied.
  • The previous running release, so rollback starts from an existing worktree. Promoting the running release again also keeps this previous release.
  • The worktree named by /var/lib/hephaestus/tooling.
  • A worktree locked with git worktree lock for manual work.
  • A worktree with files absent from its release, including modified, untracked, or ignored files. This includes acme.json written to docker/letsencrypt/ by an older proxy.

The journal names each worktree the reconciler removes, and says why it keeps a locked worktree or one that holds files. Pruning deletes worktrees and lock files only. It never touches a Docker volume, so the certificates in proxy_letsencrypt and the database stay as they are.

  1. To keep a worktree during use, lock it.

  2. When you finish, unlock it:

    sudo git -C /var/lib/hephaestus/checkout worktree lock --reason "incident review" \
      /var/lib/hephaestus/releases/<release>
    sudo git -C /var/lib/hephaestus/checkout worktree unlock /var/lib/hephaestus/releases/<release>

    A worktree kept because of its files remains until you remove it.

  3. Save its files first.

  4. Use:

    sudo git -C /var/lib/hephaestus/checkout worktree remove --force <worktree>

A rollback to a pruned release creates its worktree again at the commit the host accepts for it. The proxy then finds its certificates in the volume, so nothing is issued again. This is not true for a release whose proxy still used the bind mount. See Moving an existing host.

If pruning fails, the pass fails after the new release is running, recorded and its tooling adopted. The journal gives the reason. The next apply prunes everything that is still obsolete.

Systemd runs one pass at a time. This prevents another pass from using a worktree that pruning removes.

  1. To run a pass manually, use:

    sudo systemctl start hephaestus-reconcile.service

  2. Do not start the reconciler directly.

Images and free space​

After it prunes the worktrees, the reconciler removes the images that neither the release just applied nor the previous release names. It considers only the repositories that these two releases name, and only images without a tag. A release image is pulled by digest and has no tag, so a tagged image of another workload stays. Docker refuses to remove an image that a container uses, running or stopped. The journal names each image that is removed or kept.

A rollback to the previous release therefore needs no download. A rollback to an older release downloads its images again. If a step after the apply fails, the next apply removes the images.

Before it pulls, the reconciler checks the free space on the filesystems that hold Docker's data and the containerd image store. It checks only when the release needs a pull, so a rollback whose images are on the host is never refused. Below HEPHAESTUS_MIN_FREE_GIB, it first removes the images that neither the running, the previous nor the new release names. If the space is still too small, it refuses the apply, stops nothing and records nothing. The journal states the free space and the threshold, and the next tick tries again. Sizing § Disk gives the default threshold and the disk budget.

Recover a host that is already full​

A host runs the reconciler of the release it last applied. A reconciler from before the image cleanup removes no images. When such a host is full, its pull fails with no space left on device, and it cannot pull the release that brings the cleanup.

If you set HEPHAESTUS_METRICS_FILE, a file without hephaestus_deploy_free_bytes shows a reconciler from before the cleanup. Without the file, you can still follow these steps: they are safe on a newer reconciler too. Free the space once by hand:

  1. Disable the timer with sudo systemctl disable --now hephaestus-reconcile.timer.

  2. Wait for an active tick to finish:

    while [ "$(systemctl show --property=ActiveState --value hephaestus-reconcile.service)" = activating ]; do sleep 5; done

  3. In /var/lib/hephaestus/applied.json, read release and, if it is there, previous.

  4. Open /var/lib/hephaestus/release-locks/<release>.env for each of them. If a lock is missing, stop. Enable the timer again as step 11 shows. Do not guess which images to keep.

  5. For each HEPHAESTUS_IMAGE_* value in the locks, read its image ID with docker image inspect --format '{{.Id}}' <reference>. If Docker reports no such image, the host does not hold it. Skip it. These are the kept IDs. The agent image and the carried PostgreSQL image are among them.

  6. List the images on the host with docker image ls --all --digests --no-trunc. Docker Engine 29 with the overlay2 storage driver lists an image that has no tag only with --all. A release image is pulled by digest, so it has no tag.

  7. Find the images of the locks' repositories whose TAG is <none> and whose IMAGE ID is not a kept ID. Leave an image that has a tag: a release image never has one. A <none> repository is a layer, not an image of these repositories.

  8. Remove each of them with docker image rm <repository>@<digest>. Do not force the removal. Docker refuses only an image that a container uses, and the agent image has no container between reviews.

  9. Find Docker's data directory with docker info --format '{{.DockerRootDir}}'.

  10. Check the free space with df -h on that directory and on /var/lib/containerd, if it exists. Both need at least the free space that Sizing § Disk sets as the default threshold. If they do not, free more space or extend the disk, and check again.

  11. Enable the timer again with sudo systemctl enable --now hephaestus-reconcile.timer.

The next tick applies a pending release. From then on, the reconciler keeps the images itself.

Watching it​

  1. Set HEPHAESTUS_METRICS_FILE to a .prom file in an existing node_exporter textfile collector directory.

After deployment succeeds, each run writes the metrics below. A run that only adopts the applied release's tooling leaves them unchanged:

hephaestus_deploy_info{channel="staging",release="v1.2.3",channel_commit="…"} 1
hephaestus_deploy_reconcile_success 1
hephaestus_deploy_reconcile_timestamp_seconds …
hephaestus_deploy_tooling_pending 0
hephaestus_deploy_last_success_timestamp_seconds …
hephaestus_deploy_free_bytes …

Every run that writes metrics, including a failed run, reports hephaestus_deploy_free_bytes, the least free space on the filesystems that hold Docker's data. Warn when it falls below twice HEPHAESTUS_MIN_FREE_GIB, before the reconciler refuses an apply.

The group hephaestus-pull-deploy of the shipped alert rules alerts on these metrics. It covers a failing reconciler, a reconciler that finishes no run, absent metrics, and tooling that no apply recorded. hephaestus_deploy_last_success_timestamp_seconds records the last successful apply. An unchanged channel does not advance it, so its age is not a reconciliation-liveness check.

A promotion waits for the public webapp to report the requested version. This checks the webapp version, not the readiness of every service. Use host metrics and logs for the Compose result.

A promotion also records a GitHub deployment of the release or commit that it promotes. The record names who requested the promotion. Its final state tells whether the public webapp reported the version. Its time is when the promotion ran, not when a host applied it.

Host metrics remain the record of what each host runs. A skipped automatic promotion records no deployment. Release Management § Deployment history lists the recorded fields.

The promotion job summary explicitly names this boundary, including when a version check fails. It does not fetch host metrics. The textfile collector and authenticated monitoring access belong to the host, not the public application. A matching frontend version does not establish server, worker, webhook, database, NATS, or integration readiness.

  1. On each participating host, confirm a fresh successful reconciliation.
  2. Confirm the expected release/channel commit.
  3. Inspect Compose health.
  4. Inspect service logs.
  5. Do not expose management endpoints to replace operator verification with the public version check.

What it will not do​

It does not restore a database. A failed Compose apply or stopped major upgrade may leave some services updated. Schema migrations are forward-only. Older images may be incompatible with the migrated database. Promote a fix, or follow Rollback if database recovery is needed.

It does not manage anything it does not own. The reconciler touches only the Compose projects named in HEPHAESTUS_STACKS, or the one Helm release of a chart host. Containers belonging to anything else on the host are left alone.