# Deploy from a release channel A host polls a **channel** naming a release or commit, verifies its signature, and applies the verified Helm chart. Deployment needs no inbound administration connection to the host. Chart-only release These instructions describe the chart-only release. For a 0.95.x deployment, use that release's versioned instructions and signed assets. The [0.x upgrade path](/next/admin/upgrade-to-1-0.md) requires 0.95.1 and its installed reconciler tooling before the cutover. ## What the host trusts For a chart release deployment, the host verifies three signatures: | Artifact | Signed by | Answers | | ----------------------------------------------- | ---------------------- | --------------------------------------------------------------- | | The channel (`channels/.json`) | the promotion workflow | may this host move, and who said so | | The joint lock (`release-.chart.json`) | the release workflow | which source, chart digest, and image digests form this release | | The chart at its OCI digest | the release workflow | whether the chart package came from that workflow | The host rejects a new channel commit that does not descend from its last accepted channel commit. It rejects lower release versions and older or divergent commits unless `allowRollback` is set. A release-to-commit transition is ordered against the applied source commit. A major release downgrade requires database recovery, including when rollback is allowed. Each environment acts only on its own signed request: its channel file and signature bundle together. A promotion of another environment also adds a commit to `deploy-state`. If the host already applied its own unchanged request, it verifies the signature again. It records the new commit as accepted. It restarts nothing. If its own request is not applied yet, for example after a failed apply, the host tries it again. Each promotion signs again, so a promotion of the target that already runs applies again. A frozen promotion of the target that already runs is the exception: it restarts nothing, unless it also asks for **refresh-database-image**. A change to the site values applies the target again, also on a frozen channel ([Settings for the chart](#settings-for-the-chart)). A frozen promotion that does not ask for it also replaces a request for that target that the host has not applied yet. `applied.json` records the last-applied request as `channelIdentity`. A record without `channelIdentity` was written by an earlier release. For such a record, only its accepted commit counts as the applied request. The first unchanged run records the identity. If another environment was promoted first, the host applies its own request once more. ## The channel ``` { "release": "v1.2.3", "allowRollback": false, "freeze": false } ``` | Field | Effect | | --------------- | ---------------------------------------------------------------------------------------------- | | `release` | the `vX.Y.Z` release to apply | | `allowRollback` | lets the host apply an older release or commit. It does not restore the database or any volume | | `freeze` | keeps a following environment on this target: automatic promotion leaves the channel alone | Staging can follow successful builds of `main` instead of releases. Its channel names a full `commit` SHA instead of `release`, plus an `images` map of digest-pinned references and a `refreshDatabaseImage` flag (see [What restarts the database](#what-restarts-the-database)). The promotion workflow verifies first-party image provenance against that source commit before signing the channel. Upstream image pins come from that commit's inventory. Automatic promotions check the latest channel while holding the promotion lock. A frozen channel, an already-promoted commit, or an older completed build leaves the channel unchanged, including when switching from a release to a commit. An unreadable channel or failed history comparison stops promotion rather than clearing a freeze. Establish the initial channel with a manual promotion before enabling continuous staging. A host applies a frozen channel like any other: `freeze` matters only to automatic promotion. Thus, one promotion moves Staging to a release or commit and keeps it there. Only Staging follows `main`, so only Staging can be frozen. A host that runs tooling from before this rule ignores a frozen channel. On such a host, a frozen promotion to a new target fails its version check. Promote that host once without **freeze** first, to a commit that includes this rule. An automatic promotion that would advance the channel checks the most recent successful CI/CD run of a push to `main` in the same repository. If that run's commit strictly descends from the requested commit, the promotion moves the channel to that commit instead. The run's title still names the requested commit. The channel and the deployment record name the promoted one. If the lookup fails, the requested commit continues through the existing promotion checks. Manual promotions never do this lookup. ## Before you start Install `git`, `jq`, Cosign, the GitHub CLI and the Node.js version pinned in `package.json#devEngines.runtime`. Verify downloaded tools against their published checksums. Allow outbound HTTPS to the [host destinations](/next/admin/network-egress.md#host). The [chart requirements](#requirements-for-the-chart) add Helm, kubectl and cluster access. ## Install Install and verify the chart before configuring its reconciler. For an existing updater, complete the [0.95.1 bootstrap](https://github.com/hephaestus-build/Hephaestus/blob/v0.95.1/docs/admin/pull-based-deployment.mdx#follow-a-channel-with-the-chart) before this release. For a new host, authenticate the release lock before using its source commit for the tooling checkout. Stop if a verification command fails. ``` VERSION=X.Y.Z LOCK="release-v$VERSION.chart.json" gh release download "v$VERSION" --repo hephaestus-build/Hephaestus \ --pattern "$LOCK" --pattern "$LOCK.sigstore.json" cosign verify-blob --bundle "$LOCK.sigstore.json" \ --certificate-identity 'https://github.com/hephaestus-build/Hephaestus/.github/workflows/release.yml@refs/heads/main' \ --certificate-oidc-issuer https://token.actions.githubusercontent.com "$LOCK" source_commit=$(jq -er --arg release "v$VERSION" \ 'select(.schemaVersion == 2 and .release == $release) | .commit | select(test("^[a-f0-9]{40}$"))' "$LOCK") sudo install -d -m 0750 /var/lib/hephaestus sudo git clone --no-checkout https://github.com/hephaestus-build/Hephaestus.git /var/lib/hephaestus/checkout sudo git -C /var/lib/hephaestus/checkout checkout --detach "$source_commit" sudo ln -s /var/lib/hephaestus/checkout /var/lib/hephaestus/tooling sudo install -d -m 0755 /etc/hephaestus sudo install -m 0600 /var/lib/hephaestus/tooling/deploy/reconcile/reconcile.env.example /etc/hephaestus/reconcile.env sudo install -m 0600 site.yaml /etc/hephaestus/chart.yaml sudo install -m 0644 /var/lib/hephaestus/tooling/deploy/reconcile/hephaestus-reconcile.service \ /var/lib/hephaestus/tooling/deploy/reconcile/hephaestus-reconcile.timer /etc/systemd/system/ ``` Edit `/etc/hephaestus/reconcile.env` for the intended channel, Helm release, namespace and kubeconfig. Then enable the timer: ``` sudo systemctl daemon-reload sudo systemctl enable --now hephaestus-reconcile.timer ``` ## Follow a channel with the chart The host applies its channel with the [Helm chart](/next/admin/kubernetes.md). A release channel verifies the signed joint lock, its evidence manifest, and the chart signature. It pulls the chart by its OCI digest and checks the reported digest, metadata, and image pins. The installed verifier performs these checks before any candidate tooling runs. It applies the signed package for a release channel. A commit channel applies the source chart with the image pins from its signed channel. One exception applies. On a host [with tooling from before 1.0.0-rc.4](/next/admin/upgrade.md#hosts-with-reconciler-tooling-from-before-100-rc4), you authenticate the release lock by hand. A release can add, remove or rename a chart image. For such a release, a host with older tooling follows [Hosts with reconciler tooling of 1.0.0-rc.4](/next/admin/upgrade.md#hosts-with-reconciler-tooling-of-100-rc4) first. A commit chart has no release version. To switch it to a release channel, pause reconciliation and apply the verified target through Database maintenance first. Then resume the timer. The next tick reads the target chart's release version. Prepare the verified target source tree first, as [Database maintenance](/next/admin/maintenance.md#database-maintenance) describes. Keep writers stopped until the target migration and server succeed. Use the target source commit from its verified signed lock as ``: ``` sudo install -d -m 0750 /var/lib/hephaestus/chart sudo node /var/lib/hephaestus/tooling/scripts/prepare-release-chart.ts \ /var/lib/hephaestus/chart helm \ /var/lib/hephaestus/releases//security/release-images.json helm upgrade --install /var/lib/hephaestus/chart/hephaestus-.tgz \ --namespace \ --values /var/lib/hephaestus/releases//deploy/chart/hephaestus/profiles/.yaml \ --values /etc/hephaestus/chart.yaml \ --values /var/lib/hephaestus/chart/.images.yaml \ --set server.replicas=1 \ --wait=watcher --wait-for-jobs --timeout s ``` After an apply, the reconciler advances its tooling and both unit copies only to a descendant source commit. It reloads systemd before it moves the tooling link. If that step stops, the next tick completes it before reading another channel. A runtime rollback keeps newer installed tooling. Startup recovery from `applied.json` also keeps a newer verifier that an operator installed by hand. The applied record describes the runtime. The tooling link names its installed verifier. A commit channel takes its runnable image keys from the target chart's schema. It checks them against the target commit's inventory and takes their references from the signed channel. The chart ignores other scanned images in the inventory. The verifier has no fixed list of commit image names. It still refuses a missing digest or an upstream reference that differs from the inventory pin. ### Install newer reconciler tooling Use this procedure when a maintainer requires new verifier tooling before a runtime upgrade. Use the full source commit of the reviewed change after it merges into `main`. Check its verified signature on GitHub. The procedure installs that immutable source tree, the tooling link, and the service and timer files. It changes neither the applied record nor the site values. Set `TOOLING_COMMIT` to that full commit and stop the timer. Wait for an active reconciliation to finish before you continue. Do not stop a service that is applying a chart. ``` ( set -euo pipefail TOOLING_COMMIT=FULL_MERGED_SOURCE_COMMIT [[ "$TOOLING_COMMIT" =~ ^[0-9a-f]{40}$ ]] sudo systemctl stop hephaestus-reconcile.timer service_state=$(sudo systemctl show hephaestus-reconcile.service --property=ActiveState --value) service_pid=$(sudo systemctl show hephaestus-reconcile.service --property=MainPID --value) if [[ ! "$service_state" =~ ^(inactive|failed)$ || "$service_pid" != 0 ]]; then echo "Wait for the active reconciliation to finish" >&2 exit 1 fi sudo env TOOLING_COMMIT="$TOOLING_COMMIT" bash -s <<'SCRIPT' set -euo pipefail [[ "$TOOLING_COMMIT" =~ ^[0-9a-f]{40}$ ]] STATE=/var/lib/hephaestus TREE="$STATE/releases/$TOOLING_COMMIT" git -C "$STATE/checkout" fetch --quiet origin +refs/heads/main:refs/remotes/origin/main git -C "$STATE/checkout" merge-base --is-ancestor "$TOOLING_COMMIT" refs/remotes/origin/main if [ ! -e "$TREE" ]; then git -C "$STATE/checkout" worktree add --quiet --detach "$TREE" "$TOOLING_COMMIT" fi [ "$(git -C "$TREE" rev-parse HEAD)" = "$TOOLING_COMMIT" ] [ -z "$(git -C "$TREE" status --porcelain=v1 --untracked-files=no)" ] if [ "$(readlink "$STATE/tooling")" != "$TREE" ]; then readlink "$STATE/tooling" > "$STATE/tooling-before-upgrade" fi install -m 0644 "$TREE/deploy/reconcile/hephaestus-reconcile.service" \ "$TREE/deploy/reconcile/hephaestus-reconcile.timer" /etc/systemd/system/ rm -f "$STATE/tooling.next" ln -s "$TREE" "$STATE/tooling.next" mv -Tf "$STATE/tooling.next" "$STATE/tooling" systemctl daemon-reload SCRIPT service_status=0 sudo systemctl start hephaestus-reconcile.service || service_status=$? sudo journalctl -u hephaestus-reconcile.service -n 80 --no-pager if [ "$service_status" -ne 0 ]; then echo "Reconciliation failed. The timer remains stopped. Inspect the journal before resuming." >&2 exit "$service_status" fi sudo systemctl start hephaestus-reconcile.timer ) ``` If a command fails after the timer stops, reconciliation stays paused. Read the error and the service journal, fix the cause, and run the procedure again. The undo target is kept when the selected tooling is already installed. Confirm that the service applied its signed channel or reported an unchanged target. Its journal can say `Keeping reconciler tooling` when the applied runtime predates the installed verifier. That message preserves the newer verifier. Channel verification and deployment continue. To undo the tooling installation before starting the service, stop the timer and select the tree recorded in `/var/lib/hephaestus/tooling-before-upgrade`. Restore its service and timer files, move the tooling link to it, and reload systemd. After a runtime change, follow [Rollback](/next/admin/upgrade.md#rollback) instead. A channel cannot downgrade the verifier. If the verifier itself needs replacement, use this procedure with another reviewed source commit. ### Requirements for the chart 1. A cluster and the platform items that [Helm chart reference](/next/admin/helm-chart.md#supported-clusters) lists. On one k3s host, install and check them with [Install only the platform](/next/admin/k3s.md#install-only-the-platform). 2. The namespaces, the Secrets and the values that [Kubernetes § Install](/next/admin/kubernetes.md#install) describes. 3. `kubectl` on `PATH`, with access to read Pods and Jobs in the release and sandbox namespaces. 4. Helm 4.3.1. Install the pinned binary and verify its checksum. On `amd64`: ``` curl -fsSLO https://get.helm.sh/helm-v4.3.1-linux-amd64.tar.gz echo "0403a88ae20913d11b10a2467b50087721fea7581736da6b9d3992f5357fe54b helm-v4.3.1-linux-amd64.tar.gz" \ | sha256sum --check tar -xzf helm-v4.3.1-linux-amd64.tar.gz linux-amd64/helm sudo install -m 755 linux-amd64/helm /usr/local/bin/helm ``` On `arm64`, use `linux-arm64` and the checksum `8cf8f66ecbc252ac8f7f06cca68aba88508ddfde3a6a34feb1678ad1694b4f88`. ### Settings for the chart Set these values in `/etc/hephaestus/reconcile.env`: | Setting | Default | Value | | ---------------------------- | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `HEPHAESTUS_CHART_RELEASE` | `hephaestus` | The Helm release name. | | `HEPHAESTUS_CHART_NAMESPACE` | `hephaestus` | The release namespace. | | `HEPHAESTUS_CHART_PROFILE` | `small` | One or more values files in the chart's `profiles/`, without `.yaml`, separated by spaces or commas. Each profile applies over the one before it. The reconciler refuses a profile that the release does not have. | | `HEPHAESTUS_HELM` | `helm` | The path of the Helm binary, for example `/usr/local/bin/helm`. | | `KUBECONFIG` | None | The cluster's kubeconfig. On k3s, `/etc/rancher/k3s/k3s.yaml`. | | `HEPHAESTUS_WAIT_TIMEOUT` | `2700` on a chart host | The Helm timeout in seconds. It must be longer than the 10 minutes that a new Pod waits for the schema, and it must [cover the server drain](/next/admin/upgrade.md#upgrade-a-schema-compatible-release). | Put your values from [Kubernetes § Install](/next/admin/kubernetes.md#install) in `/etc/hephaestus/chart.yaml`. Set its ownership to `root:root` and its mode to `0600`. Do not set `images` or `release` in it. The reconciler writes the verified pins to `/var/lib/hephaestus/chart/.images.yaml`. That file is the last values file, so it overrides any image in your file. A change to `/etc/hephaestus/chart.yaml` or to `HEPHAESTUS_CHART_PROFILE` takes effect at the next tick. The reconciler records a digest of both in `applied.json`. When the current digest differs from the recorded digest, it applies the running target again. A change does not wait for a new promotion, and a frozen channel does not stop it. If that apply fails, the record keeps the old digest, and each tick tries again until you correct the file. Pause the timer while you edit the file, so that the tick does not read a half-written file. Also keep the timer paused while you hold replica counts by hand that the file does not carry, for example in [Database maintenance](/next/admin/maintenance.md#database-maintenance). When you resume the timer, the next tick applies the file with its own counts. A commit apply runs this command: ``` helm upgrade --install /deploy/chart/hephaestus \ --namespace \ --values /deploy/chart/hephaestus/profiles/.yaml \ --values /deploy/chart/hephaestus/profiles/.yaml \ --values /etc/hephaestus/chart.yaml \ --values /var/lib/hephaestus/chart/.images.yaml \ --rollback-on-failure --wait=watcher --wait-for-jobs --timeout s ``` A release apply uses `/var/lib/hephaestus/chart/hephaestus-.tgz` as the chart argument. It uses that release's profiles and pinned values. The chart and image downloads have finite time limits. Helm status and history reads have a two-minute limit. The upgrade and recovery commands have deadlines derived from `HEPHAESTUS_WAIT_TIMEOUT`. The unit stops the whole tick after two hours. If you increase the Helm timeout, set `TimeoutStartSec` above three times that timeout plus 40 minutes. Use a systemd drop-in for the override. The revision's Job migrates the database. If the migration or a Pod fails, Helm returns the workloads to their previous revision. Helm does not roll back the database. The tick then fails, the metrics show the failure, and the applied record does not move. An upgrade that stops before it ends, for example at a reboot or at the unit's timeout, leaves the Helm release in a `pending-*` status. Helm then refuses every upgrade. For an upgrade within the same major version, the next tick finds that status. It rolls back to the newest revision whose status is `deployed` or `superseded`, then upgrades. To do it by hand, read `helm history --namespace ` and run `helm rollback --namespace --wait`. A release that Helm never finished, such as a first install that stopped, has no such revision. The tick then fails and names `helm uninstall --namespace `. The uninstall keeps the data volumes and Secrets ([Install on a Kubernetes cluster § Undo](/next/admin/kubernetes.md#undo)). After you run it, the next tick installs the release again. ### Stopped major upgrades For 0.95.x → 1.0, read [Upgrade to 1.0](/next/admin/upgrade-to-1-0.md). Use the four [Database maintenance](/next/admin/maintenance.md#database-maintenance) phases. Prepare the target site values before phase 2, as [Values and keys to move](/next/admin/upgrade-to-1-0.md#values-and-keys-to-move) describes. Pause the timer before phase 1. Save `/var/lib/hephaestus/applied.json` as `/var/lib/hephaestus/applied-before-cutover.json` before you change the release. Record the stopped phase-1 Helm revision from `helm history --namespace `. Verify the backups, record the server count, and stop the source release as [Cut over to 1.0.0](/next/admin/upgrade-to-1-0.md#cut-over-to-100) describes. Use the verified source chart and its replica settings. Wait until every source role Pod and sandbox Pod and Job is removed. Then promote the target release. Restart the timer for phase 2 only after the channel names the target. A tick before that applies your edited file to the source release and starts the writers again. The reconciler checks both its applied record and the installed chart version. For a major transition, it requires zero server replicas in the installed values and no remaining writer workloads. The step from a release candidate to its major's first stable release, such as `1.0.0-rc.6` to `1.0.0`, is a major transition too. After that check, the reconciler applies the verified target chart with one server. If Helm finished before the applied record was written, the next tick checks the target identity and image pins. A matching deployed target completes its record without running the migration hooks again. This also applies after you restore the server count. It omits `--rollback-on-failure` because Helm cannot restore the old schema. A pending cutover stops reconciliation and requires database recovery. It never rolls old workloads onto a migrated database. After the server is ready, pause the timer and restore the recorded server count with the verified target chart, as phase 3 describes. Keep the desired server count in `/etc/hephaestus/chart.yaml`, then resume the timer. After the cutover succeeds, a change to `/etc/hephaestus/chart.yaml` or a new promotion of the same target also restores that count. The reconciler removes no image on a chart host. The kubelet's image garbage collection removes unused images. ### Recover a stopped cutover Before you promote 1.0, set `HEPHAESTUS_WAIT_TIMEOUT` above the expected migration time. Set the systemd deadline as [Settings for the chart](#settings-for-the-chart) describes. A Helm timeout stops its wait. It does not stop a migration Job. If a stopped cutover fails or remains pending: 1. Stop the timer: `sudo systemctl stop hephaestus-reconcile.timer`. 2. Wait for the active reconciler service to finish. 3. Read `helm history --namespace ` and record the failed revision. 4. Inspect `kubectl get job -migration- --namespace `. Wait for its Complete or Failed condition before you restore data. After its 24-hour TTL, an absent Job is finished only if no migration Pod remains. If you must cancel it, delete that Job and wait for its Pods to disappear first. 5. Scale every role Deployment to zero. Wait for their Pods and sandboxes to stop. Restore the pre-upgrade database with steps 1 to 14 of [If migration fails](/next/admin/upgrade.md#if-migration-fails). Defer its application startup and checks in steps 9 to 12 until the selected release succeeds in step 7 below. 6. Run `helm rollback --namespace --wait`. Use the recorded source revision with zero replicas of every role. 7. To retry the target, resume the timer. To return to the source release, first restore `applied-before-cutover.json` to `applied.json` if the completed target already has an applied record. Then promote the signed source release with rollback enabled and resume the timer. 8. Complete the deferred startup and checks under pre-boot isolation. Then restore the server count as phase 3 describes. A signed rollback request does not permit an automatic major release downgrade. A commit channel must descend from the recorded release source. For a recorded pre-1.0 release tag, a signed request can permit an older commit. For a recorded 1.0 or later release tag, use the manual recovery steps above first. An applied commit record has no version. This version boundary does not apply to it. ## Operating it | You want to | Do this | | ------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Move an environment | Run the **Release / Promote** workflow: pick the environment and release or commit | | Roll back | Pause reconciliation, then follow [Rollback](/next/admin/upgrade.md#rollback). After the database matches the older release, promote that release with **allow-rollback** | | Keep Staging on a release or commit | Promote it with **freeze** | | Let Staging follow `main` again | Promote a commit of `main` with **freeze** cleared. The next successful build moves it on. Promoting the commit that runs applies it again | | Reapply chart configuration | Edit `/etc/hephaestus/chart.yaml`: the next tick applies it. To apply an unchanged file again, promote the release or commit the host already runs | | Move a following host onto the commit's database image | Promote the commit with **refresh-database-image** | | Prevent new reconciliation runs | Use the timer-disable command below | | See what a host runs | Use the applied-record command below | | See what was promoted to an environment, and when | Open the environment's deployment history on GitHub | | Keep a release worktree while you work in it | Lock it. See [What it keeps on disk](#what-it-keeps-on-disk) | Timer-disable command: ``` sudo systemctl disable --now hephaestus-reconcile.timer ``` Applied-record command: ``` cat /var/lib/hephaestus/applied.json ``` CAUTION: Pause reconciliation before manual changes An active reconciliation can conflict with manual changes. Disable the timer before editing. Confirm that the service no longer runs with: ``` systemctl status hephaestus-reconcile.service ``` A disabled timer does not stop an active reconciliation. Re-enable the timer after recovery. A new promotion reapplies the chart configuration. The reconciler refuses a release checkout with modified tracked files. ### What restarts the database An apply replaces a PostgreSQL Pod when its image or StatefulSet configuration changes. PostgreSQL recreation drops all database connections until the application services stop and return. Each default-branch commit rebuilds the PostgreSQL image to include Debian security updates. A host that follows commits therefore receives a new digest on each apply. Applying each digest would restart the database several times a day. The reconciler instead keeps the current image and applies the rest of the channel. On such a host, the database restarts in these cases: * The commit changes `docker/postgres/` or the workflows that build the image. * The host applies the first commit of a later day. That day's security updates arrive within a day of the push that built them. * The host has no usable image record. This includes a missing applied commit, missing lock, or checkout unable to resolve the two commits. * You promote with **refresh-database-image**. That takes the commit's image once. The next automatic promotion carries it forward again. A release is applied exactly as its signed lock names it, so a release that rebuilt the image restarts the database. ### What it keeps on disk Each applied release or commit gets a Git worktree under `/var/lib/hephaestus/releases/` and a lock under `/var/lib/hephaestus/release-locks/`. After a successful apply, the reconciler removes unnecessary worktrees with `git worktree remove`. It then removes the locks of releases with no worktree. It keeps these worktrees: * The release just applied. * The previous running release, so rollback starts from an existing worktree. Promoting the running release again also keeps this previous release. * The worktree named by `/var/lib/hephaestus/tooling`. * A worktree locked with `git worktree lock` for manual work. * A worktree with files absent from its release, including modified, untracked, or ignored files. The disposable kubectl discovery cache does not keep a worktree. The reconciler sets `KUBECACHEDIR` to `/var/lib/hephaestus/kubernetes-cache` for kubectl and Helm. Before it checks for operator files, pruning removes untracked files from an old tree's `.kube/cache` directory. This also removes the cache from a tree that other operator files keep. The journal names each worktree the reconciler removes, and says why it keeps a locked worktree or one that holds files. Pruning deletes worktrees and lock files. It does not change persistent volumes or cluster Secrets. 1. To keep a worktree during use, lock it. 2. When you finish, unlock it: ``` sudo git -C /var/lib/hephaestus/checkout worktree lock --reason "incident review" \ /var/lib/hephaestus/releases/ sudo git -C /var/lib/hephaestus/checkout worktree unlock /var/lib/hephaestus/releases/ ``` A worktree kept because of its files remains until you remove it. 3. Save its files first. 4. Use: ``` sudo git -C /var/lib/hephaestus/checkout worktree remove --force ``` A rollback to a pruned release creates its worktree again at the commit the host accepts for it. If pruning fails, the pass fails after the new release is running, recorded and its tooling adopted. The journal gives the reason. The next apply prunes everything that is still obsolete. Systemd runs one pass at a time. This prevents another pass from using a worktree that pruning removes. 1. To run a pass manually, use: ``` sudo systemctl start hephaestus-reconcile.service ``` 2. Do not start the reconciler directly. ## Watching it 1. Set `HEPHAESTUS_METRICS_FILE` to a `.prom` file in an existing node_exporter textfile collector directory. After deployment succeeds, each run writes the metrics below. A run that only adopts the applied release's tooling leaves them unchanged: ``` hephaestus_deploy_info{channel="staging",release="v1.2.3",channel_commit="…"} 1 hephaestus_deploy_reconcile_success 1 hephaestus_deploy_reconcile_timestamp_seconds … hephaestus_deploy_tooling_pending 0 hephaestus_deploy_last_success_timestamp_seconds … ``` The group `hephaestus-pull-deploy` of the shipped [alert rules](/next/admin/alert-rules.md) alerts on these metrics. It covers a failing reconciler, a reconciler that finishes no run, absent metrics, and tooling that no apply recorded. `hephaestus_deploy_last_success_timestamp_seconds` records the last successful **apply**. An unchanged channel does not advance it, so its age is not a reconciliation-liveness check. A promotion waits for the public webapp to report the requested version. This checks the webapp version. Use cluster readiness, reconciliation metrics and logs to verify the workloads. A promotion also records a GitHub deployment of the release or commit that it promotes. The record names who requested the promotion. Its final state tells whether the public webapp reported the version. Its time is when the promotion ran, not when a host applied it. Host metrics remain the record of what each host runs. A skipped automatic promotion records no deployment. [Release Management § Deployment history](/next/contributor/release-management.md#deployment-history) lists the recorded fields. The promotion job summary explicitly names this boundary, including when a version check fails. It does not fetch host metrics. The textfile collector and authenticated monitoring access belong to the host, not the public application. A matching frontend version does not establish server, database, or integration readiness. ## Check that it worked Do these steps on **each** participating host. 1. Read the metrics file that `HEPHAESTUS_METRICS_FILE` names. Confirm a fresh successful reconciliation: `hephaestus_deploy_reconcile_success` is `1`. 2. Confirm that `hephaestus_deploy_info` names the expected release and channel commit. 3. Inspect Pod and Job readiness. 4. Inspect service logs. Do not expose management endpoints to replace this check with the public version check. ## What it will not do **It does not restore a database.** A failed or stopped major upgrade can leave some workloads updated. Schema migrations are forward-only. Older images may be incompatible with the migrated database. Promote a fix, or follow [Rollback](/next/admin/upgrade.md#rollback) if database recovery is needed. **It manages the configured Helm release.** Other workloads remain under their own owners.