Deploy from a release channel

A host polls a channel naming a release or commit, verifies its signature, and applies the verified Helm chart. Deployment needs no inbound administration connection to the host.

Chart-only release

These instructions describe the chart-only release. For a 0.95.x deployment, use that release's versioned instructions and signed assets. The 0.x upgrade path requires 0.95.1 and its installed reconciler tooling before the cutover.

What the host trusts

For a chart release deployment, the host verifies three signatures:

Artifact	Signed by	Answers
The channel (channels/<env>.json)	the promotion workflow	may this host move, and who said so
The joint lock (release-<version>.chart.json)	the release workflow	which source, chart digest, and image digests form this release
The chart at its OCI digest	the release workflow	whether the chart package came from that workflow

The host rejects a new channel commit that does not descend from its last accepted channel commit. It rejects lower release versions and older or divergent commits unless allowRollback is set. A release-to-commit transition is ordered against the applied source commit. A major release downgrade requires database recovery, including when rollback is allowed.

Each environment acts only on its own signed request: its channel file and signature bundle together. A promotion of another environment also adds a commit to deploy-state. If the host already applied its own unchanged request, it verifies the signature again. It records the new commit as accepted. It restarts nothing.

If its own request is not applied yet, for example after a failed apply, the host tries it again. Each promotion signs again, so a promotion of the target that already runs applies again. A frozen promotion of the target that already runs is the exception: it restarts nothing, unless it also asks for refresh-database-image. A change to the site values applies the target again, also on a frozen channel (Settings for the chart). A frozen promotion that does not ask for it also replaces a request for that target that the host has not applied yet.

applied.json records the last-applied request as channelIdentity. A record without channelIdentity was written by an earlier release. For such a record, only its accepted commit counts as the applied request. The first unchanged run records the identity. If another environment was promoted first, the host applies its own request once more.

The channel

{ "release": "v1.2.3", "allowRollback": false, "freeze": false }

Field	Effect
release	the vX.Y.Z release to apply
allowRollback	lets the host apply an older release or commit. It does not restore the database or any volume
freeze	keeps a following environment on this target: automatic promotion leaves the channel alone

Staging can follow successful builds of main instead of releases. Its channel names a full commit SHA instead of release, plus an images map of digest-pinned references and a refreshDatabaseImage flag (see What restarts the database). The promotion workflow verifies first-party image provenance against that source commit before signing the channel. Upstream image pins come from that commit's inventory.

Automatic promotions check the latest channel while holding the promotion lock. A frozen channel, an already-promoted commit, or an older completed build leaves the channel unchanged, including when switching from a release to a commit. An unreadable channel or failed history comparison stops promotion rather than clearing a freeze. Establish the initial channel with a manual promotion before enabling continuous staging.

A host applies a frozen channel like any other: freeze matters only to automatic promotion. Thus, one promotion moves Staging to a release or commit and keeps it there. Only Staging follows main, so only Staging can be frozen. A host that runs tooling from before this rule ignores a frozen channel. On such a host, a frozen promotion to a new target fails its version check. Promote that host once without freeze first, to a commit that includes this rule.

An automatic promotion that would advance the channel checks the most recent successful CI/CD run of a push to main in the same repository. If that run's commit strictly descends from the requested commit, the promotion moves the channel to that commit instead. The run's title still names the requested commit. The channel and the deployment record name the promoted one. If the lookup fails, the requested commit continues through the existing promotion checks. Manual promotions never do this lookup.

Before you start

Install git, jq, Cosign, the GitHub CLI and the Node.js version pinned in package.json#devEngines.runtime. Verify downloaded tools against their published checksums. Allow outbound HTTPS to the host destinations. The chart requirements add Helm, kubectl and cluster access.

Install

Install and verify the chart before configuring its reconciler. For an existing updater, complete the 0.95.1 bootstrap before this release.

For a new host, authenticate the release lock before using its source commit for the tooling checkout. Stop if a verification command fails.

VERSION=X.Y.Z
LOCK="release-v$VERSION.chart.json"
gh release download "v$VERSION" --repo hephaestus-build/Hephaestus \
  --pattern "$LOCK" --pattern "$LOCK.sigstore.json"
cosign verify-blob --bundle "$LOCK.sigstore.json" \
  --certificate-identity 'https://github.com/hephaestus-build/Hephaestus/.github/workflows/release.yml@refs/heads/main' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com "$LOCK"
source_commit=$(jq -er --arg release "v$VERSION" \
  'select(.schemaVersion == 2 and .release == $release) | .commit | select(test("^[a-f0-9]{40}$"))' "$LOCK")
sudo install -d -m 0750 /var/lib/hephaestus
sudo git clone --no-checkout https://github.com/hephaestus-build/Hephaestus.git /var/lib/hephaestus/checkout
sudo git -C /var/lib/hephaestus/checkout checkout --detach "$source_commit"
sudo ln -s /var/lib/hephaestus/checkout /var/lib/hephaestus/tooling
sudo install -d -m 0755 /etc/hephaestus
sudo install -m 0600 /var/lib/hephaestus/tooling/deploy/reconcile/reconcile.env.example /etc/hephaestus/reconcile.env
sudo install -m 0600 site.yaml /etc/hephaestus/chart.yaml
sudo install -m 0644 /var/lib/hephaestus/tooling/deploy/reconcile/hephaestus-reconcile.service \
  /var/lib/hephaestus/tooling/deploy/reconcile/hephaestus-reconcile.timer /etc/systemd/system/

Edit /etc/hephaestus/reconcile.env for the intended channel, Helm release, namespace and kubeconfig. Then enable the timer:

sudo systemctl daemon-reload
sudo systemctl enable --now hephaestus-reconcile.timer

Follow a channel with the chart

The host applies its channel with the Helm chart. A release channel verifies the signed joint lock, its evidence manifest, and the chart signature. It pulls the chart by its OCI digest and checks the reported digest, metadata, and image pins. The installed verifier performs these checks before any candidate tooling runs. It applies the signed package for a release channel. A commit channel applies the source chart with the image pins from its signed channel.

One exception applies. On a host with tooling from before 1.0.0-rc.4, you authenticate the release lock by hand. A release can add, remove or rename a chart image. For such a release, a host with older tooling follows Hosts with reconciler tooling of 1.0.0-rc.4 first.

A commit chart has no release version. To switch it to a release channel, pause reconciliation and apply the verified target through Database maintenance first. Then resume the timer. The next tick reads the target chart's release version.

Prepare the verified target source tree first, as Database maintenance describes. Keep writers stopped until the target migration and server succeed. Use the target source commit from its verified signed lock as <commit>:

sudo install -d -m 0750 /var/lib/hephaestus/chart
sudo node /var/lib/hephaestus/tooling/scripts/prepare-release-chart.ts \
  <tag> <commit> /var/lib/hephaestus/chart helm \
  /var/lib/hephaestus/releases/<tag>/security/release-images.json
helm upgrade --install <release> /var/lib/hephaestus/chart/hephaestus-<version>.tgz \
  --namespace <namespace> \
  --values /var/lib/hephaestus/releases/<tag>/deploy/chart/hephaestus/profiles/<profile>.yaml \
  --values /etc/hephaestus/chart.yaml \
  --values /var/lib/hephaestus/chart/<tag>.images.yaml \
  --set server.replicas=1 \
  --wait=watcher --wait-for-jobs --timeout <timeout>s

After an apply, the reconciler advances its tooling and both unit copies only to a descendant source commit. It reloads systemd before it moves the tooling link. If that step stops, the next tick completes it before reading another channel. A runtime rollback keeps newer installed tooling. Startup recovery from applied.json also keeps a newer verifier that an operator installed by hand.

The applied record describes the runtime. The tooling link names its installed verifier.

A commit channel takes its runnable image keys from the target chart's schema. It checks them against the target commit's inventory and takes their references from the signed channel. The chart ignores other scanned images in the inventory. The verifier has no fixed list of commit image names. It still refuses a missing digest or an upstream reference that differs from the inventory pin.

Install newer reconciler tooling

Use this procedure when a maintainer requires new verifier tooling before a runtime upgrade. Use the full source commit of the reviewed change after it merges into main. Check its verified signature on GitHub. The procedure installs that immutable source tree, the tooling link, and the service and timer files. It changes neither the applied record nor the site values.

Set TOOLING_COMMIT to that full commit and stop the timer. Wait for an active reconciliation to finish before you continue. Do not stop a service that is applying a chart.

(
set -euo pipefail
TOOLING_COMMIT=FULL_MERGED_SOURCE_COMMIT
[[ "$TOOLING_COMMIT" =~ ^[0-9a-f]{40}$ ]]
sudo systemctl stop hephaestus-reconcile.timer
service_state=$(sudo systemctl show hephaestus-reconcile.service --property=ActiveState --value)
service_pid=$(sudo systemctl show hephaestus-reconcile.service --property=MainPID --value)
if [[ ! "$service_state" =~ ^(inactive|failed)$ || "$service_pid" != 0 ]]; then
  echo "Wait for the active reconciliation to finish" >&2
  exit 1
fi
sudo env TOOLING_COMMIT="$TOOLING_COMMIT" bash -s <<'SCRIPT'
set -euo pipefail
[[ "$TOOLING_COMMIT" =~ ^[0-9a-f]{40}$ ]]
STATE=/var/lib/hephaestus
TREE="$STATE/releases/$TOOLING_COMMIT"
git -C "$STATE/checkout" fetch --quiet origin +refs/heads/main:refs/remotes/origin/main
git -C "$STATE/checkout" merge-base --is-ancestor "$TOOLING_COMMIT" refs/remotes/origin/main
if [ ! -e "$TREE" ]; then
  git -C "$STATE/checkout" worktree add --quiet --detach "$TREE" "$TOOLING_COMMIT"
fi
[ "$(git -C "$TREE" rev-parse HEAD)" = "$TOOLING_COMMIT" ]
[ -z "$(git -C "$TREE" status --porcelain=v1 --untracked-files=no)" ]
if [ "$(readlink "$STATE/tooling")" != "$TREE" ]; then
  readlink "$STATE/tooling" > "$STATE/tooling-before-upgrade"
fi
install -m 0644 "$TREE/deploy/reconcile/hephaestus-reconcile.service" \
  "$TREE/deploy/reconcile/hephaestus-reconcile.timer" /etc/systemd/system/
rm -f "$STATE/tooling.next"
ln -s "$TREE" "$STATE/tooling.next"
mv -Tf "$STATE/tooling.next" "$STATE/tooling"
systemctl daemon-reload
SCRIPT
service_status=0
sudo systemctl start hephaestus-reconcile.service || service_status=$?
sudo journalctl -u hephaestus-reconcile.service -n 80 --no-pager
if [ "$service_status" -ne 0 ]; then
  echo "Reconciliation failed. The timer remains stopped. Inspect the journal before resuming." >&2
  exit "$service_status"
fi
sudo systemctl start hephaestus-reconcile.timer
)

If a command fails after the timer stops, reconciliation stays paused. Read the error and the service journal, fix the cause, and run the procedure again. The undo target is kept when the selected tooling is already installed.

Confirm that the service applied its signed channel or reported an unchanged target. Its journal can say Keeping reconciler tooling when the applied runtime predates the installed verifier. That message preserves the newer verifier. Channel verification and deployment continue.

To undo the tooling installation before starting the service, stop the timer and select the tree recorded in /var/lib/hephaestus/tooling-before-upgrade. Restore its service and timer files, move the tooling link to it, and reload systemd. After a runtime change, follow Rollback instead. A channel cannot downgrade the verifier. If the verifier itself needs replacement, use this procedure with another reviewed source commit.

Requirements for the chart

1. A cluster and the platform items that Helm chart reference lists. On one k3s host, install and check them with Install only the platform.

2. The namespaces, the Secrets and the values that Kubernetes § Install describes.

3. kubectl on PATH, with access to read Pods and Jobs in the release and sandbox namespaces.

4. Helm 4.3.1. Install the pinned binary and verify its checksum. On amd64:

   curl -fsSLO https://get.helm.sh/helm-v4.3.1-linux-amd64.tar.gz
   echo "0403a88ae20913d11b10a2467b50087721fea7581736da6b9d3992f5357fe54b  helm-v4.3.1-linux-amd64.tar.gz" \
     | sha256sum --check
   tar -xzf helm-v4.3.1-linux-amd64.tar.gz linux-amd64/helm
   sudo install -m 755 linux-amd64/helm /usr/local/bin/helm

   On arm64, use linux-arm64 and the checksum 8cf8f66ecbc252ac8f7f06cca68aba88508ddfde3a6a34feb1678ad1694b4f88.

Settings for the chart

Set these values in /etc/hephaestus/reconcile.env:

Setting	Default	Value
HEPHAESTUS_CHART_RELEASE	hephaestus	The Helm release name.
HEPHAESTUS_CHART_NAMESPACE	hephaestus	The release namespace.
HEPHAESTUS_CHART_PROFILE	small	One or more values files in the chart's profiles/, without .yaml, separated by spaces or commas. Each profile applies over the one before it. The reconciler refuses a profile that the release does not have.
HEPHAESTUS_HELM	helm	The path of the Helm binary, for example /usr/local/bin/helm.
KUBECONFIG	None	The cluster's kubeconfig. On k3s, /etc/rancher/k3s/k3s.yaml.
HEPHAESTUS_WAIT_TIMEOUT	2700 on a chart host	The Helm timeout in seconds. It must be longer than the 10 minutes that a new Pod waits for the schema, and it must cover the server drain.

Put your values from Kubernetes § Install in /etc/hephaestus/chart.yaml. Set its ownership to root:root and its mode to 0600. Do not set images or release in it. The reconciler writes the verified pins to /var/lib/hephaestus/chart/<target>.images.yaml. That file is the last values file, so it overrides any image in your file.

A change to /etc/hephaestus/chart.yaml or to HEPHAESTUS_CHART_PROFILE takes effect at the next tick. The reconciler records a digest of both in applied.json. When the current digest differs from the recorded digest, it applies the running target again. A change does not wait for a new promotion, and a frozen channel does not stop it. If that apply fails, the record keeps the old digest, and each tick tries again until you correct the file.

Pause the timer while you edit the file, so that the tick does not read a half-written file. Also keep the timer paused while you hold replica counts by hand that the file does not carry, for example in Database maintenance. When you resume the timer, the next tick applies the file with its own counts.

A commit apply runs this command:

helm upgrade --install <release> <commit tree>/deploy/chart/hephaestus \
  --namespace <namespace> \
  --values <commit tree>/deploy/chart/hephaestus/profiles/<first profile>.yaml \
  --values <commit tree>/deploy/chart/hephaestus/profiles/<next profile>.yaml \
  --values /etc/hephaestus/chart.yaml \
  --values /var/lib/hephaestus/chart/<commit>.images.yaml \
  --rollback-on-failure --wait=watcher --wait-for-jobs --timeout <timeout>s

A release apply uses /var/lib/hephaestus/chart/hephaestus-<version>.tgz as the chart argument. It uses that release's profiles and pinned values. The chart and image downloads have finite time limits. Helm status and history reads have a two-minute limit. The upgrade and recovery commands have deadlines derived from HEPHAESTUS_WAIT_TIMEOUT. The unit stops the whole tick after two hours.

If you increase the Helm timeout, set TimeoutStartSec above three times that timeout plus 40 minutes. Use a systemd drop-in for the override.

The revision's Job migrates the database. If the migration or a Pod fails, Helm returns the workloads to their previous revision. Helm does not roll back the database. The tick then fails, the metrics show the failure, and the applied record does not move.

An upgrade that stops before it ends, for example at a reboot or at the unit's timeout, leaves the Helm release in a pending-* status. Helm then refuses every upgrade. For an upgrade within the same major version, the next tick finds that status. It rolls back to the newest revision whose status is deployed or superseded, then upgrades. To do it by hand, read helm history <release> --namespace <namespace> and run helm rollback <release> <revision> --namespace <namespace> --wait.

A release that Helm never finished, such as a first install that stopped, has no such revision. The tick then fails and names helm uninstall <release> --namespace <namespace>. The uninstall keeps the data volumes and Secrets (Install on a Kubernetes cluster § Undo). After you run it, the next tick installs the release again.

Stopped major upgrades

For 0.95.x → 1.0, read Upgrade to 1.0. Use the four Database maintenance phases. Prepare the target site values before phase 2, as Values and keys to move describes. Pause the timer before phase 1. Save /var/lib/hephaestus/applied.json as /var/lib/hephaestus/applied-before-cutover.json before you change the release. Record the stopped phase-1 Helm revision from helm history <release> --namespace <namespace>.

Verify the backups, record the server count, and stop the source release as Cut over to 1.0.0 describes. Use the verified source chart and its replica settings. Wait until every source role Pod and sandbox Pod and Job is removed.

Then promote the target release. Restart the timer for phase 2 only after the channel names the target. A tick before that applies your edited file to the source release and starts the writers again.

The reconciler checks both its applied record and the installed chart version. For a major transition, it requires zero server replicas in the installed values and no remaining writer workloads. The step from a release candidate to its major's first stable release, such as 1.0.0-rc.6 to 1.0.0, is a major transition too. After that check, the reconciler applies the verified target chart with one server.

If Helm finished before the applied record was written, the next tick checks the target identity and image pins. A matching deployed target completes its record without running the migration hooks again. This also applies after you restore the server count.

It omits --rollback-on-failure because Helm cannot restore the old schema. A pending cutover stops reconciliation and requires database recovery. It never rolls old workloads onto a migrated database.

After the server is ready, pause the timer and restore the recorded server count with the verified target chart, as phase 3 describes. Keep the desired server count in /etc/hephaestus/chart.yaml, then resume the timer. After the cutover succeeds, a change to /etc/hephaestus/chart.yaml or a new promotion of the same target also restores that count.

The reconciler removes no image on a chart host. The kubelet's image garbage collection removes unused images.

Recover a stopped cutover

Before you promote 1.0, set HEPHAESTUS_WAIT_TIMEOUT above the expected migration time. Set the systemd deadline as Settings for the chart describes. A Helm timeout stops its wait. It does not stop a migration Job.

If a stopped cutover fails or remains pending:

1. Stop the timer: sudo systemctl stop hephaestus-reconcile.timer.
2. Wait for the active reconciler service to finish.
3. Read helm history <release> --namespace <namespace> and record the failed revision.
4. Inspect kubectl get job <release>-migration-<revision> --namespace <namespace>. Wait for its Complete or Failed condition before you restore data. After its 24-hour TTL, an absent Job is finished only if no migration Pod remains. If you must cancel it, delete that Job and wait for its Pods to disappear first.
5. Scale every role Deployment to zero. Wait for their Pods and sandboxes to stop. Restore the pre-upgrade database with steps 1 to 14 of If migration fails. Defer its application startup and checks in steps 9 to 12 until the selected release succeeds in step 7 below.
6. Run helm rollback <release> <phase-1-revision> --namespace <namespace> --wait. Use the recorded source revision with zero replicas of every role.
7. To retry the target, resume the timer. To return to the source release, first restore applied-before-cutover.json to applied.json if the completed target already has an applied record. Then promote the signed source release with rollback enabled and resume the timer.
8. Complete the deferred startup and checks under pre-boot isolation. Then restore the server count as phase 3 describes.

A signed rollback request does not permit an automatic major release downgrade. A commit channel must descend from the recorded release source. For a recorded pre-1.0 release tag, a signed request can permit an older commit. For a recorded 1.0 or later release tag, use the manual recovery steps above first. An applied commit record has no version. This version boundary does not apply to it.

Operating it

You want to	Do this
Move an environment	Run the Release / Promote workflow: pick the environment and release or commit
Roll back	Pause reconciliation, then follow Rollback. After the database matches the older release, promote that release with allow-rollback
Keep Staging on a release or commit	Promote it with freeze
Let Staging follow main again	Promote a commit of main with freeze cleared. The next successful build moves it on. Promoting the commit that runs applies it again
Reapply chart configuration	Edit /etc/hephaestus/chart.yaml: the next tick applies it. To apply an unchanged file again, promote the release or commit the host already runs
Move a following host onto the commit's database image	Promote the commit with refresh-database-image
Prevent new reconciliation runs	Use the timer-disable command below
See what a host runs	Use the applied-record command below
See what was promoted to an environment, and when	Open the environment's deployment history on GitHub
Keep a release worktree while you work in it	Lock it. See What it keeps on disk

Timer-disable command:

sudo systemctl disable --now hephaestus-reconcile.timer

Applied-record command:

cat /var/lib/hephaestus/applied.json

CAUTION: Pause reconciliation before manual changes

An active reconciliation can conflict with manual changes. Disable the timer before editing. Confirm that the service no longer runs with:

systemctl status hephaestus-reconcile.service

A disabled timer does not stop an active reconciliation. Re-enable the timer after recovery.

A new promotion reapplies the chart configuration. The reconciler refuses a release checkout with modified tracked files.

What restarts the database

An apply replaces a PostgreSQL Pod when its image or StatefulSet configuration changes. PostgreSQL recreation drops all database connections until the application services stop and return. Each default-branch commit rebuilds the PostgreSQL image to include Debian security updates. A host that follows commits therefore receives a new digest on each apply. Applying each digest would restart the database several times a day. The reconciler instead keeps the current image and applies the rest of the channel.

On such a host, the database restarts in these cases:

• The commit changes docker/postgres/ or the workflows that build the image.
• The host applies the first commit of a later day. That day's security updates arrive within a day of the push that built them.
• The host has no usable image record. This includes a missing applied commit, missing lock, or checkout unable to resolve the two commits.
• You promote with refresh-database-image. That takes the commit's image once. The next automatic promotion carries it forward again.

A release is applied exactly as its signed lock names it, so a release that rebuilt the image restarts the database.

What it keeps on disk

Each applied release or commit gets a Git worktree under /var/lib/hephaestus/releases/ and a lock under /var/lib/hephaestus/release-locks/. After a successful apply, the reconciler removes unnecessary worktrees with git worktree remove. It then removes the locks of releases with no worktree. It keeps these worktrees:

• The release just applied.
• The previous running release, so rollback starts from an existing worktree. Promoting the running release again also keeps this previous release.
• The worktree named by /var/lib/hephaestus/tooling.
• A worktree locked with git worktree lock for manual work.
• A worktree with files absent from its release, including modified, untracked, or ignored files.

The disposable kubectl discovery cache does not keep a worktree. The reconciler sets KUBECACHEDIR to /var/lib/hephaestus/kubernetes-cache for kubectl and Helm. Before it checks for operator files, pruning removes untracked files from an old tree's .kube/cache directory. This also removes the cache from a tree that other operator files keep.

The journal names each worktree the reconciler removes, and says why it keeps a locked worktree or one that holds files. Pruning deletes worktrees and lock files. It does not change persistent volumes or cluster Secrets.

1. To keep a worktree during use, lock it.

2. When you finish, unlock it:

   sudo git -C /var/lib/hephaestus/checkout worktree lock --reason "incident review" \
     /var/lib/hephaestus/releases/<release>
   sudo git -C /var/lib/hephaestus/checkout worktree unlock /var/lib/hephaestus/releases/<release>

   A worktree kept because of its files remains until you remove it.

3. Save its files first.

4. Use:

   sudo git -C /var/lib/hephaestus/checkout worktree remove --force <worktree>

A rollback to a pruned release creates its worktree again at the commit the host accepts for it.

If pruning fails, the pass fails after the new release is running, recorded and its tooling adopted. The journal gives the reason. The next apply prunes everything that is still obsolete.

Systemd runs one pass at a time. This prevents another pass from using a worktree that pruning removes.

1. To run a pass manually, use:

   sudo systemctl start hephaestus-reconcile.service

2. Do not start the reconciler directly.

Watching it

1. Set HEPHAESTUS_METRICS_FILE to a .prom file in an existing node_exporter textfile collector directory.

After deployment succeeds, each run writes the metrics below. A run that only adopts the applied release's tooling leaves them unchanged:

hephaestus_deploy_info{channel="staging",release="v1.2.3",channel_commit="…"} 1
hephaestus_deploy_reconcile_success 1
hephaestus_deploy_reconcile_timestamp_seconds …
hephaestus_deploy_tooling_pending 0
hephaestus_deploy_last_success_timestamp_seconds …

The group hephaestus-pull-deploy of the shipped alert rules alerts on these metrics. It covers a failing reconciler, a reconciler that finishes no run, absent metrics, and tooling that no apply recorded. hephaestus_deploy_last_success_timestamp_seconds records the last successful apply. An unchanged channel does not advance it, so its age is not a reconciliation-liveness check.

A promotion waits for the public webapp to report the requested version. This checks the webapp version. Use cluster readiness, reconciliation metrics and logs to verify the workloads.

A promotion also records a GitHub deployment of the release or commit that it promotes. The record names who requested the promotion. Its final state tells whether the public webapp reported the version. Its time is when the promotion ran, not when a host applied it.

Host metrics remain the record of what each host runs. A skipped automatic promotion records no deployment. Release Management § Deployment history lists the recorded fields.

The promotion job summary explicitly names this boundary, including when a version check fails. It does not fetch host metrics. The textfile collector and authenticated monitoring access belong to the host, not the public application. A matching frontend version does not establish server, database, or integration readiness.

Check that it worked

Do these steps on each participating host.

1. Read the metrics file that HEPHAESTUS_METRICS_FILE names. Confirm a fresh successful reconciliation: hephaestus_deploy_reconcile_success is 1.
2. Confirm that hephaestus_deploy_info names the expected release and channel commit.
3. Inspect Pod and Job readiness.
4. Inspect service logs.

Do not expose management endpoints to replace this check with the public version check.

What it will not do

It does not restore a database. A failed or stopped major upgrade can leave some workloads updated. Schema migrations are forward-only. Older images may be incompatible with the migrated database. Promote a fix, or follow Rollback if database recovery is needed.

It manages the configured Helm release. Other workloads remain under their own owners.