Upgrade

This page owns the upgrade and recovery of the Helm release. The Compatibility Policy defines the guarantee and the support window. The release notes and MIGRATION.md state the actions for each release.

Pre-1.0 upgrades may require an intermediate release or manual preparation. The compatibility policy defines skip-version support. Read the release notes of every release between yours and the target. In-place downgrades are unsupported. Upgrade to 1.0 collects the actions on the way to 1.0.

Expect a release candidate (X.Y.Z-rc.N) before major upgrades, stopped cutovers and irreversible migrations. See the release policy.

Validate the actual production data

Use a restricted staging copy of production with the backup's access and retention rules. Running the target release's migrate run mode with the production profile runs the Liquibase update against staging. The update must use that exact release's changelog, contexts, PostgreSQL image/extensions, secrets, and role configuration.

Apply pre-boot isolation before restoring secrets or starting application code.
Check the database prerequisites before candidate startup.
If an intermediate release is required, rehearse that upgrade on the same isolated copy.
Run Liquibase update, not just validate or SQL generation.
Keep ingress and sandbox execution disabled.
Keep external destinations denied.
Do not use a test/specs profile, Hibernate ddl-auto, or an empty database as this proof.
Record migration duration, errors, free disk/WAL growth, and locks.
Record before/after checks for representative workspace relationships and encrypted-credential readability.
Record before/after checks for the practice catalog and ledger row counts.
Check databasechangelog for the expected target changesets.
Check databasechangeloglock for a released lock.
Verify application readiness.
Verify authenticated reads.
Set an outage budget.
If the rehearsal exceeds it, abort.
After fixing a migration failure, repeat against a fresh copy.

CI migration checks cover empty and seeded databases, not your production data. They do not replace this staging update.

Convert stored secrets for 1.0

Before the 1.0 migration, back up the database and encryption keys. Stop all database writers. For chart hosts, use Stopped schema upgrades. Run the migrate run mode with the existing system encryption key and the active credential key and version. Configure any missing keys before the migration. Keep the prior credential key and its version configured if stored credentials still use them.

The migration encrypts plaintext login-provider and LLM connection secrets and seals active and inactive JWT signing keys. It preserves the JWT signing identities. For integration credentials with no key version, it tries the configured active, prior and legacy system keys. Only a successful authenticated decryption permits conversion to the active credential key. It records that key's version without changing the connection state.

A missing or incorrect key stops the conversion and rolls back its writes. Keep writers stopped, correct the key configuration, then retry. Start the server after the migration succeeds. The server refuses plaintext secrets after the conversion.

Upgrade the host

Check the database prerequisites before you use the commands below.

Pause pull-based deployment, if enabled, and wait for its active run to finish.
Complete the silent deployment checklist, including a verified backup and a migration rehearsal.
Read the target release's migration instructions and update your protected site.yaml and Secret as required.
Verify the target release's joint lock, as Independent verification describes. Then follow Upgrade a schema-compatible release: verify the chart and apply the verified package with your site values.
Check the migration Job and each role's readiness before traffic or provider writes resume.
If Helm times out or reports a pending or failed release, follow Recover a stopped cutover. A failed Helm status does not prove that the migration Job has stopped.
If reconciliation is enabled, make its channel name the installed release before you resume its timer.

Upgrade a schema-compatible release

Read the release notes and migration instructions before an upgrade. If any intervening release says Stop every role, use Stopped schema upgrades. One-host installations use that procedure for every release change. Do not use automatic rollback across a migration that cannot run with the previous release.

For an upgrade whose schema remains compatible with the previous release, verify the release chart. Then apply the new chart, its profile, and your own values:

helm upgrade --install hephaestus "chart-$VERSION/hephaestus" -n hephaestus \
  -f "chart-$VERSION/hephaestus/profiles/small.yaml" -f site.yaml \
  --rollback-on-failure --wait=watcher --wait-for-jobs --timeout 45m

The migration Job of the new revision and the new Pods start together. The new Pods wait for the migrated schema, and the previous release serves until they are ready. --wait-for-jobs fails the upgrade when the migration fails, and --rollback-on-failure then returns the Deployments to the previous revision. The timeout is longer than the 10 minutes that a new Pod waits for the schema.

The server uses a rolling update. Its replacement is ready before the old Pod stops serving. The single-node profile reserves room for one replacement beside its draining server. Qualify sandbox isolation with the platform isolation gate before enabling work.

An old server drains for up to server.drainTimeoutSeconds, and Kubernetes no longer counts it while it stops. The value is 30 minutes by default and 10 minutes in the single-node profile. Existing review runs keep their gateway during this interval. Ready replacements accept new claims.

The Helm --timeout must cover the drain, 2 minutes of shutdown and up to 10 minutes of start. Past the timeout, Helm rolls the release back. If you set a longer drain, set a longer timeout. Start the next upgrade only after the old server is gone: kubectl --namespace hephaestus get pods -l app.kubernetes.io/component=server shows no Terminating Pod. A review runs in its own sandbox, not in a server (Agent runs).

For other database maintenance, use Database maintenance.

Roll forward to fix a failed upgrade. helm rollback and an older image do not undo a migration: restore the database from the backup taken before the upgrade (Restore).

Stopped schema upgrades

Use this procedure for a one-host release change or when the migration guide says Stop every role. It takes precedence over the normal upgrade and database-maintenance procedures. Do not use --rollback-on-failure: a rollback can restart old roles against a changed schema. Pause deployment automation that can change the release until this procedure is complete.

Before the first phase, record the current chart, profile, site values and all three replica counts. Prepare PostgreSQL backups, verify the restore, and keep the matching signed release lock and encryption keys. After phase 1, take and verify the final restore point while all roles are stopped, before phase 2 can change the schema. Verify the new release's chart before you use it. Each phase is a separate helm upgrade with that phase's chart, profile and site values:

Phase	Chart	`migration.enabled`	`server.replicas`	Before the next phase
1. Stop every role	Current release	`false`	0	Every old role Pod, including terminating Pods, and its sandbox Pods and Jobs are gone.
2. Migrate	New release	`true`	0	The migration Job of this revision completed successfully.
3. Start the new roles	New release	`true`	Recorded count	Readiness, authenticated reads, a practice review and a conversation pass.

Keep PostgreSQL running throughout these phases. Use server.replicas=0, not sandbox.enabled=false, to stop the server.

Before the cutover to conversation sandboxes, stop the source release's conversation loop too. After every server has stopped, foreground-delete its SandboxClaims and legacy sandbox Jobs, and wait for their Pods to disappear. Upgrade to 1.0 gives the stopped cutover procedure. The pull-based reconciler checks that procedure before migration.

Apply each row's values after your profile and site values, so that they take precedence. Use --wait=watcher --wait-for-jobs --timeout 45m, without automatic rollback. A server can drain for up to its configured timeout. If the first phase times out, keep all counts at zero and wait for every old Pod and sandbox to stop before phase 2.

In phase 2, identify the migration Job of the new Helm revision. Its default name is hephaestus-migration-<revision> for release hephaestus. Save its log with kubectl --namespace hephaestus logs job/<migration-job> -c migration > migration.log. Require its Complete condition: kubectl --namespace hephaestus wait --for=condition=complete job/<migration-job> --timeout=20m. A failed or timed-out Job does not permit phase 3. Keep all roles stopped, preserve the log and failed change set before the Job's 24-hour expiry, and follow failed-migration recovery.

A refusal can follow earlier committed changesets. Before you return to the old release, restore the matching pre-upgrade backup. An image or Helm rollback does not undo those changes.

Phase 3 creates another migration Job for the same release. It has no new schema changes to apply. Resume deployment automation only after its release and replica counts match the verified installation. On one host, keep it paused through the final installer run in Upgrade the host.

Change a volume size

A StatefulSet does not allow a change of its volumeClaimTemplates. The PostgreSQL StatefulSet has a volume claim template:

<release>-postgresql, sized by postgresql.persistence.size.

If an upgrade changes its size, for example through a new profile, helm upgrade fails. The error says StatefulSet ... is invalid: spec: Forbidden: updates to statefulset spec for fields other than .... With --rollback-on-failure, Helm then returns to the previous revision, and nothing changes.

Kubernetes cannot shrink a volume. If the new size is smaller, keep the current size in your own values file, as Storage and retention says.

If the new size is larger, follow these steps for the PostgreSQL StatefulSet. The volume of PostgreSQL is data-<release>-postgresql-0.

Check that the storage class allows expansion:
```
kubectl get storageclass
```
If ALLOWVOLUMEEXPANSION is not true for the class of the volume, stop. Keep the current size in your own values file.

Make the volume larger. Use the new size from the chart in place of <new size>:

kubectl --namespace hephaestus patch pvc data-hephaestus-postgresql-0 \
  --patch '{"spec":{"resources":{"requests":{"storage":"<new size>"}}}}'

Wait until kubectl --namespace hephaestus get pvc shows the new capacity. If the capacity stays at the old size, check the conditions of the volume:
```
kubectl --namespace hephaestus get pvc data-hephaestus-postgresql-0 -o jsonpath='{.status.conditions[*].type}'
```
FileSystemResizePending means that the storage driver resizes the file system only when the Pod starts again. Then continue with steps 4 and 5, and after step 5 delete the Pod: kubectl --namespace hephaestus delete pod hephaestus-postgresql-0. The StatefulSet starts it again with the larger volume.

Delete the StatefulSet and keep its Pods and volumes:

kubectl --namespace hephaestus delete statefulset hephaestus-postgresql --cascade=orphan

Run the same helm upgrade again. Helm creates the StatefulSet with the new template. The new StatefulSet takes over the Pods and the volumes. It restarts a Pod only if the upgrade also changes the Pod template.

Before you change the volume of PostgreSQL, take a backup (Backup and restore).

Check that it worked

Run helm history hephaestus --namespace hephaestus. The new revision is deployed.
Read the log of the migration Job of that revision: kubectl --namespace hephaestus logs job/hephaestus-migration-<revision>. It logs Database migration completed.
Run kubectl --namespace hephaestus get pods. Each role Pod of the new revision is ready.
Query application_build_info in Prometheus, as Release metric shows. Each role reports the new version.

If migration fails

CAUTION: Failed migration

Do not clear checksums, edit released changelogs, or mark the changeset as run. Do not run a release other than the previous one against the partly migrated database. If the migration guide entry of the new release starts with Stop every role, do not run the previous one either (Rollouts and rollback). Do not release a Liquibase lock by hand except as One migration at a time allows. Removal of a stale lock does not repair a failed migration.

Keep ingress and all application writers stopped.
If the host uses pull-based deployment, pause reconciliation.
Preserve the error and failed changeset identity.
Preserve databasechangelog/lock diagnostics and the target image lock.
If feasible, preserve a forensic snapshot.
Follow Backup & restore to restore the pre-upgrade backup into a fresh database.
Use matching PostgreSQL version/extensions, encryption keys, and the previous signed release lock.
Reconcile external writes, as Reconcile external writes after restore describes.
Boot only under pre-boot isolation.
Read back silent mode, and turn it on if it is off.
Verify the restored schema, representative data, and readiness.
Complete Prove silence, Inspect and decide, and Turn off silent mode deliberately in the silent checklist.
Report actual recovery point/time and any lost writes.
Preserve the failed database until investigation/retention permits removal.
Promote the recovered release, with allow-rollback if it is older than the failed upgrade. Until the channel names it, the next tick can reapply the failed upgrade.
When the promotion has published the channel, re-enable the reconcile timer. The promotion's version check passes when the host applies the channel. It fails if the timer stays off for 15 minutes.

Drill

Introduce a controlled migration failure only in a disposable rehearsal.
Restore the old backup/release pair.
Prove old-version readiness and data integrity without any provider write.

Rollback

CAUTION: Images alone cannot reverse a schema change

Reverting images alone is not a rollback. Never run a release older than the previous one against a newer schema. Rollouts and rollback says when the previous release can run on it. Never reconstruct an old lock or resolve mutable image tags.

To roll back, follow If migration fails to restore the previous release's database, secrets, and signed lock.

Update checks

Instance administration → Overview shows the release this server reports from its lock (runtime identity handoff) and whether GitHub has published a newer one. A failed, rate-limited, disabled or never-completed check is shown as exactly that, never as "up to date". For a newer release the card links its notes and the upgrade guide and says whether that release carries schema migrations. The flag describes the newest release only: read every release note between yours and it before upgrading. Nothing here verifies an artifact or performs an upgrade.

The server asks GitHub a minute after start and then once a day. After a failure it retries after 15 minutes, doubling up to a day, or later when GitHub names a longer wait. Check now ignores the daily cache, never delays the automatic retry, and only waits out a wait GitHub named. A restart forgets the last answer. A commit channel or a local build runs no release and is never checked.

Each check is one unauthenticated HTTPS GET to https://api.github.com/repos/hephaestus-build/Hephaestus/releases/latest with the static user agent Hephaestus-release-check and, after the first answer, its ETag. It carries no version, credential, workspace data or configuration. GitHub sees the source IP and the time.

On an air-gapped host, set config.releaseCheckEnabled: false in site.yaml.
Apply the values with the upgrade command.

The overview then shows the identity without a check. A failed check reaches only instance administration, never health or readiness.

Deployment environment and release history

Set deploymentEnvironment in site.yaml to the name of the deployment, such as production or staging. Apply the values with the upgrade command. Use lowercase letters, digits, ., _ or -, at most 64 characters. Hephaestus refuses to start with another value. When it is unset, the instance reports local. The webapp, the Release card, log lines and exported traces all use this name.

The Release card also says when this instance first started the release that it runs. Show release history lists the last ten releases that the server started, newest first. The server records a release when it starts one that differs from the last recorded release. A restart adds nothing. A rollback adds the older release again.

The time is when the server first started that release, not when its images were pulled. The history names no person. Each server reports its own identity in /actuator/info only.

Release-specific procedures

Remove the retired sync broker values before upgrading

Remove nats and images.nats from every values file before rendering the target chart, including site and profile overrides. The chart refuses those keys. Regenerate image values from the target release lock.

This procedure replaces these pending entries:

NATS broker certificate names.
External NATS broker: NATS Server 2.11 or later.
Move the GitHub webhook byte bound into the stream map.

Their broker instructions no longer apply.

Remove hephaestus.sync.nats.* properties and broker environment settings and credentials as the migration entry of the same name lists. After the upgraded server delivers sync hints through PostgreSQL, stop any external broker used only by this release.

The old bundled PersistentVolumeClaim can remain after Helm removes the workload. Remove only its confirmed release-owned claim when its data is no longer needed.

Some releases need a procedure of their own before the new runtime roles start. The database migration instructions say when one applies.

Prepare an older database: intermediate upgrades for older databases.
Native Hibernate JSON cutover: the stopped-writer accounting cutover.
Practice model cutover checks: the pilot rehearsal and a refused practice model upgrade.

Hosts with reconciler tooling of 1.0.0-rc.4

Do this procedure once, on a host with pull-based deployment whose tooling is that of 1.0.0-rc.4. The tooling link names the release whose tooling runs.

1.0.0-rc.8 removes the nats image with the sync broker. The verifier of 1.0.0-rc.4 compares a fixed list of chart images, so it refuses a release that removes one. The verifiers from 1.0.0-rc.5 on read the images from the signed release lock and accept it.

Promote the channel to v1.0.0-rc.5, and complete its migration instructions.
Wait until the reconciler logs Applied v1.0.0-rc.5. Then /var/lib/hephaestus/tooling names /var/lib/hephaestus/releases/v1.0.0-rc.5.
Promote the target release.

A host whose tooling is 1.0.0-rc.5 or later needs no step.

Hosts with reconciler tooling from before 1.0.0-rc.4

Do this procedure once, on a host with pull-based deployment.

The reconciler verifies a release with the installed tooling before it adopts newer tooling. Tooling from before 1.0.0-rc.4 expects five chart images, and 1.0.0-rc.4 has nine. Thus, the old verifier stops each run with does not pin the v1.0.0-rc.4 images. The running release is not affected.

Do not move the tooling link by hand. Each run sets it again from applied.json.

Make the channel name v1.0.0-rc.4, without freeze. A frozen channel adopts the tooling, but it never verifies the release.
If the host still runs an earlier release, complete the migration instructions of each release first. The second run below applies 1.0.0-rc.4.
Stop the timer: sudo systemctl stop hephaestus-reconcile.timer. Confirm that hephaestus-reconcile.service no longer runs.

Authenticate the release lock and record the release that it names. The script stops at the first failure and leaves applied.json as it was:

sudo bash -s <<'SCRIPT'
set -euo pipefail
VERSION=1.0.0-rc.4
STATE=/var/lib/hephaestus
LOCK="release-v$VERSION.chart.json"
BASE="https://github.com/hephaestus-build/Hephaestus/releases/download/v$VERSION"
WORK=$(mktemp -d)
if systemctl is-active --quiet hephaestus-reconcile.timer; then
  echo "Stop the timer first" >&2
  exit 1
fi
if [ "$(jq -er .release "$STATE/applied.json")" = "v$VERSION" ]; then
  echo "applied.json already names v$VERSION" >&2
  exit 1
fi
curl -fsSL "$BASE/$LOCK" -o "$WORK/$LOCK"
curl -fsSL "$BASE/$LOCK.sigstore.json" -o "$WORK/$LOCK.sigstore.json"
cosign verify-blob --bundle "$WORK/$LOCK.sigstore.json" \
  --certificate-identity 'https://github.com/hephaestus-build/Hephaestus/.github/workflows/release.yml@refs/heads/main' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com "$WORK/$LOCK"
commit=$(jq -er --arg release "v$VERSION" \
  'select(.schemaVersion == 2 and .release == $release) | .commit | select(test("^[a-f0-9]{40}$"))' "$WORK/$LOCK")
git -C "$STATE/checkout" fetch --quiet origin "$commit"
cp -p "$STATE/applied.json" "$STATE/applied-before-tooling.json"
cp -p "$STATE/applied.json" "$STATE/applied.json.new"
jq --tab --arg release "v$VERSION" --arg commit "$commit" \
  '.previous = .release | .release = $release | .commit = $commit
   | .appliedAt = (now | strftime("%Y-%m-%dT%H:%M:%S.000Z"))' \
  "$STATE/applied-before-tooling.json" > "$STATE/applied.json.new"
mv "$STATE/applied.json.new" "$STATE/applied.json"
SCRIPT

Read /var/lib/hephaestus/applied.json. To undo, restore applied-before-tooling.json to applied.json now, while the timer is stopped.
Start the timer: sudo systemctl start hephaestus-reconcile.timer.

The next run logs Adopted the tooling of v1.0.0-rc.4. The run after it logs Applied v1.0.0-rc.4: the new verifier checks the release, and a host that already runs it restarts no Pod.

After the timer starts, do not restore the record. The older tooling would then apply the older chart. Use Rollback.

Before a later release, also do Hosts with reconciler tooling of 1.0.0-rc.4.

Sandbox capacity during upgrade

A supplied sandbox.budget must hold both the steady-state requests and the drained-upgrade peak. Sandbox capacity budget owns the formula and the values to supply. auto requires both CPU and memory values. With explicit counts, the budget is optional. A supplied budget rejects a count that exceeds either phase.

Finish the review drain before the upgrade's transient peak. Do not resume review work while migration and stanza Jobs or replacement-role surge still consume that budget. A drain timeout does not prove that reviews have ended. Check the remaining runs and complete the stopped procedure before proceeding. Conversation sandboxes can remain warm within their admitted quota. The cutover to 1.0 still removes the source release's claims and legacy Jobs before migration, as Upgrade to 1.0 describes.

The small profile supplies its existing allocatable budget. Managed clusters have no assumed aggregate budget. When moving a managed cluster to auto, declare its allocatable CPU and memory before upgrading. Keep the current template requests until workload qualification supplies measured replacements.

Validate the actual production data​

Convert stored secrets for 1.0​

Upgrade the host​

Upgrade a schema-compatible release​

Stopped schema upgrades​

Change a volume size​

Check that it worked​

If migration fails​

Rollback​

Update checks​

Deployment environment and release history​

Release-specific procedures​

Remove the retired sync broker values before upgrading​

Hosts with reconciler tooling of 1.0.0-rc.4​

Hosts with reconciler tooling from before 1.0.0-rc.4​

Sandbox capacity during upgrade​