Back up and restore

Back up the database, the keys and the release identity of an instance, and restore the instance from them. The operations runbook has the recovery gates.

Backup policy for 1.0

Scheduled off-host backups and recovery-time qualification are deferred for the 1.0 release. The chart leaves postgresql.backup.enabled false by default. It then takes no scheduled database backup and does not archive WAL. Loss of the database volume can lose all application data. 1.0 makes no measured recovery-point or recovery-time promise.

The procedures below remain available for operators who configure backups. Each upgrade still requires a verified recovery point. If backups are off, first configure the chart backup procedure and verify an isolated restore. Do not start the upgrade without that proof.

What to back up

Data	Recovery requirement
PostgreSQL	Preserve application state, encrypted provider credentials, JWT signing keys, practice results and delivery history.
Encryption keys and configuration	Keep the keys and required Secret values outside the cluster as Secret custody describes. The database alone cannot recover encrypted values.
Signed release lock and site values	Restore the release and configuration that match the database. Verify its chart and image digests.
Platform recovery material	Keep the k3s token, datastore, manifests, certificates, and storage configuration as Platform recovery material describes.

Repository mirrors and inputs for active attempts are disposable. The roles rebuild them from PostgreSQL and the upstream provider. Do not use copied attempt folders to resume an in-flight review.

Platform recovery material

The application backup does not contain the Kubernetes datastore or persistent volume contents. Keep the k3s configuration, the installed platform manifests, and the storage layout with the recovery record. Keep the server token outside the host, with restricted access. The token decrypts confidential data in the k3s datastore.

Follow the k3s datastore backup procedure for the actual datastore type. SQLite and embedded etcd use different procedures. A datastore backup does not replace the PostgreSQL recovery set. Before relying on host recovery, rehearse both on an isolated replacement host. No replacement-host recovery time is qualified for 1.0.

Automated off-host backup

Enable the chart's pgBackRest backup after you configure an off-host S3-compatible repository and its credentials. Back up the database gives the chart values, backup Jobs and TLS client identity. Keep the repository passphrase outside the cluster.

A configured schedule is not proof of a usable recovery point. Check WAL archival, completed backups and repository access. Rehearse an isolated restore, including clone lockdown, before relying on the backup. For an installation that uses backups, define recovery objectives with the operations runbook.

Backup retention

postgresql.backup.retentionFull sets the number of full pgBackRest backup chains to keep. Its default is 2. Erased data can remain in a backup until that backup expires under this setting and the repository expires it. This setting is a count, not a time limit. Keep the restricted erasure recovery manifest for the same window as every backup that can restore erased data, including separately retained copies.

Back up the database

Backup is off by default. While the bundled PostgreSQL runs without it, each helm install and helm upgrade prints a warning that links to this section. The chart cannot see a backup that you take yourself, for example a volume snapshot. In that case, the warning stays.

With postgresql.backup.enabled=true, the bundled PostgreSQL archives its WAL to one S3-compatible bucket with pgBackRest, as Backup and restore describes. The repository is encrypted with BACKUP_CIPHER_PASS. A CronJob takes a full backup each Sunday at 02:15 UTC. Another takes a differential backup on the other days at the same time. Backup retention defines how many full backup chains stay and how long erasure recovery decisions must be available. A Job of each release creates the repository's stanza, so that the WAL archive works from the install.

The backup Jobs do not mount the PostgreSQL volume. They read the data directory through the pgBackRest TLS server in the PostgreSQL Pod. That server admits only a client certificate for the common name hephaestus-backup that ca.crt signed.

To prepare backups:

Create the bucket for this instance alone, and an identity with read, write and delete rights in it.
Add BACKUP_S3_KEY and BACKUP_S3_KEY_SECRET to the release Secret. Run sh hephaestus-keys.sh --backups to generate BACKUP_CIPHER_PASS, as Create the Secret describes. Keep a copy of the passphrase as Secret custody describes.
Set postgresql.backup.clientTlsSecret to a TLS Secret for the common name hephaestus-backup. It needs ca.crt, tls.crt and tls.key, signed by the PostgreSQL CA. The one-host platform issues hephaestus-postgresql-backup-tls and sets this value.
Set postgresql.backup.enabled=true and postgresql.backup.s3 with the bucket, the endpoint host name and the region. Set cidr and port when the endpoint is in a private address range. When a private CA signs the endpoint's certificate, add it to the PostgreSQL TLS Secret as s3-ca.crt, and set postgresql.backup.s3.privateCa=true.
Upgrade the release, and check that the stanza Job completed.

Backup bucket credentials and Backup cipher passphrase give the rotation of these values.

A failed WAL archive keeps the WAL on the PostgreSQL volume until the volume is full. The alert rules watch the WAL archive, the age of the latest backup and the free space of the volume.

Preserve the database, keys and signed release identity as one recovery set. Keep the originals until the recovery is verified.

Restore

Choose the isolated restore drill or live recovery procedure for the intended destination. An isolated clone must complete lockdown before migrations or application roles start. Do not apply clone lockdown to the live recovery database.

A restore replaces the bundled database with a backup. It loses every change after the backup or the time that you select. The restored database is readable only with the keys that sealed its values. Keep them as Secret custody describes. Reconcile external writes, as Reconcile external writes after restore describes.

Before reducing application replicas, record SHOW max_connections and set postgresql.parameters.max_connections to that value. Keep this override until WAL replay finishes. Include it in the recovery set for a replacement host. Automatic connection sizing decreases with replica counts, but PostgreSQL rejects recovery below the source setting. The PostgreSQL recovery requirements describe this constraint.

Prepare replacement storage

On a replacement host, restore the release keys, TLS Secrets, verified chart and site values first. Keep migration disabled and all application replica counts at zero. Keep the failed volume until recovery is verified.

Before enabling restore, provision an empty claim named data-<release>-postgresql-0 in the release namespace. Use the PostgreSQL storage class, access mode and size from the site values. For node-local storage, bind the new volume to the replacement node. The restore Job needs this claim before PostgreSQL starts. Do not initialize a new database in it.

Then follow the isolated or live recovery procedure below for the intended destination.

Isolated restore drill

Run the following procedure only on a disposable, isolated clone. It changes control settings with clone lockdown SQL. Use Recover the live installation for a live recovery.

Upgrade the release with postgresql.backup.restore.enabled=true. Set postgresql.backup.restore.time to restore the database as it was at that time, for example 2026-10-24 18:00:00+00. Leave it empty to restore the latest backup and every archived WAL segment. While the setting is on, the server and PostgreSQL stop, and the backups pause.

Wait for the Job <release>-postgresql-backup-restore-<revision> to complete:

kubectl --namespace hephaestus wait --for=condition=complete \
  job/hephaestus-postgresql-backup-restore-<revision> --timeout=30m

The Job retries while PostgreSQL still shuts down.

Keep the clone isolated before any application code starts. Deny provider and model egress independently of silent mode.
Upgrade with postgresql.backup.restore.enabled=false, migration.enabled=false, and a server replica count of zero. PostgreSQL alone replays the WAL up to the target. Readiness does not prove that replay finished. Wait until this command prints f:
```
kubectl -n hephaestus exec hephaestus-postgresql-0 -c postgresql -- \
  sh -c 'exec psql -Atc "SELECT pg_is_in_recovery()" -U "$POSTGRES_USER" -d "$POSTGRES_DB"'
```

Apply the lockdown SQL from the verified chart, before starting migrations or application roles:

kubectl -n hephaestus exec -i hephaestus-postgresql-0 -c postgresql -- \
  sh -c 'exec psql -v ON_ERROR_STOP=1 -U "$POSTGRES_USER" -d "$POSTGRES_DB"' \
  < "chart-$VERSION/hephaestus/files/restore-clone-lockdown.sql"

Read back silent mode, paused feedback and disabled conversation bindings. Keep external egress denied.
Resolve provider write receipts before the server starts.
Set migration.enabled=true and start the server alone. Verify migrations, readiness and authenticated reads.
Restore the other replica counts only after the isolated checks pass. Follow Prove silence before opening any external destination.

The restored database has the role passwords that it had at the backup. The migration Job of step 8 sets the passwords of hephaestus_owner and hephaestus_app from the Secret again. If DATABASE_ADMIN_PASSWORD changed since the backup, put the old value back in the Secret before step 3. Rehearse a restore on a copy of the instance before you need one.

Recover the live installation

Close traffic and provider reception. Stop application roles and sandboxes, and pause reconciliation. Provider deliveries that arrive while reception is closed are lost. Step 10 recovers their effect.
Keep a copy of the failed data and record the chosen recovery point and expected lost writes.
Check whether DATABASE_ADMIN_PASSWORD changed after the recovery point. If it changed, restore the matching value in the Secret before startup. The migration Job sets the passwords of hephaestus_owner and hephaestus_app again. Complete steps 1, 2 and 4 of the isolated drill with application replicas at zero. Keep provider and model egress denied. Wait for pg_is_in_recovery() to return f.
Do not apply restore-clone-lockdown.sql to the live recovery database. Verify silent mode, delivery status, review triggers and conversation bindings against the intended live settings.
Plan the provider reconciliation, as Reconcile external writes after restore describes.
Enable migrations and start the server alone. Verify readiness, authenticated reads and encrypted values.
Before access or delivery resumes, reapply post-backup security and privacy decisions from the restricted incident record. Before restarting services gives the required suspension, session, consent and deletion checks.
Start the worker and require its isolation probe. Permit only the approved model endpoint for the controlled test while provider writes remain blocked. Test an approved practice review and conversation with delivery paused.
Complete Prove silence, Inspect and decide, and Turn off silent mode deliberately in the silent checklist. Restore the intended live control settings, provider reception and egress. Start the remaining roles and reopen traffic after the checks pass.
Verify every connected provider and feedback destination. Resume reconciliation only after its release and apply mode match the recovered deployment. Then recover the deliveries after the backup as Which providers deliver again describes.

Reapply erasure decisions before processing resumes

Keep the normal server Deployment stopped while PostgreSQL is restored. Keep public access and provider egress closed. Before replay or normal processing, the server must pass the erasure-recovery startup profile. This profile runs the existing person-erasure service before lifecycle workers, scheduled tasks or inbox consumers start. A missing or invalid manifest, an unknown provider instance, or an incomplete erasure stops startup.

From the restricted incident record, prepare a JSON manifest of every erasure after the chosen recovery point. Include each person's account ID, when known, and all known exact native identities. Use numeric user IDs for GitHub and GitLab. Use a user ID and team ID for Slack. Use a user UUID for Outline. Specify the exact provider type and server URL from the restored provider inventory. Never substitute a login, display name or email.
Keep the file outside the database and mount it read-only in the server Pod. Restrict access to the recovery operators. Do not put it in Helm values, a ConfigMap, a command-line argument or shared logs. Only its mounted path goes in configuration.
Keep retained server evidence volumes under access controls until recovery is complete. Before reuse, their servers must reconcile their folders against the recovered receipts. Keep provider egress denied. Follow A server with evidence is offline.
Reconcile external writes before the server starts.
Start the server with erasure-recovery added to its normal active profiles. Set HEPHAESTUS_PRIVACY_RECOVERY_MANIFEST to the mounted path. Set AGENT_ENABLED=false. Keep public access closed until normal operation resumes.
Wait for server startup to complete. If it stops, keep the other Deployments stopped, correct the private input or unfinished erasure, and retry. Do not remove the recovery profile to bypass the failure.
Remove the recovery profile and temporary manifest mount before you resume normal operation. The Kubernetes procedure below restores the original Deployment.
After normal server startup completes, open access and provider egress. Start the other roles.

The manifest uses this structure:

{
  "version": 1,
  "restorePoint": "2026-11-01T00:00:00Z",
  "administratorAccountId": 10,
  "noLaterErasures": false,
  "people": [
    {
      "accountId": 42,
      "identities": [
        {"providerType": "GITHUB", "serverUrl": "https://github.com", "subject": "123456"},
        {"providerType": "GITLAB", "serverUrl": "https://gitlab.example.test", "subject": "234567"},
        {"providerType": "SLACK", "serverUrl": "https://slack.com", "subject": "U123456", "teamId": "T123456"},
        {"providerType": "OUTLINE", "serverUrl": "https://outline.example.test", "subject": "00000000-0000-4000-8000-000000000042"}
      ],
      "externalCopiesRemoved": true
    }
  ]
}

The administrator must be active and have instance-admin rights in the restored database. Another administrator must authorize that administrator's own erasure. Set externalCopiesRemoved to true only after checking the provider locators and confirming removal as the un-deliver procedure requires. Recovery does not delete provider-hosted copies itself.

If no erasure happened after the recovery point, the operator must declare that explicitly:

{
  "version": 1,
  "restorePoint": "2026-11-01T00:00:00Z",
  "noLaterErasures": true,
  "people": []
}

An empty list without this declaration does not permit startup. The file itself is never stored in the database or logged. The existing erasure workflow temporarily keeps its frozen subject scope while a request is active. After completion, only its normal suppression controls and erasure receipts remain. Keep the manifest under the backup retention rule.

Mount the manifest for a Kubernetes recovery

The chart uses the prod profile and a startup probe. A large erasure manifest can take longer than that probe permits. Use this temporary Deployment patch while normal workloads and public access remain stopped. It preserves the readiness probe and removes the startup and liveness probes until recovery completes. It extends the rollout progress deadline to one day.

The mounted Secret contains private identities. Restrict its namespace access to recovery operators. This Secret procedure requires a manifest below 1 MiB. For a larger manifest, mount a protected read-only volume at the same path instead of a Secret. Reapply every later decision in one startup before you permit processing.

Set NAMESPACE, SERVER_DEPLOYMENT and ERASURE_MANIFEST to the namespace, server Deployment and private manifest file.

umask 077
server_recovery_dir=$(mktemp -d)
erasure_secret="${SERVER_DEPLOYMENT}-erasure-recovery"
kubectl -n "$NAMESPACE" get deployment "$SERVER_DEPLOYMENT" -o json > "$server_recovery_dir/before.json"
kubectl -n "$NAMESPACE" create secret generic "$erasure_secret" --from-file="manifest.json=$ERASURE_MANIFEST"
server_container=$(jq -r '.spec.template.spec.containers[0].name' "$server_recovery_dir/before.json")
jq -n --arg container "$server_container" --arg secret "$erasure_secret" '
  {spec: {replicas: 1, progressDeadlineSeconds: 86400, template: {spec: {
    containers: [{name: $container, startupProbe: null, livenessProbe: null,
      env: [
        {name: "SPRING_PROFILES_ACTIVE", value: "prod,erasure-recovery"},
        {name: "HEPHAESTUS_PRIVACY_RECOVERY_MANIFEST", value: "/var/run/hephaestus/erasure-recovery/manifest.json"},
        {name: "AGENT_ENABLED", value: "false"}],
      volumeMounts: [{name: "erasure-recovery", mountPath: "/var/run/hephaestus/erasure-recovery", readOnly: true}]}],
    volumes: [{name: "erasure-recovery", secret: {secretName: $secret, defaultMode: 288}}]
  }}}}' > "$server_recovery_dir/recovery-patch.json"
kubectl -n "$NAMESPACE" patch deployment "$SERVER_DEPLOYMENT" --type=strategic --patch-file="$server_recovery_dir/recovery-patch.json"
kubectl -n "$NAMESPACE" rollout status deployment "$SERVER_DEPLOYMENT" --timeout=60m

A command timeout does not stop the Pod. If recovery exceeds one hour, repeat only the wait command. If it approaches one day, extend the progress deadline before it expires. If the deadline has expired, wait for the recovery Pod's readiness with kubectl wait --for=condition=Ready pod <pod-name> --timeout=60m. A failed startup logs person_data.recovery.failed with a closed person_data.recovery.reason code. It does not log the manifest or an exception that can quote its identities.

After recovery succeeds, restore the original Deployment specification, which has zero replicas from the maintenance stop. Wait for its recovery Pod to stop before you remove the Secret. Then start the normal server.

jq '{spec: .spec}' "$server_recovery_dir/before.json" > "$server_recovery_dir/restore-patch.json"
kubectl -n "$NAMESPACE" patch deployment "$SERVER_DEPLOYMENT" --type=merge --patch-file="$server_recovery_dir/restore-patch.json"
server_selector=$(jq -r '.spec.selector.matchLabels | to_entries | map(.key + "=" + .value) | join(",")' "$server_recovery_dir/before.json")
kubectl -n "$NAMESPACE" wait --for=delete pod -l "$server_selector" --timeout=5m
kubectl -n "$NAMESPACE" delete secret "$erasure_secret"
kubectl -n "$NAMESPACE" scale deployment "$SERVER_DEPLOYMENT" --replicas=1
kubectl -n "$NAMESPACE" rollout status deployment "$SERVER_DEPLOYMENT" --timeout=15m

Keep the original private manifest under the backup retention rule.

Before restarting services

A restore reverses account suspensions, session revocations, recipient consent and deletions as well as application data. Sessions issued after the snapshot must sign in again. Sessions revoked after it can become valid again. Ingest deduplication cannot prevent external duplicates after PostgreSQL loses delivery history. Scheduled sync cannot recover all events in the gap.

Reapply post-backup security and privacy decisions before access or delivery resumes.
Keep the signing keys. Diagnose missing encryption keys separately from corrupt signing data.
Recover the provider gap, as Reconcile external writes after restore describes.
Restore with the backup's signed release lock. Then upgrade.

Do not restore into an older application version.

Check that it worked

Run kubectl --namespace hephaestus get jobs. The stanza Job and the latest backup Job are Complete.
Check that HephaestusBackupStale does not fire (Alert rules).
Restore the latest backup on a clone, as Isolated restore drill describes.

Unsupported PostgreSQL versions

Use the PostgreSQL version and image of the verified release. The compatibility policy owns the supported versions and upgrade prerequisites.

Backup policy for 1.0​

What to back up​

Platform recovery material​

Automated off-host backup​

Backup retention​

Back up the database​

Restore​

Prepare replacement storage​

Isolated restore drill​

Recover the live installation​

Reapply erasure decisions before processing resumes​

Mount the manifest for a Kubernetes recovery​

Before restarting services​

Check that it worked​

Unsupported PostgreSQL versions​