Backup & Restore

This page applies to the self-host compose stack.

On the Helm chart, Kubernetes owns backup and restore. Secret custody and rotation lists the keys that a chart restore needs.

PostgreSQL holds application state. The server and worker keep repository mirrors and inputs for active attempts on disk. These files are disposable. The roles rebuild them from PostgreSQL and the upstream provider.

What to back up

Data	Where	Priority
PostgreSQL database	Compose-managed `postgresql-data` volume mounted by `postgres`	Everything. Accounts, workspaces, practice results, feedback, encrypted provider tokens, JWT signing keys.
`.env`	`/opt/hephaestus/docker/self-host/.env`	Equal priority. Its encryption keys protect values inside the database backup. Without them, provider credentials and signing material are unreadable.
Signed release lock and deployment overrides	`release-lock.env` and any operator-owned Compose/configuration files	Restore the exact release and configuration associated with the database, not the latest release.
TLS certificates	`letsencrypt` volume mounted by `reverse-proxy`	Optional — `Let's Encrypt` re-issues when permitted to connect (rate limits permitting).
Repository mirrors and attempt folders	`git-repos` volume (server) and `worker-git-repos` volume (worker), both mounted at `/data/git-repos`	These files are disposable. Do not back them up as review evidence. A missing mirror is cloned again on the next sync or review. Restoring PostgreSQL preserves admitted results and citation checks. In-flight reviews must be retried, not resumed from copied folders.
NATS JetStream	`nats-data` volume	Preserve broker snapshots or cold copies when event-gap recovery/forensics requires them. Never replay a store divergent from PostgreSQL without the runbook recovery checks.

For production backup automation and recovery gates, start with the production operations runbook. The logical dump below is the minimal fallback, not arbitrary point-in-time recovery.

Automated off-host backup (self-host Compose)

The optional pgBackRest overlay archives WAL to one operator-owned S3-compatible bucket. It encrypts repository content with AES-256-CBC. It keeps two full backup chains with their dependent differential backups. The host's systemd timers run full backups each Sunday at 02:15 UTC. They run differential backups Monday through Saturday at 02:15 UTC. PostgreSQL also forces a WAL segment switch after 60 seconds of unarchived activity.

This is a target, not a guaranteed RPO. Repository failure, a stalled archiver, or a missing key can make the usable recovery point older. The operations runbook defines the release gate and broker procedure.

CAUTION: Backup secrets and archive failures

A lost passphrase makes the repository unusable. Generate BACKUP_CIPHER_PASS independently with this command:

openssl rand -base64 48

Store it outside the host with the release lock and application encryption keys. Keep .env mode 0600. Never print docker compose config into shared logs: it contains the repository key and passphrase. Do not place credentials in pgbackrest.conf or the systemd units.

A failed archive command keeps WAL on the database host until it fills the disk. Alert on archive failures and free space, not only the next scheduled backup.

Name a backup owner.
Name a separate restore approver.
Create a private bucket for one instance in a separate failure domain.
Give the scheduled backup identity only the read/write/delete rights pgBackRest needs within that bucket.
Give a rehearsal clone a separate read-only identity.
Require TLS.
Restrict deletion with a destination policy independent of the database host.
Test that the policy still permits pgBackRest to expire old backup chains. Blanket object locks can prevent expiration.
Set the six BACKUP_* values in /opt/hephaestus/docker/self-host/.env. BACKUP_S3_ENDPOINT is a hostname without https://. TLS verification stays enabled.
Enable the overlay only after the destination and all credentials are ready.

From /opt/hephaestus/docker/self-host, use this exact Compose command for all backup actions:

dc() { docker compose --env-file .env --env-file release-lock.env -f compose.yaml -f compose.backup.yaml "$@"; }
dc up -d postgres
dc run --rm --no-deps pgbackrest --stanza=hephaestus stanza-create
dc run --rm --no-deps pgbackrest --stanza=hephaestus check
dc run --rm --no-deps pgbackrest --stanza=hephaestus --type=full backup
dc run --rm --no-deps pgbackrest --stanza=hephaestus verify
dc run --rm --no-deps pgbackrest --stanza=hephaestus --output=json info

PostgreSQL can retain WAL before stanza-create succeeds, but it cannot complete archival then.

If check, the first backup, or verify fails, stop.
If check, the first backup, or verify fails, investigate.

verify checks repository contents. It does not prove that a clean instance boots or that delivery stays silent.
Install docker/self-host/systemd/hephaestus-backup@.service and the two timer files into /etc/systemd/system/.
Run:
```
systemctl daemon-reload
```

Run:

systemctl enable --now hephaestus-backup-full.timer hephaestus-backup-diff.timer

Check:

systemctl list-timers 'hephaestus-backup*'

Check each service's journal.
Until operator alerting owns the alert path, install a host-level failed-unit alert.
Until operator alerting owns the path, test that host-level alert.

A failed service must alert through the operator alert channel. A configured timer is not a completed backup.
Before relying on the timer, run one scheduled execution.
Before relying on the timer, observe that execution.
Before relying on the timer, run a deliberately failed execution against an isolated destination.
Before relying on the timer, observe that failure.

Keep .env, release-lock.env, operator overrides, and all active/prior application credential keys in a separate encrypted off-host custody path. The pgBackRest repository contains only PostgreSQL and WAL.

Restrict bucket and key access to backup operators.
Audit reads and deletion.
Test retrieval without the production host.
Set destination lifecycle and hold rules to your data-retention policy.
Do not delete WAL needed by the oldest retained full backup.
Rehearse after each policy change.

Count-based pgBackRest retention is not a legal or time-based erasure policy.

For a clean PITR rehearsal, isolate the clone before restoring secrets, as the runbook requires. The restore requires these conditions:

The backup's signed PostgreSQL image and configuration.
A new empty PostgreSQL data volume.
The same repository and cipher passphrase.
No PostgreSQL process running on that volume.

CAUTION: Isolated restore only

Never point a rehearsal at the production Compose project or volumes. Do not combine the backup and restore overlays. Keep the clone's provider egress denied. Keep AGENT_ENABLED=false on server and worker. Turn on silent mode before any application role starts.

CAUTION: Production data and application startup

Never pipe this file into the production database. The restore overlay also refuses direct starts of the server, worker, and webhook roles. For later application-health checks, use a separately reviewed isolated override. Turning silent mode off is an explicit step, not part of the restore command.

Choose a recorded UTC target covered by a full backup and uninterrupted archived WAL.
Set BACKUP_RESTORE_S3_KEY and BACKUP_RESTORE_S3_KEY_SECRET to a read-only bucket identity on the clone.
From the isolated self-host directory, run the block below.

The restore overlay requires the read-only identity for both the restore service and PostgreSQL WAL replay. It never passes BACKUP_S3_KEY or BACKUP_S3_KEY_SECRET to either container. A plain up with the restore overlay starts only PostgreSQL. Profiles exclude application, broker, and public ingress services. They also exclude the database migration, so the clone keeps the schema of the backup's release.
```
(
set -eu
dr() { docker compose --env-file .env --env-file release-lock.env -f compose.yaml -f compose.restore.yaml "$@"; }
dr run --rm --no-deps pgbackrest-restore --stanza=hephaestus --type=time \
  --target='YYYY-MM-DD HH:MM:SS+00' --target-action=promote restore
dr up -d --wait postgres
for attempt in $(seq 1 360); do
  recovery=$(dr exec -T postgres psql -U root -d hephaestus -Atc 'SELECT pg_is_in_recovery()')
  [ "$recovery" = f ] && break
  sleep 5
done
[ "$recovery" = f ] || { printf 'PostgreSQL did not promote within 30 minutes\n' >&2; exit 1; }
dr exec -T postgres psql -U root -d hephaestus -v ON_ERROR_STOP=1 \
  < ../../deploy/chart/hephaestus/files/restore-clone-lockdown.sql
)
```
The PostgreSQL health check alone does not prove that recovery has finished. The command above waits for promotion before it changes restored data. The lockdown transaction:
- Turns on silent mode.
- Pauses feedback delivery in all workspaces.
- Disables automatic/manual reviews.
- Turns off all Hephaestus rows under each workspace's AI models, so Hephaestus answers nobody.
- Advances the rollout revision so prepared work cannot use an earlier policy.
The lockdown changes the clone only. It does not record which Hephaestus rows were on before. After lockdown ends, a workspace admin turns those rows back on under AI models.
Record the selected WAL timeline.
Record the data fingerprint and recovery point.
Record the measured restore time and application health.
Record each manual step.
Preserve broker evidence separately.
Before consumers start, choose the JetStream strategy.

Backup

A logical dump can run while the application runs. PostgreSQL supplies a consistent database snapshot. It does not coordinate external writes, JetStream, or secret rotation.

CAUTION: Rollback consistency and secrets

For a deployment rollback point, first stop application writers and key rotation as the runbook requires. Do not deploy or rotate keys until you capture the database, configuration, and release lock. Do not enable shell tracing around secrets.

Run each shell block as a unit.

Stop on any failure.

(
set -eu
umask 077
cd /opt/hephaestus/docker/self-host
dc() { docker compose --env-file .env --env-file release-lock.env "$@"; }
backup_dir=$(mktemp -d "/var/tmp/hephaestus-backup-$(date -u +%Y%m%dT%H%M%SZ)-XXXXXX")
dc exec -T postgres pg_dump -U root -Fc hephaestus > "$backup_dir/hephaestus.dump"
test -s "$backup_dir/hephaestus.dump"
dc exec -T postgres pg_restore --list < "$backup_dir/hephaestus.dump" >/dev/null
cp .env release-lock.env "$backup_dir/"
(cd "$backup_dir" && sha256sum hephaestus.dump .env release-lock.env > SHA256SUMS)
printf 'Backup directory: %s\n' "$backup_dir"
)

Include operator-owned overrides.
Include each active/prior credential key version the snapshot needs, as credential key rotation describes.
Ship the complete directory off the host, encrypted.
Verify its checksums there.
Perform an isolated restore.
Set schedule, retention, and freshness alerts against your recovery objectives.
After verification and any active recovery, remove plaintext copies from the private /var/tmp directory according to retention.

Checksums detect corruption, not authenticity. Use a trusted backup destination. pg_restore --list reads the archive inventory, not each data block. It does not prove recoverability. The private /var/tmp directory is temporary staging, not the backup repository.

Restore

CAUTION: Restore replaces data

The commands below replace the named database. Do not run them until you confirm the instance, backup, and permitted data loss. If investigation requires the failed database, preserve it separately. Keep credential rotation disabled.

Before any application startup, complete the runbook pre-boot isolation.
On a fresh host, prepare the supported prerequisites.
Check out the backup's release, not the install guide's latest release.
Verify its published signed lock.
Before PostgreSQL starts, restore the matching .env and operator overrides.
Preserve the general encryption key.
Preserve all active/prior credential keys and version numbers that this snapshot needs.
Apply isolation settings to the restored configuration.
Set BACKUP_DIR to the absolute path printed by the backup command, or its restored off-host copy.

Run this block:

(
set -eu
cd /opt/hephaestus/docker/self-host
BACKUP_DIR=/absolute/path/to/backup-directory
dc() { docker compose --env-file .env --env-file release-lock.env "$@"; }
(cd "$BACKUP_DIR" && sha256sum -c SHA256SUMS)
cmp "$BACKUP_DIR/release-lock.env" release-lock.env
test -s "$BACKUP_DIR/hephaestus.dump"
dc stop application-server webhook-server
dc up -d --wait --wait-timeout 120 postgres
dc exec -T postgres pg_restore --list < "$BACKUP_DIR/hephaestus.dump" >/dev/null
dc exec -T postgres dropdb -U root --if-exists hephaestus
dc exec -T postgres createdb -U root hephaestus
dc exec -T postgres pg_restore -U root -d hephaestus --no-owner --no-acl --single-transaction \
  < "$BACKUP_DIR/hephaestus.dump"
)

Complete broker recovery.
Complete Prove silence in the silent checklist.
Complete Inspect and decide.
Complete Turn off silent mode deliberately before restoring normal access.
Verify sign-in.
Verify workspace relationships and activity up to the recovery timestamp.

Before restarting services

A restore reverses account suspensions, session revocations, recipient consent, and deletions as well as application data. Sessions issued after the snapshot must sign in again. Sessions revoked after it can become valid again. Ingest deduplication cannot prevent external duplicates after PostgreSQL loses delivery history. Scheduled sync cannot recover all events in the gap.

Before access or delivery reopens, reapply post-backup security/privacy decisions from the restricted incident record.
Do not erase signing keys as a generic restore step.
Diagnose missing encryption keys separately from corrupt signing data.
Before consumers start, follow the runbook broker recovery procedure.
Restore with the same signed release lock that the backup used.
Then upgrade.

Do not restore into an older app version.

Unsupported PostgreSQL versions

Only PostgreSQL 18 is supported.

For an existing PostgreSQL 17 installation:

Follow the v0.77.4 PostgreSQL 17 to 18 procedure. Do not attach an older data directory to PostgreSQL 18. Do not change the image tag as an upgrade.
Keep an encrypted, off-host backup.
Before removing a data volume, successfully rehearse its restore.

pg_restore --list checks the archive contents, not recoverability.

What to back up​

Automated off-host backup (self-host Compose)​

Backup​

Restore​

Before restarting services​

Unsupported PostgreSQL versions​