Skip to main content

Backup & Restore

For the self-host compose stack. PostgreSQL holds application state. The Context Fabric holds bounded evidence copies that may become irreproducible after upstream content changes or erasure.

What to back up​

DataWherePriority
PostgreSQL volumeCompose-managed volume mounted by postgresEverything. Accounts, workspaces, observations, feedback, encrypted provider tokens, JWT signing keys.
.env/opt/hephaestus/docker/self-host/.envEqual priority. Its encryption keys protect values inside the database backup. Without them, provider credentials and signing material are unreadable.
TLS certificates./letsencrypt/Optional — Let's Encrypt re-issues on first boot (rate limits permitting).
Context Fabricgit-repos volumeBack up when replayable evidence is required. Skipping it discards repository clones and bounded replay/CAS content; durable job provenance remains in PostgreSQL, but the exact source bytes may be unrecoverable.
NATS JetStreamnats-data volumeSkip — a transient event buffer, not a source of truth.

Backup​

Logical dump while everything runs (safe — Postgres MVCC gives a consistent snapshot):

cd /opt/hephaestus/docker/self-host
dc() { docker compose --env-file .env --env-file release-lock.env "$@"; }
dc exec -T postgres pg_dump -U root -Fc hephaestus \
  > hephaestus-$(date +%F).dump
cp .env hephaestus-$(date +%F).env

Ship both files off the host, encrypted. Cron it daily; test-restore it at least once (below).

Restore​

On a fresh host, complete install steps 1–2 first, restoring the saved .env (same POSTGRES_PASSWORD and both HEPHAESTUS_SECURITY_*ENCRYPTION_KEY values). Then:

cd /opt/hephaestus/docker/self-host
dc() { docker compose --env-file .env --env-file release-lock.env "$@"; }
dc up -d postgres
dc stop application-server application-worker webhook-server 2>/dev/null || true

test -s hephaestus-YYYY-MM-DD.dump
dc exec -T postgres pg_restore --list < hephaestus-YYYY-MM-DD.dump >/dev/null
dc exec -T postgres dropdb -U root --if-exists hephaestus
dc exec -T postgres createdb -U root hephaestus
dc exec -T postgres pg_restore -U root -d hephaestus --no-owner --no-acl --single-transaction \
  < hephaestus-YYYY-MM-DD.dump

dc up -d --wait

Verify: sign in, open a workspace, check recent activity is present up to the backup timestamp.

PostgreSQL 17 to 18​

PostgreSQL major-version data directories are not binary compatible. The PostgreSQL 18 image stores PGDATA under /var/lib/postgresql/18/docker. Do not point the PostgreSQL 18 container at the old PostgreSQL 17 volume and do not delete the old volume until the restored instance passes verification.

From the PostgreSQL 17 release, stop application writes, make a logical dump, and record the exact Compose volume name:

cd /opt/hephaestus/docker/self-host
dc() { docker compose --env-file .env --env-file release-lock.env "$@"; }
dc stop application-server application-worker webhook-server
dc exec -T postgres pg_dump -U root -Fc hephaestus > /var/tmp/hephaestus-pg17.dump
test -s /var/tmp/hephaestus-pg17.dump
dc exec -T postgres pg_restore --list < /var/tmp/hephaestus-pg17.dump >/dev/null
sha256sum /var/tmp/hephaestus-pg17.dump > /var/tmp/hephaestus-pg17.dump.sha256
PG17_VOLUME=$(docker inspect "$(dc ps -q postgres)" \
  --format '{{range .Mounts}}{{if eq .Destination "/var/lib/postgresql/data"}}{{.Name}}{{end}}{{end}}')
test -n "$PG17_VOLUME"
dc down
printf 'Preserved PostgreSQL 17 volume: %s\n' "$PG17_VOLUME"

Check out the new release and prepare its signed release lock as described in Upgrades. The supported self-host stack creates no custom roles or tablespaces; operators who added either must migrate those cluster-global objects separately. Restore into the new PostgreSQL 18 volume:

dc() { docker compose --env-file .env --env-file release-lock.env "$@"; }
dc up -d postgres
until dc exec -T postgres pg_isready -U root -d hephaestus; do sleep 2; done
test "$(dc exec -T postgres psql -U root -d hephaestus -Atc \
  "SELECT current_setting('server_version_num')::int / 10000 || ':' || (to_regclass('public.databasechangelog') IS NULL)")" = "18:true"
sha256sum -c /var/tmp/hephaestus-pg17.dump.sha256
dc exec -T postgres dropdb -U root hephaestus
dc exec -T postgres createdb -U root hephaestus
dc exec -T postgres pg_restore -U root -d hephaestus --no-owner --no-acl \
  --single-transaction < /var/tmp/hephaestus-pg17.dump
dc exec -T postgres psql -U root -d hephaestus -v ON_ERROR_STOP=1 \
  -c 'ALTER EXTENSION pg_partman UPDATE'
dc up -d --wait

Verify application health, sign in, and inspect recent workspace activity. Then verify the database major, extension, and partition policy:

dc exec -T postgres psql -U root -d hephaestus -Atc \
  "SELECT current_setting('server_version_num')::int / 10000 = 18; SELECT extversion = '5.5.0' FROM pg_extension WHERE extname = 'pg_partman'; SELECT count(*) = 1 FROM partman.part_config WHERE parent_table = 'public.auth_event';"

Keep the dump and PostgreSQL 17 volume until acceptance checks pass and backup policy permits their deletion. Rollback means stopping PostgreSQL 18 and starting the prior release against the preserved volume; never run the prior application against the PostgreSQL 18 database.

After a point-in-time restore​

  • Sessions: users signed in after the backup was taken must sign in again. If the restored jwt_signing_key table is unusable, truncate it and restart — a fresh key is auto-seeded and everyone re-logs-in. No data loss beyond sessions.
  • NATS JetStream: the stream on nats-data may hold events the restored database has already processed (or never saw). Webhook ingest deduplicates by delivery id and idempotency keys, so replays are absorbed on ingest; the end-to-end redelivery-safety drill (including whether to reset consumers after a restore) is part of #1370. The conservative option after a restore from an old backup: docker compose down, docker volume rm hephaestus_nats-data, docker compose up -d — events in the gap are re-fetched by the scheduled sync.
  • Restore into an older app version: don't. Restore with the same signed release lock the backup was taken from, then upgrade.