Secret custody and rotation

This page applies to the Helm chart. It lists each secret value that the chart reads. For each value, it states what you lose without it, where to keep a copy, and how to replace it.

How the chart reads secrets

The chart reads secret values from two existing Secrets in the release namespace:

• The Secret in secret.name, as Create the Secret describes.
• The Secret in postgresql.tls.existingSecret, as PostgreSQL TLS Secret describes.

The chart gives each key of the first Secret to its Pods as an environment variable. A process reads its environment only when it starts. The chart does not read the content of a Secret, so a changed Secret does not roll any Pod. After you change a key, restart each role that reads it, as Restart a role shows. A changed chart value, for example in config.credentialKey, rolls each role that reads it at the next helm upgrade.

What the database seals

This section applies to every deployment, also to Compose. A database backup contains values that two keys seal. It does not contain the keys.

HEPHAESTUS_SECURITY_ENCRYPTION_KEY, the system key, seals these values:

• The JWT signing keys of Hephaestus. Each session token carries their signature.
• The client secrets of the login providers.
• The API keys of the AI provider connections of the instance and of each workspace.

HEPHAESTUS_SECURITY_CREDENTIAL_ENCRYPTION_KEY, the credential key, seals the stored credentials of integration connections. These are GitHub and GitLab tokens, Slack bot tokens, and Outline keys. Each sealed credential records the version of the key that sealed it. A GitHub App connection uses the app's private key, not a stored credential.

Inventory

Outside the cluster tells you where to keep a copy:

• Required: a restore onto new storage loses data without this value.
• Recommended: without this value, a part of the instance stops until you repair it at a provider or in PostgreSQL.
• No: you can make a new value at any time.

Custody

Key	Needed	Read by	Protects	If you lose it	Outside the cluster
HEPHAESTUS_SECURITY_ENCRYPTION_KEY	Yes	Server, worker	The values in What the database seals	Sign-in stops, as System key describes.	Required
HEPHAESTUS_SECURITY_CREDENTIAL_ENCRYPTION_KEY	Yes	Server, worker	The stored integration credentials	Each workspace admin must replace each stored credential.	Required, with its version
HEPHAESTUS_SECURITY_PRIOR_CREDENTIAL_ENCRYPTION_KEY	During a rotation	Server, worker	Credentials that the prior key sealed	Credentials that it sealed become unreadable.	Required, with its version, while a row or a kept backup needs it
HEPHAESTUS_AUTH_STATE_COOKIE_KEY	Yes	Server	The short-lived cookies of a sign-in in progress	Sign-ins in progress fail.	No
WEBHOOK_SECRET	Yes	Server, webhook	GitHub webhook signatures, the GitLab hooks that you created, and integration connections in progress	Set a new value here and at each provider.	Recommended
WEBHOOK_ROUTING_SECRET, WEBHOOK_ROUTING_PREVIOUS_SECRET	The first one	Server, webhook	The token of each GitLab group hook and the signing secret of each Outline subscription that Hephaestus registers	The receiver refuses each registered GitLab group hook and Outline subscription, as Routing secret describes.	Recommended
HEPHAESTUS_INTEGRATION_SLACK_SIGNING_SECRET	With Slack	Server, webhook	Slack event and interactivity requests	Copy it again from the Slack app.	No
HEPHAESTUS_INTEGRATION_SLACK_CLIENT_SECRET	With Slack	Server	Slack installs, and the first Slack login provider	Make a new secret in the Slack app.	No
GH_APP_PRIVATE_KEY	With a GitHub App	Server, worker	GitHub App access	Make a new key in the GitHub App.	No
GH_APP_CLIENT_SECRET	With a GitHub App	Server	GitHub App user authorization	Make a new secret in the GitHub App.	No
GH_AUTH_TOKEN	No	Server, worker	GitHub API reads of the contributor list	Make a new token.	No
GITHUB_OAUTH_CLIENT_SECRET, GITLAB_OAUTH_CLIENT_SECRET, OUTLINE_OAUTH_CLIENT_SECRET	No	Server, at the first start	The first login provider of each kind	Make a new secret at the provider.	No
HEPHAESTUS_AUTH_BOOTSTRAP_TOKEN	No	Server	The recovery of an instance without an admin	Nothing.	No
SPRING_MAIL_PASSWORD	No	Server	The SMTP login	Set a new password at the mail provider.	No
DATABASE_USERNAME, DATABASE_PASSWORD	Yes	Server, worker, migration Job, bundled PostgreSQL	The database login	Set a new password in PostgreSQL. You need the user name for this.	Recommended
NATS_USERNAME, NATS_PASSWORD	Yes	Server, webhook, bundled NATS	The broker login	Set new values.	No
BACKUP_S3_KEY, BACKUP_S3_KEY_SECRET	With backups	Bundled PostgreSQL, backup Jobs	Access to the backup bucket	Make a new identity for the bucket. The backups stay readable.	No
BACKUP_CIPHER_PASS	With backups	Bundled PostgreSQL, backup Jobs	The encryption of the backup repository	Each backup in the repository is unusable.	Required
ca.crt	Yes	Server, worker, migration Job, PostgreSQL, backup Jobs	The check of the PostgreSQL server certificate	Make a new CA and new certificates.	No
tls.crt, tls.key	With the bundled PostgreSQL	Bundled PostgreSQL	The PostgreSQL server and the pgBackRest TLS server	Make a new certificate.	No
tls.crt, tls.key in postgresql.backup.clientTlsSecret	With backups	Backup Jobs	The client certificate for hephaestus-backup	Renew the client certificate as Backup client certificate describes.	No
s3-ca.crt	With a private S3 CA	Bundled PostgreSQL, backup Jobs	The check of the S3 endpoint certificate	Copy it again from the endpoint's CA.	No

The server certificate and S3 CA stay in the PostgreSQL TLS Secret. The backup client has a separate TLS Secret, named by postgresql.backup.clientTlsSecret, with ca.crt, tls.crt and tls.key. The chart mounts the client certificate and key as backup.crt and backup.key. Non-certificate keys are in the Secret in secret.name. Production setup and the configuration reference describe each setting.

Generation and rotation

Key	Generate	Rotation
HEPHAESTUS_SECURITY_ENCRYPTION_KEY	openssl rand -hex 16	Not supported
HEPHAESTUS_SECURITY_CREDENTIAL_ENCRYPTION_KEY	openssl rand -hex 16	Supported, with key versions
HEPHAESTUS_SECURITY_PRIOR_CREDENTIAL_ENCRYPTION_KEY	The previous credential key	Credential key
HEPHAESTUS_AUTH_STATE_COOKIE_KEY	openssl rand -base64 32	Supported
WEBHOOK_SECRET	openssl rand -hex 32	One value only
WEBHOOK_ROUTING_SECRET, WEBHOOK_ROUTING_PREVIOUS_SECRET	openssl rand -hex 32	Partly supported
HEPHAESTUS_INTEGRATION_SLACK_SIGNING_SECRET	The Slack app settings	One value only
HEPHAESTUS_INTEGRATION_SLACK_CLIENT_SECRET	The Slack app settings	Supported
GH_APP_PRIVATE_KEY, GH_APP_CLIENT_SECRET	The GitHub App settings	Supported, without a gap
GH_AUTH_TOKEN	A GitHub token	Supported
GITHUB_OAUTH_CLIENT_SECRET, GITLAB_OAUTH_CLIENT_SECRET, OUTLINE_OAUTH_CLIENT_SECRET	The OAuth application at the provider	In Instance admin
HEPHAESTUS_AUTH_BOOTSTRAP_TOKEN	openssl rand -hex 32	Remove after use
SPRING_MAIL_PASSWORD	The mail provider	Supported
DATABASE_USERNAME, DATABASE_PASSWORD	openssl rand -hex 16 for the password	Supported
NATS_USERNAME, NATS_PASSWORD	printf 'heph%s\n' "$(openssl rand -hex 16)"	Supported, with a gap
BACKUP_S3_KEY, BACKUP_S3_KEY_SECRET	The storage provider	Supported
BACKUP_CIPHER_PASS	openssl rand -base64 48	Only with a new repository
ca.crt, tls.crt, tls.key	Your CA	TLS certificates
s3-ca.crt	The CA of the endpoint	TLS certificates

Generate each value separately, and do not use one value for two keys. GitLab rollout bundle states which webhook values must differ.

Keep the required keys outside the cluster

A total loss of the cluster leaves you with the backup repository and the copies that you kept elsewhere. A restore onto empty storage then needs every key that the inventory marks Required.

1. Keep one encrypted copy of these values for each instance:
   • HEPHAESTUS_SECURITY_ENCRYPTION_KEY.
   • HEPHAESTUS_SECURITY_CREDENTIAL_ENCRYPTION_KEY and its config.credentialKey.version.
   • Each prior credential key and its version that a kept backup needs.
   • BACKUP_CIPHER_PASS.
2. Keep your values file, the release version, and DATABASE_USERNAME with these keys.
3. Keep the copy away from the backup bucket and from the bucket's credentials.
4. Give access to the copy only to the backup operators.
5. After each change of a required key, update the copy.
6. Rehearse a recovery without the cluster. Retrieve the copy, restore a backup into a test install with it, and sign in.

A person who has the backup and these keys can read every value that the database seals. Thus, the keys and the backup must not share one storage location or one set of credentials. A cluster backup of the release namespace, such as an etcd snapshot, also contains both Secrets. That copy does not count as a copy outside the cluster when it shares the cluster's storage.

A restored database keeps the PostgreSQL role of the backup. Thus, DATABASE_USERNAME in the new Secret must name that role. A reset of the password with Database password also needs that role.

Restart a role

Each command uses the object names of the chart, <release>-<component>. Install gives the rule for a long release name.

Role	Command	Effect
Server	kubectl --namespace <release namespace> rollout restart deployment/<release>-server	The server stops before its new Pod starts. The API and sign-in are unavailable until the new Pod is ready.
Webhook	kubectl --namespace <release namespace> rollout restart deployment/<release>-webhook	A new Pod starts before an old Pod stops.
Worker	kubectl --namespace <release namespace> rollout restart deployment/<release>-worker	New workers start first. Each old worker drains for up to worker.drainTimeoutSeconds.
PostgreSQL	kubectl --namespace <release namespace> rollout restart statefulset/<release>-postgresql	The database is unavailable until it starts again.
NATS	kubectl --namespace <release namespace> rollout restart statefulset/<release>-nats	The receiver cannot store deliveries until the broker starts again.

The webapp reads no secret.

Rotate a value

System key

This release cannot rotate HEPHAESTUS_SECURITY_ENCRYPTION_KEY. It reads one system key and has no setting for a prior key. Keep the key unchanged, and keep its exact value in off-host storage as Keep the required keys outside the cluster describes.

If the key in the Secret is lost or changed, first restore its exact value from the off-host copy. Then restart every role, as Restart a role describes. Without the key, a server with the prod profile does not start. With a different 32-byte key, the server starts, but it cannot read these values:

• The JWT signing keys. Hephaestus makes a new signing key only when the database has no active signing key. Thus, it does not replace a key that it cannot read, and sign-in stops.
• The client secrets of the login providers.
• The API keys of the AI provider connections.

No supported procedure makes sign-in work again without the original key. Only an admin who can sign in can enter the login provider secrets and AI provider keys again.

If the key leaks, no procedure replaces it. The key alone does not show a sealed value: a reader also needs the database or a backup. Keep the database and each backup away from each person who can have the key. You can revoke the login provider secrets and the AI provider keys at their providers, and enter new values. The new values are sealed with the same key. The JWT signing keys stay sealed with the leaked key.

Credential key

Rotate integration credential keys owns the procedure. The procedure keeps each credential readable: the server seals each credential again with the new key. On the chart, its settings are in these places:

Setting	Place
HEPHAESTUS_SECURITY_CREDENTIAL_ENCRYPTION_KEY, HEPHAESTUS_SECURITY_PRIOR_CREDENTIAL_ENCRYPTION_KEY	The Secret in secret.name
The active version, the prior version, rotation on or off	config.credentialKey.version, config.credentialKey.priorVersion, config.credentialKey.rotationEnabled

For each deployment that the procedure names, first change the Secret. Then run helm upgrade with the changed values without delay. The changed values roll the server, the webhook role, and the worker. A role that starts between the two steps refuses to start, because its prior key and prior version do not match. config.credentialKey.rotationEnabled applies to each role, but only the server role seals credentials again.

Webhook secret

Hephaestus has no setting for a previous WEBHOOK_SECRET. Thus, a rotation has a gap in which the receiver refuses deliveries.

1. Generate a new value.
2. Put it in the Secret as WEBHOOK_SECRET.
3. Restart the server and the webhook role.
4. Set the same value as the webhook secret of the GitHub App.
5. Set the same value on each GitHub webhook that you created by hand.
6. Set the same value on each GitLab hook that you created by hand on /webhooks/gitlab.

Between step 3 and the provider change, the receiver refuses each delivery with the old signature. GitHub does not redeliver a refused delivery automatically. Redeliver these deliveries by hand, as Failed GitHub deliveries describes. An integration connection that a person started before step 3 fails, and the person must start it again.

Routing secret

The routing secret signs the token of each GitLab group hook and derives the signing secret of each Outline subscription. The receiver accepts the current and the previous routing secret.

1. Generate a new value.
2. Move the current value to WEBHOOK_ROUTING_PREVIOUS_SECRET.
3. Put the new value in WEBHOOK_ROUTING_SECRET.
4. Restart the server and the webhook role.

Each hook that the previous value signed continues to deliver. The server signs a hook with the new value only when it registers that hook. It registers a hook when a GitLab connection becomes active. Its check every 6 hours also registers a hook again when GitLab reports it deleted or disabled. It does not register a hook again while the hook delivers, and Hephaestus does not show which hooks use the previous value.

The check every 6 hours registers each Outline subscription again under the new value, and deletes the old subscription. Outline deliveries are not lost while it does.

You can keep WEBHOOK_ROUTING_PREVIOUS_SECRET set. To retire it, delete each Hephaestus hook in its GitLab group. The next check registers a new hook with the new value. GitLab events between the deletion and the new hook do not reach Hephaestus.

If the routing secret is lost, the receiver refuses each registered hook and Outline subscription. The next check registers each Outline subscription again under the new value. The server does not register a hook again while GitLab keeps it under a key that the server does not know. After GitLab disables the hook for failed deliveries, the next check registers a new hook. You can also delete the hook in GitLab to start this sooner.

Slack secrets

Slack signs each request with one signing secret, and Hephaestus has no setting for a previous one.

1. Make a new signing secret in the Slack app settings.
2. Put it in the Secret as HEPHAESTUS_INTEGRATION_SLACK_SIGNING_SECRET.
3. Restart the server and the webhook role.

After Slack starts to use the new secret, the receiver refuses Slack requests until step 3.

The client secret has two consumers: Slack installs read the Secret, and Slack sign-in reads the login provider.

1. Make a new client secret in the Slack app settings.
2. Put it in the Secret as HEPHAESTUS_INTEGRATION_SLACK_CLIENT_SECRET.
3. Restart the server.
4. Enter the same value on the Slack login provider, as Login provider secrets describes.

GitHub App keys

A GitHub App can have more than one private key and more than one client secret. Thus, these rotations have no gap.

1. Make a new private key in the GitHub App settings.
2. Put its PEM content in the Secret as GH_APP_PRIVATE_KEY.
3. Restart the server and the worker.
4. Make sure that a GitHub App workspace still syncs.
5. Delete the old private key in the GitHub App settings.

For the client secret, do the same steps with GH_APP_CLIENT_SECRET, and restart only the server.

Login provider secrets

At its first start, the server copies GITHUB_OAUTH_CLIENT_SECRET, GITLAB_OAUTH_CLIENT_SECRET, OUTLINE_OAUTH_CLIENT_SECRET, and the Slack client secret into the database. It copies a value only when the login provider does not exist yet. After that, a changed Secret has no effect on sign-in.

1. Make a new client secret in the OAuth application at the provider.
2. Open Instance admin → Login providers.
3. Enter the new client secret on the provider.
4. Put the same value in the Secret, so that a new installation gets the current value.

No restart is necessary.

Database password

1. Generate a new password.

2. For the bundled PostgreSQL, open psql in its container:

   kubectl --namespace <release namespace> exec -it statefulset/<release>-postgresql -c postgresql -- \
     sh -c 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB"'

3. Enter \password and the new password. For an external PostgreSQL, change the password on that server instead.

4. Put the new password in the Secret as DATABASE_PASSWORD.

5. Restart the server, the webhook role, and the worker.

Open connections stay open after step 3. Until its restart, a role cannot open a new connection. The migration Job of the next revision reads the new value. The bundled PostgreSQL reads DATABASE_PASSWORD only at its first start on an empty volume. Restore states which password a restored database has.

NATS credentials

The broker accepts one user name and password.

1. Generate new values.
2. Put them in the Secret as NATS_USERNAME and NATS_PASSWORD.
3. Restart NATS.
4. Restart the server and the webhook role.

For an external broker, change the login on that broker in step 3. Until step 4 completes, the receiver cannot store deliveries. Webhook ingestion operations states which providers send them again. The JetStream data stays on its volume.

Backup bucket credentials

1. Make a new identity with read, write, and delete rights in the backup bucket.
2. Put its key in the Secret as BACKUP_S3_KEY and BACKUP_S3_KEY_SECRET.
3. Restart PostgreSQL.
4. Make sure that the next backup Job completes.
5. Delete the old identity.

PostgreSQL reads the bucket credentials only when it starts. Until step 3, it archives the WAL with the old identity. Each backup Job reads the credentials when it starts. A new identity does not change the backups, so they stay readable.

Backup cipher passphrase

pgBackRest cannot change the encryption settings of a repository after the stanza exists. The chart uses one repository, at the path /hephaestus in the bucket of postgresql.backup.s3.bucket. Thus, a new passphrase needs a new bucket.

1. Prepare a new bucket and its identity, as Backup and restore describes.

2. Generate a new passphrase.

3. Keep the old passphrase, the old bucket name, and the old identity outside the cluster.

4. Put the new values in the Secret as BACKUP_S3_KEY, BACKUP_S3_KEY_SECRET, and BACKUP_CIPHER_PASS.

5. Set postgresql.backup.s3.bucket to the new bucket, and run helm upgrade without delay.

6. Make sure that the stanza Job of the revision completed.

7. Start a full backup:

   kubectl --namespace <release namespace> create job <release>-postgresql-backup-full-now \
     --from=cronjob/<release>-postgresql-backup-full

The bucket change restarts PostgreSQL, and the stanza Job creates the stanza in the new bucket. The new repository has no backup until the full backup of step 7 completes. The old backups stay in the old bucket, and only the old passphrase opens them. The restore Job reads only the repository that the values name. Delete the old bucket only after you no longer need its backups.

TLS certificates

The chart creates no certificate, so you must watch the expiry dates yourself.

A renewal under the same CA needs only a restart of PostgreSQL:

1. Update tls.crt and tls.key in the PostgreSQL TLS Secret.
2. Restart PostgreSQL.

The kubelet updates the mounted files after a delay. PostgreSQL and the pgBackRest TLS server read their certificate and key only when they start. The roles connect again when PostgreSQL is ready.

For the separate backup client Secret, use Backup client certificate. Each backup Job reads backup.crt, backup.key, and s3-ca.crt when it starts. PostgreSQL reads s3-ca.crt when it starts, so restart PostgreSQL after you change it.

A new CA in ca.crt changes what each role accepts. Change it only with the database maintenance procedure.

Backup client certificate

Renew this certificate before it expires:

1. Issue a new client certificate for the common name hephaestus-backup under the same PostgreSQL CA, with the client auth usage.
2. Replace tls.crt and tls.key together in the Secret named by postgresql.backup.clientTlsSecret. Keep ca.crt unchanged. For cert-manager certificates, verify that renewal updated this Secret and that the Certificate is Ready.
3. Wait for active backup Jobs to finish. Start a new backup Job and verify that it authenticates and completes. New Jobs read the new certificate. PostgreSQL needs no restart for a client renewal under the same CA.

Use the database maintenance procedure for a CA change.

Other values

Key	Procedure	Effect
HEPHAESTUS_AUTH_STATE_COOKIE_KEY	Put a new value in the Secret. Restart the server.	Sign-ins in progress fail, and the people must sign in again.
GH_AUTH_TOKEN	Put a new token in the Secret. Restart the server and the worker. Revoke the old token.	None.
SPRING_MAIL_PASSWORD	Change the password at the mail provider. Put it in the Secret. Restart the server.	Email fails between the change and the restart.
HEPHAESTUS_AUTH_BOOTSTRAP_TOKEN	Set it only for a recovery, and remove it after the recovery, as Instance admin describes.	None.