# Secret custody and rotation This page applies to the [Helm chart](/admin/kubernetes.md). It lists each secret value that the chart reads. For each value, it states what you lose without it, where to keep a copy, and how to replace it. ## How the chart reads secrets The chart reads secret values from two existing Secrets in the release namespace: * The Secret in `secret.name`, as [Create the Secret](/admin/kubernetes.md#create-the-secret) describes. * The Secret in `postgresql.tls.existingSecret`, as [PostgreSQL TLS Secret](/admin/kubernetes.md#postgresql-tls-secret) describes. The chart gives each key of the first Secret to its Pods as an environment variable. A process reads its environment only when it starts. The chart does not read the content of a Secret, so a changed Secret does not roll any Pod. After you change a key, restart each role that reads it, as [Restart a role](#restart-a-role) shows. A changed chart value, for example in `config.credentialKey`, rolls each role that reads it at the next `helm upgrade`. ## What the database seals This section applies to every deployment, also to Compose. A database backup contains values that two keys seal. It does not contain the keys. `HEPHAESTUS_SECURITY_ENCRYPTION_KEY`, the system key, seals these values: * The JWT signing keys of Hephaestus. Each session token carries their signature. * The client secrets of the login providers. * The API keys of the AI provider connections of the instance and of each workspace. `HEPHAESTUS_SECURITY_CREDENTIAL_ENCRYPTION_KEY`, the credential key, seals the stored credentials of integration connections. These are GitHub and GitLab tokens, Slack bot tokens, and Outline keys. Each sealed credential records the version of the key that sealed it. A GitHub App connection uses the app's private key, not a stored credential. ## Inventory **Outside the cluster** tells you where to keep a copy: * **Required**: a restore onto new storage loses data without this value. * **Recommended**: without this value, a part of the instance stops until you repair it at a provider or in PostgreSQL. * **No**: you can make a new value at any time. ### Custody | Key | Needed | Read by | Protects | If you lose it | Outside the cluster | | ----------------------------------------------------------------------------------------- | --------------------------- | ------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | | `HEPHAESTUS_SECURITY_ENCRYPTION_KEY` | Yes | Server, worker | The values in [What the database seals](#what-the-database-seals) | Sign-in stops, as [System key](#system-key) describes. | Required | | `HEPHAESTUS_SECURITY_CREDENTIAL_ENCRYPTION_KEY` | Yes | Server, worker | The stored integration credentials | Each workspace admin must replace each stored credential. | Required, with its version | | `HEPHAESTUS_SECURITY_PRIOR_CREDENTIAL_ENCRYPTION_KEY` | During a rotation | Server, worker | Credentials that the prior key sealed | Credentials that it sealed become unreadable. | Required, with its version, while a row or a kept backup needs it | | `HEPHAESTUS_AUTH_STATE_COOKIE_KEY` | Yes | Server | The short-lived cookies of a sign-in in progress | Sign-ins in progress fail. | No | | `WEBHOOK_SECRET` | Yes | Server, webhook | GitHub webhook signatures, the GitLab hooks that you created, and integration connections in progress | Set a new value here and at each provider. | Recommended | | `WEBHOOK_ROUTING_SECRET`, `WEBHOOK_ROUTING_PREVIOUS_SECRET` | The first one | Server, webhook | The token of each GitLab group hook and the signing secret of each Outline subscription that Hephaestus registers | The receiver refuses each registered GitLab group hook and Outline subscription, as [Routing secret](#routing-secret) describes. | Recommended | | `HEPHAESTUS_INTEGRATION_SLACK_SIGNING_SECRET` | With Slack | Server, webhook | Slack event and interactivity requests | Copy it again from the Slack app. | No | | `HEPHAESTUS_INTEGRATION_SLACK_CLIENT_SECRET` | With Slack | Server | Slack installs, and the first Slack login provider | Make a new secret in the Slack app. | No | | `GH_APP_PRIVATE_KEY` | With a GitHub App | Server, worker | GitHub App access | Make a new key in the GitHub App. | No | | `GH_APP_CLIENT_SECRET` | With a GitHub App | Server | GitHub App user authorization | Make a new secret in the GitHub App. | No | | `GH_AUTH_TOKEN` | No | Server, worker | GitHub API reads of the contributor list | Make a new token. | No | | `GITHUB_OAUTH_CLIENT_SECRET`, `GITLAB_OAUTH_CLIENT_SECRET`, `OUTLINE_OAUTH_CLIENT_SECRET` | No | Server, at the first start | The first login provider of each kind | Make a new secret at the provider. | No | | `HEPHAESTUS_AUTH_BOOTSTRAP_TOKEN` | No | Server | The recovery of an instance without an admin | Nothing. | No | | `SPRING_MAIL_PASSWORD` | No | Server | The SMTP login | Set a new password at the mail provider. | No | | `DATABASE_USERNAME`, `DATABASE_PASSWORD` | Yes | Server, worker, migration Job, bundled PostgreSQL | The database login | Set a new password in PostgreSQL. You need the user name for this. | Recommended | | `NATS_USERNAME`, `NATS_PASSWORD` | Yes | Server, webhook, bundled NATS | The broker login | Set new values. | No | | `BACKUP_S3_KEY`, `BACKUP_S3_KEY_SECRET` | With backups | Bundled PostgreSQL, backup Jobs | Access to the backup bucket | Make a new identity for the bucket. The backups stay readable. | No | | `BACKUP_CIPHER_PASS` | With backups | Bundled PostgreSQL, backup Jobs | The encryption of the backup repository | Each backup in the repository is unusable. | Required | | `ca.crt` | Yes | Server, worker, migration Job, PostgreSQL, backup Jobs | The check of the PostgreSQL server certificate | Make a new CA and new certificates. | No | | `tls.crt`, `tls.key` | With the bundled PostgreSQL | Bundled PostgreSQL | The PostgreSQL server and the pgBackRest TLS server | Make a new certificate. | No | | `tls.crt`, `tls.key` in `postgresql.backup.clientTlsSecret` | With backups | Backup Jobs | The client certificate for `hephaestus-backup` | Renew the client certificate as [Backup client certificate](#backup-client-certificate) describes. | No | | `s3-ca.crt` | With a private S3 CA | Bundled PostgreSQL, backup Jobs | The check of the S3 endpoint certificate | Copy it again from the endpoint's CA. | No | The server certificate and S3 CA stay in the PostgreSQL TLS Secret. The backup client has a separate TLS Secret, named by `postgresql.backup.clientTlsSecret`, with `ca.crt`, `tls.crt` and `tls.key`. The chart mounts the client certificate and key as `backup.crt` and `backup.key`. Non-certificate keys are in the Secret in `secret.name`. [Production setup](/admin/production-setup.md) and the [configuration reference](/admin/configuration-reference.md) describe each setting. ### Generation and rotation | Key | Generate | Rotation | | ----------------------------------------------------------------------------------------- | --------------------------------------------- | ------------------------------------------------------- | | `HEPHAESTUS_SECURITY_ENCRYPTION_KEY` | `openssl rand -hex 16` | [Not supported](#system-key) | | `HEPHAESTUS_SECURITY_CREDENTIAL_ENCRYPTION_KEY` | `openssl rand -hex 16` | [Supported, with key versions](#credential-key) | | `HEPHAESTUS_SECURITY_PRIOR_CREDENTIAL_ENCRYPTION_KEY` | The previous credential key | [Credential key](#credential-key) | | `HEPHAESTUS_AUTH_STATE_COOKIE_KEY` | `openssl rand -base64 32` | [Supported](#other-values) | | `WEBHOOK_SECRET` | `openssl rand -hex 32` | [One value only](#webhook-secret) | | `WEBHOOK_ROUTING_SECRET`, `WEBHOOK_ROUTING_PREVIOUS_SECRET` | `openssl rand -hex 32` | [Partly supported](#routing-secret) | | `HEPHAESTUS_INTEGRATION_SLACK_SIGNING_SECRET` | The Slack app settings | [One value only](#slack-secrets) | | `HEPHAESTUS_INTEGRATION_SLACK_CLIENT_SECRET` | The Slack app settings | [Supported](#slack-secrets) | | `GH_APP_PRIVATE_KEY`, `GH_APP_CLIENT_SECRET` | The GitHub App settings | [Supported, without a gap](#github-app-keys) | | `GH_AUTH_TOKEN` | A GitHub token | [Supported](#other-values) | | `GITHUB_OAUTH_CLIENT_SECRET`, `GITLAB_OAUTH_CLIENT_SECRET`, `OUTLINE_OAUTH_CLIENT_SECRET` | The OAuth application at the provider | [In Instance admin](#login-provider-secrets) | | `HEPHAESTUS_AUTH_BOOTSTRAP_TOKEN` | `openssl rand -hex 32` | [Remove after use](#other-values) | | `SPRING_MAIL_PASSWORD` | The mail provider | [Supported](#other-values) | | `DATABASE_USERNAME`, `DATABASE_PASSWORD` | `openssl rand -hex 16` for the password | [Supported](#database-password) | | `NATS_USERNAME`, `NATS_PASSWORD` | `printf 'heph%s\n' "$(openssl rand -hex 16)"` | [Supported, with a gap](#nats-credentials) | | `BACKUP_S3_KEY`, `BACKUP_S3_KEY_SECRET` | The storage provider | [Supported](#backup-bucket-credentials) | | `BACKUP_CIPHER_PASS` | `openssl rand -base64 48` | [Only with a new repository](#backup-cipher-passphrase) | | `ca.crt`, `tls.crt`, `tls.key` | Your CA | [TLS certificates](#tls-certificates) | | `s3-ca.crt` | The CA of the endpoint | [TLS certificates](#tls-certificates) | Generate each value separately, and do not use one value for two keys. [GitLab rollout bundle](/admin/production-setup.md#gitlab-rollout-bundle) states which webhook values must differ. ## Keep the required keys outside the cluster A total loss of the cluster leaves you with the backup repository and the copies that you kept elsewhere. A restore onto empty storage then needs every key that the [inventory](#inventory) marks **Required**. 1. Keep one encrypted copy of these values for each instance: * `HEPHAESTUS_SECURITY_ENCRYPTION_KEY`. * `HEPHAESTUS_SECURITY_CREDENTIAL_ENCRYPTION_KEY` and its `config.credentialKey.version`. * Each prior credential key and its version that a kept backup needs. * `BACKUP_CIPHER_PASS`. 2. Keep your values file, the release version, and `DATABASE_USERNAME` with these keys. 3. Keep the copy away from the backup bucket and from the bucket's credentials. 4. Give access to the copy only to the backup operators. 5. After each change of a required key, update the copy. 6. Rehearse a recovery without the cluster. Retrieve the copy, restore a backup into a test install with it, and sign in. A person who has the backup and these keys can read every value that the database seals. Thus, the keys and the backup must not share one storage location or one set of credentials. A cluster backup of the release namespace, such as an etcd snapshot, also contains both Secrets. That copy does not count as a copy outside the cluster when it shares the cluster's storage. A restored database keeps the PostgreSQL role of the backup. Thus, `DATABASE_USERNAME` in the new Secret must name that role. A reset of the password with [Database password](#database-password) also needs that role. ## Restart a role Each command uses the object names of the chart, `-`. [Install](/admin/kubernetes.md#install) gives the rule for a long release name. | Role | Command | Effect | | ---------- | ------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------- | | Server | `kubectl --namespace rollout restart deployment/-server` | The server stops before its new Pod starts. The API and sign-in are unavailable until the new Pod is ready. | | Webhook | `kubectl --namespace rollout restart deployment/-webhook` | A new Pod starts before an old Pod stops. | | Worker | `kubectl --namespace rollout restart deployment/-worker` | New workers start first. Each old worker drains for up to `worker.drainTimeoutSeconds`. | | PostgreSQL | `kubectl --namespace rollout restart statefulset/-postgresql` | The database is unavailable until it starts again. | | NATS | `kubectl --namespace rollout restart statefulset/-nats` | The receiver cannot store deliveries until the broker starts again. | The webapp reads no secret. ## Rotate a value ### System key This release cannot rotate `HEPHAESTUS_SECURITY_ENCRYPTION_KEY`. It reads one system key and has no setting for a prior key. Keep the key unchanged, and keep its exact value in off-host storage as [Keep the required keys outside the cluster](#keep-the-required-keys-outside-the-cluster) describes. If the key in the Secret is lost or changed, first restore its exact value from the off-host copy. Then restart every role, as [Restart a role](#restart-a-role) describes. Without the key, a server with the `prod` profile does not start. With a different 32-byte key, the server starts, but it cannot read these values: * The JWT signing keys. Hephaestus makes a new signing key only when the database has no active signing key. Thus, it does not replace a key that it cannot read, and sign-in stops. * The client secrets of the login providers. * The API keys of the AI provider connections. No supported procedure makes sign-in work again without the original key. Only an admin who can sign in can enter the login provider secrets and AI provider keys again. If the key leaks, no procedure replaces it. The key alone does not show a sealed value: a reader also needs the database or a backup. Keep the database and each backup away from each person who can have the key. You can revoke the login provider secrets and the AI provider keys at their providers, and enter new values. The new values are sealed with the same key. The JWT signing keys stay sealed with the leaked key. ### Credential key [Rotate integration credential keys](/admin/credential-key-rotation.md) owns the procedure. The procedure keeps each credential readable: the server seals each credential again with the new key. On the chart, its settings are in these places: | Setting | Place | | ------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------- | | `HEPHAESTUS_SECURITY_CREDENTIAL_ENCRYPTION_KEY`, `HEPHAESTUS_SECURITY_PRIOR_CREDENTIAL_ENCRYPTION_KEY` | The Secret in `secret.name` | | The active version, the prior version, rotation on or off | `config.credentialKey.version`, `config.credentialKey.priorVersion`, `config.credentialKey.rotationEnabled` | For each deployment that the procedure names, first change the Secret. Then run `helm upgrade` with the changed values without delay. The changed values roll the server, the webhook role, and the worker. A role that starts between the two steps refuses to start, because its prior key and prior version do not match. `config.credentialKey.rotationEnabled` applies to each role, but only the server role seals credentials again. ### Webhook secret Hephaestus has no setting for a previous `WEBHOOK_SECRET`. Thus, a rotation has a gap in which the receiver refuses deliveries. 1. Generate a new value. 2. Put it in the Secret as `WEBHOOK_SECRET`. 3. Restart the server and the webhook role. 4. Set the same value as the webhook secret of the GitHub App. 5. Set the same value on each GitHub webhook that you created by hand. 6. Set the same value on each GitLab hook that you created by hand on `/webhooks/gitlab`. Between step 3 and the provider change, the receiver refuses each delivery with the old signature. GitHub does not redeliver a refused delivery automatically. Redeliver these deliveries by hand, as [Failed GitHub deliveries](/admin/webhook-ingestion-operations.md#failed-github-deliveries) describes. An integration connection that a person started before step 3 fails, and the person must start it again. ### Routing secret The routing secret signs the token of each GitLab group hook and derives the signing secret of each Outline subscription. The receiver accepts the current and the previous routing secret. 1. Generate a new value. 2. Move the current value to `WEBHOOK_ROUTING_PREVIOUS_SECRET`. 3. Put the new value in `WEBHOOK_ROUTING_SECRET`. 4. Restart the server and the webhook role. Each hook that the previous value signed continues to deliver. The server signs a hook with the new value only when it registers that hook. It registers a hook when a GitLab connection becomes active. Its check every 6 hours also registers a hook again when GitLab reports it deleted or disabled. It does not register a hook again while the hook delivers, and Hephaestus does not show which hooks use the previous value. The check every 6 hours registers each Outline subscription again under the new value, and deletes the old subscription. Outline deliveries are not lost while it does. You can keep `WEBHOOK_ROUTING_PREVIOUS_SECRET` set. To retire it, delete each Hephaestus hook in its GitLab group. The next check registers a new hook with the new value. GitLab events between the deletion and the new hook do not reach Hephaestus. If the routing secret is lost, the receiver refuses each registered hook and Outline subscription. The next check registers each Outline subscription again under the new value. The server does not register a hook again while GitLab keeps it under a key that the server does not know. After GitLab disables the hook for failed deliveries, the next check registers a new hook. You can also delete the hook in GitLab to start this sooner. ### Slack secrets Slack signs each request with one signing secret, and Hephaestus has no setting for a previous one. 1. Make a new signing secret in the Slack app settings. 2. Put it in the Secret as `HEPHAESTUS_INTEGRATION_SLACK_SIGNING_SECRET`. 3. Restart the server and the webhook role. After Slack starts to use the new secret, the receiver refuses Slack requests until step 3. The client secret has two consumers: Slack installs read the Secret, and Slack sign-in reads the login provider. 1. Make a new client secret in the Slack app settings. 2. Put it in the Secret as `HEPHAESTUS_INTEGRATION_SLACK_CLIENT_SECRET`. 3. Restart the server. 4. Enter the same value on the Slack login provider, as [Login provider secrets](#login-provider-secrets) describes. ### GitHub App keys A GitHub App can have more than one private key and more than one client secret. Thus, these rotations have no gap. 1. Make a new private key in the GitHub App settings. 2. Put its PEM content in the Secret as `GH_APP_PRIVATE_KEY`. 3. Restart the server and the worker. 4. Make sure that a GitHub App workspace still syncs. 5. Delete the old private key in the GitHub App settings. For the client secret, do the same steps with `GH_APP_CLIENT_SECRET`, and restart only the server. ### Login provider secrets At its first start, the server copies `GITHUB_OAUTH_CLIENT_SECRET`, `GITLAB_OAUTH_CLIENT_SECRET`, `OUTLINE_OAUTH_CLIENT_SECRET`, and the Slack client secret into the database. It copies a value only when the login provider does not exist yet. After that, a changed Secret has no effect on sign-in. 1. Make a new client secret in the OAuth application at the provider. 2. Open **Instance admin → Login providers**. 3. Enter the new client secret on the provider. 4. Put the same value in the Secret, so that a new installation gets the current value. No restart is necessary. ### Database password 1. Generate a new password. 2. For the bundled PostgreSQL, open `psql` in its container: ``` kubectl --namespace exec -it statefulset/-postgresql -c postgresql -- \ sh -c 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB"' ``` 3. Enter `\password` and the new password. For an external PostgreSQL, change the password on that server instead. 4. Put the new password in the Secret as `DATABASE_PASSWORD`. 5. Restart the server, the webhook role, and the worker. Open connections stay open after step 3. Until its restart, a role cannot open a new connection. The migration Job of the next revision reads the new value. The bundled PostgreSQL reads `DATABASE_PASSWORD` only at its first start on an empty volume. [Restore](/admin/kubernetes.md#restore) states which password a restored database has. ### NATS credentials The broker accepts one user name and password. 1. Generate new values. 2. Put them in the Secret as `NATS_USERNAME` and `NATS_PASSWORD`. 3. Restart NATS. 4. Restart the server and the webhook role. For an external broker, change the login on that broker in step 3. Until step 4 completes, the receiver cannot store deliveries. [Webhook ingestion operations](/admin/webhook-ingestion-operations.md) states which providers send them again. The JetStream data stays on its volume. ### Backup bucket credentials 1. Make a new identity with read, write, and delete rights in the backup bucket. 2. Put its key in the Secret as `BACKUP_S3_KEY` and `BACKUP_S3_KEY_SECRET`. 3. Restart PostgreSQL. 4. Make sure that the next backup Job completes. 5. Delete the old identity. PostgreSQL reads the bucket credentials only when it starts. Until step 3, it archives the WAL with the old identity. Each backup Job reads the credentials when it starts. A new identity does not change the backups, so they stay readable. ### Backup cipher passphrase pgBackRest cannot change the encryption settings of a repository after the stanza exists. The chart uses one repository, at the path `/hephaestus` in the bucket of `postgresql.backup.s3.bucket`. Thus, a new passphrase needs a new bucket. 1. Prepare a new bucket and its identity, as [Backup and restore](/admin/kubernetes.md#backup-and-restore) describes. 2. Generate a new passphrase. 3. Keep the old passphrase, the old bucket name, and the old identity outside the cluster. 4. Put the new values in the Secret as `BACKUP_S3_KEY`, `BACKUP_S3_KEY_SECRET`, and `BACKUP_CIPHER_PASS`. 5. Set `postgresql.backup.s3.bucket` to the new bucket, and run `helm upgrade` without delay. 6. Make sure that the stanza Job of the revision completed. 7. Start a full backup: ``` kubectl --namespace create job -postgresql-backup-full-now \ --from=cronjob/-postgresql-backup-full ``` The bucket change restarts PostgreSQL, and the stanza Job creates the stanza in the new bucket. The new repository has no backup until the full backup of step 7 completes. The old backups stay in the old bucket, and only the old passphrase opens them. The restore Job reads only the repository that the values name. Delete the old bucket only after you no longer need its backups. ### TLS certificates The chart creates no certificate, so you must watch the expiry dates yourself. A renewal under the same CA needs only a restart of PostgreSQL: 1. Update `tls.crt` and `tls.key` in the PostgreSQL TLS Secret. 2. Restart PostgreSQL. The kubelet updates the mounted files after a delay. PostgreSQL and the pgBackRest TLS server read their certificate and key only when they start. The roles connect again when PostgreSQL is ready. For the separate backup client Secret, use [Backup client certificate](#backup-client-certificate). Each backup Job reads `backup.crt`, `backup.key`, and `s3-ca.crt` when it starts. PostgreSQL reads `s3-ca.crt` when it starts, so restart PostgreSQL after you change it. A new CA in `ca.crt` changes what each role accepts. Change it only with the [database maintenance](/admin/kubernetes.md#database-maintenance) procedure. #### Backup client certificate Renew this certificate before it expires: 1. Issue a new client certificate for the common name `hephaestus-backup` under the same PostgreSQL CA, with the `client auth` usage. 2. Replace `tls.crt` and `tls.key` together in the Secret named by `postgresql.backup.clientTlsSecret`. Keep `ca.crt` unchanged. For cert-manager certificates, verify that renewal updated this Secret and that the Certificate is Ready. 3. Wait for active backup Jobs to finish. Start a new backup Job and verify that it authenticates and completes. New Jobs read the new certificate. PostgreSQL needs no restart for a client renewal under the same CA. Use the database maintenance procedure for a CA change. ### Other values | Key | Procedure | Effect | | ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------- | | `HEPHAESTUS_AUTH_STATE_COOKIE_KEY` | Put a new value in the Secret. Restart the server. | Sign-ins in progress fail, and the people must sign in again. | | `GH_AUTH_TOKEN` | Put a new token in the Secret. Restart the server and the worker. Revoke the old token. | None. | | `SPRING_MAIL_PASSWORD` | Change the password at the mail provider. Put it in the Secret. Restart the server. | Email fails between the change and the restart. | | `HEPHAESTUS_AUTH_BOOTSTRAP_TOKEN` | Set it only for a recovery, and remove it after the recovery, as [Instance admin](/admin/instance-admin.md#bootstrap-and-recovery) describes. | None. |