# Rotate integration credential keys This procedure rotates credentials stored on integration connections. It does not rotate `HEPHAESTUS_SECURITY_ENCRYPTION_KEY`, which protects the values that [What the database seals](/admin/secret-custody.md#what-the-database-seals) lists. On the Helm chart, [Secret custody](/admin/secret-custody.md#credential-key) shows where each setting of this procedure is. ## Before you begin 1. Back up the database and both credential keys. 2. Generate a new 32-character credential key. 3. Store it in the deployment secret store. 4. Choose a positive key version greater than the current version. 5. Record the initial state on the writer database: ``` SELECT credentials_key_version, count(*) FROM connection WHERE credentials_encrypted IS NOT NULL GROUP BY credentials_key_version ORDER BY credentials_key_version; ``` 6. If the result contains an unexpected version, stop. ## Rotate 1. Configure every runtime with the existing key as `HEPHAESTUS_SECURITY_CREDENTIAL_ENCRYPTION_KEY`. 2. Configure its version as `HEPHAESTUS_SECURITY_CREDENTIAL_ENCRYPTION_KEY_VERSION`. 3. Keep `HEPHAESTUS_SECURITY_CREDENTIAL_ROTATION_ENABLED=false`. 4. Wait until every instance from the previous release has stopped. 5. Deploy every runtime with the new key and version as active. Use `HEPHAESTUS_SECURITY_PRIOR_CREDENTIAL_ENCRYPTION_KEY` and `HEPHAESTUS_SECURITY_PRIOR_CREDENTIAL_ENCRYPTION_KEY_VERSION` for the old values. Keep rotation disabled. 6. Run an authenticated provider request for each configured integration kind. 7. If a credential cannot be decrypted, restore the previous configuration. 8. Set `HEPHAESTUS_SECURITY_CREDENTIAL_ROTATION_ENABLED=true` on the server runtime only. 9. If application errors, database latency, replication lag, or WAL growth leave their normal range, disable rotation. 10. Poll this verification query. Replace `` with the configured active version. ``` SELECT count(*) FILTER ( WHERE credentials_rotation_failed_at IS NULL AND credentials_key_version IS DISTINCT FROM ) AS pending, count(*) FILTER (WHERE credentials_rotation_failed_at IS NOT NULL) AS undecryptable FROM connection WHERE credentials_encrypted IS NOT NULL; ``` Rotation is complete when `pending` is zero. Do not remove the prior key while `undecryptable` is nonzero unless you replaced every affected integration credential. 11. Disable rotation. 12. Repeat the provider checks. 13. Deploy every runtime without the prior key and version. ## Failure recovery Before rotation starts, restore the previous configuration. After a row is re-encrypted, retain both keys and roll forward. The rotation job quarantines a row it cannot decrypt. It continues with later rows. The job logs the connection, workspace, and integration kind. It increments `integration_credential_rotation_failures_total` for alerts. 1. List quarantined rows without their ciphertext: ``` SELECT id, workspace_id, kind, config ->> 'type' AS mode, credentials_key_version, credentials_rotation_failed_at FROM connection WHERE credentials_rotation_failed_at IS NOT NULL AND credentials_encrypted IS NOT NULL ORDER BY credentials_rotation_failed_at, id; ``` The list includes every quarantined row, even if the connection does not use its stored credential. `mode` distinguishes these cases. `credentials_rotation_failed_at` holds the time when Hephaestus first found the credential unreadable. Hephaestus resets this field when it can read the credential again or rewrites it. Thus, each listed value describes the row only until a rewrite occurs. If a key version has no configured key, rotation stops instead of quarantine. The job fails the batch with an error that names the version. It marks no row whose ciphertext is well-formed. It resumes automatically after you configure the key. Meanwhile, `pending` stops decreasing and `undecryptable` stays unchanged. 1. Find rows for the affected version: ``` SELECT id, workspace_id, kind FROM connection WHERE credentials_key_version = ``` Hephaestus cannot distinguish a configured but incorrect key from corrupt ciphertext. It quarantines every row on which it uses that key. An incorrect prior key quarantines each row that the batch reaches. These rows appear in the list above. The update below clears them after you correct the key. 1. Determine whether the prior key or version was configured incorrectly. 2. Restore the key that can decrypt the affected rows. 3. Disable rotation. 4. Clear only the quarantine marker for those rows: ``` UPDATE connection SET credentials_rotation_failed_at = NULL WHERE id IN (); ``` 5. Re-enable rotation only after the update succeeds. If the ciphertext is corrupt or its key is permanently unavailable, replace the credential. **Disconnect Slack** and **Disconnect Outline** are not credential-replacement actions. **Disconnect Outline** erases Outline documents, collection registrations, and document events. **Disconnect Slack** erases messages, channels, threads, and associated consent records. Do not repeatedly clear the marker without a corrected credential. This creates a retry loop and alert noise. 1. For Slack, open **Administration → Integrations → Slack**. 2. For Slack, select **Reconnect Slack**. 3. For Slack, select the workspace that this connection already uses. Hephaestus replaces the bot token on the existing connection. It keeps channels, their consent, messages already ingested, and retention settings. 4. For Outline, keep the connection. 5. For Outline, replace the token through the connection API in [Integrations](/admin/integrations.md#replace-an-outline-key-without-erasing-the-mirror). 6. For a GitHub or GitLab personal access token, open **Administration → Integrations**. 7. For a personal access token, select **GitHub** or **GitLab**. 8. For a personal access token, use **Personal access token → Replace token**. This keeps monitored repositories and synced work. Workspace admins can also use `PATCH /api/workspaces/{workspaceSlug}/token`. A replacement clears the marker. 9. Confirm that the provider request succeeds. 10. Rerun the verification query before you remove the prior key. ### A quarantined GitHub App row A row whose `mode` is `GITHUB_APP` does not use its blob. It has no credential to replace. If you clear the blob, the row leaves rotation until the app connects again through the connect flow. That flow writes a new blob. A reconnect rewrites the blob and clears the marker. The statement below matches the key version and quarantine timestamp from the list. If a reconnect occurred after that list, the statement leaves the row unchanged. Without these conditions, it would clear the credential that the reconnect wrote. The version increase prevents a concurrently loaded row from restoring the old blob. 1. Copy the key version and quarantine timestamp from the list into the statement: ``` UPDATE connection SET credentials_encrypted = NULL, credentials_alg = NULL, credentials_key_version = NULL, credentials_rotation_failed_at = NULL, version = version + 1, updated_at = now() WHERE id = AND config ->> 'type' = 'GITHUB_APP' AND credentials_key_version = AND credentials_rotation_failed_at = '' RETURNING id, workspace_id; ``` 2. If the statement returns no row, list the connection again. The connection changed after the previous list. 3. If the statement returns no row, read the new row before you decide whether to clear it. ### A credential the running key cannot read Outside rotation, Hephaestus attempts to record the same marker when a request needs a credential that the configured keys cannot decrypt. Later reads repeat the attempt. This can occur after a key change or a host setup with a new key over an existing database. The request returns `409 Conflict` instead of a bare server error. Its problem detail names the connection and states what clears the marker. The workspace's integration pages show a notice that the stored token cannot be read. A GitHub App connection uses its installation and the app's own key, not its stored blob. Thus, Hephaestus never reports it this way. There are two recovery paths: 1. If the key changed by mistake, restore the key that encrypted the credential. The next read succeeds and attempts to clear the marker. Later reads also repeat this attempt. 2. Alternatively, replace the credential through the paths above. A new credential clears the marker. A credential encrypted under a key *version* with no configured key is a different case. Hephaestus refuses the read as a configuration fault. If the ciphertext is well-formed, it does not mark the row. The solution is to configure that key, not to change the row.