Rotate integration credential keys

This procedure rotates credentials stored on integration connections. It does not rotate HEPHAESTUS_SECURITY_ENCRYPTION_KEY, which protects the values that What the database seals lists. On the Helm chart, Secret custody shows where each setting of this procedure is.

Before you begin

1. Back up the database and both credential keys.

2. Generate a new 32-character credential key.

3. Store it in the deployment secret store.

4. Choose a positive key version greater than the current version.

5. Record the initial state on the writer database:

   SELECT credentials_key_version, count(*)
   FROM connection
   WHERE credentials_encrypted IS NOT NULL
   GROUP BY credentials_key_version
   ORDER BY credentials_key_version;

6. If the result contains an unexpected version, stop.

Rotate

1. Configure every runtime with the existing key as HEPHAESTUS_SECURITY_CREDENTIAL_ENCRYPTION_KEY.

2. Configure its version as HEPHAESTUS_SECURITY_CREDENTIAL_ENCRYPTION_KEY_VERSION.

3. Keep HEPHAESTUS_SECURITY_CREDENTIAL_ROTATION_ENABLED=false.

4. Wait until every instance from the previous release has stopped.

5. Deploy every runtime with the new key and version as active. Use HEPHAESTUS_SECURITY_PRIOR_CREDENTIAL_ENCRYPTION_KEY and HEPHAESTUS_SECURITY_PRIOR_CREDENTIAL_ENCRYPTION_KEY_VERSION for the old values. Keep rotation disabled.

6. Run an authenticated provider request for each configured integration kind.

7. If a credential cannot be decrypted, restore the previous configuration.

8. Set HEPHAESTUS_SECURITY_CREDENTIAL_ROTATION_ENABLED=true on the server runtime only.

9. If application errors, database latency, replication lag, or WAL growth leave their normal range, disable rotation.

10. Poll this verification query. Replace <active version> with the configured active version.

    SELECT
        count(*) FILTER (
            WHERE credentials_rotation_failed_at IS NULL
              AND credentials_key_version IS DISTINCT FROM <active version>
        ) AS pending,
        count(*) FILTER (WHERE credentials_rotation_failed_at IS NOT NULL) AS undecryptable
    FROM connection
    WHERE credentials_encrypted IS NOT NULL;

    Rotation is complete when pending is zero. Do not remove the prior key while undecryptable is nonzero unless you replaced every affected integration credential.

11. Disable rotation.

12. Repeat the provider checks.

13. Deploy every runtime without the prior key and version.

Failure recovery

Before rotation starts, restore the previous configuration. After a row is re-encrypted, retain both keys and roll forward. The rotation job quarantines a row it cannot decrypt. It continues with later rows.

The job logs the connection, workspace, and integration kind. It increments integration_credential_rotation_failures_total for alerts.

1. List quarantined rows without their ciphertext:

   SELECT id, workspace_id, kind, config ->> 'type' AS mode, credentials_key_version,
          credentials_rotation_failed_at
   FROM connection
   WHERE credentials_rotation_failed_at IS NOT NULL
     AND credentials_encrypted IS NOT NULL
   ORDER BY credentials_rotation_failed_at, id;

The list includes every quarantined row, even if the connection does not use its stored credential. mode distinguishes these cases. credentials_rotation_failed_at holds the time when Hephaestus first found the credential unreadable. Hephaestus resets this field when it can read the credential again or rewrites it. Thus, each listed value describes the row only until a rewrite occurs.

If a key version has no configured key, rotation stops instead of quarantine. The job fails the batch with an error that names the version. It marks no row whose ciphertext is well-formed. It resumes automatically after you configure the key. Meanwhile, pending stops decreasing and undecryptable stays unchanged.

1. Find rows for the affected version:

   SELECT id, workspace_id, kind FROM connection WHERE credentials_key_version = <version>

Hephaestus cannot distinguish a configured but incorrect key from corrupt ciphertext. It quarantines every row on which it uses that key. An incorrect prior key quarantines each row that the batch reaches. These rows appear in the list above. The update below clears them after you correct the key.

1. Determine whether the prior key or version was configured incorrectly.

2. Restore the key that can decrypt the affected rows.

3. Disable rotation.

4. Clear only the quarantine marker for those rows:

   UPDATE connection
   SET credentials_rotation_failed_at = NULL
   WHERE id IN (<verified connection ids>);

5. Re-enable rotation only after the update succeeds.

If the ciphertext is corrupt or its key is permanently unavailable, replace the credential. Disconnect Slack and Disconnect Outline are not credential-replacement actions. Disconnect Outline erases Outline documents, collection registrations, and document events. Disconnect Slack erases messages, channels, threads, and associated consent records. Do not repeatedly clear the marker without a corrected credential. This creates a retry loop and alert noise.

1. For Slack, open Administration → Integrations → Slack.
2. For Slack, select Reconnect Slack.
3. For Slack, select the workspace that this connection already uses. Hephaestus replaces the bot token on the existing connection. It keeps channels, their consent, messages already ingested, and retention settings.
4. For Outline, keep the connection.
5. For Outline, replace the token through the connection API in Integrations.
6. For a GitHub or GitLab personal access token, open Administration → Integrations.
7. For a personal access token, select GitHub or GitLab.
8. For a personal access token, use Personal access token → Replace token. This keeps monitored repositories and synced work. Workspace admins can also use PATCH /api/workspaces/{workspaceSlug}/token. A replacement clears the marker.
9. Confirm that the provider request succeeds.
10. Rerun the verification query before you remove the prior key.

A quarantined GitHub App row

A row whose mode is GITHUB_APP does not use its blob. It has no credential to replace. If you clear the blob, the row leaves rotation until the app connects again through the connect flow. That flow writes a new blob.

A reconnect rewrites the blob and clears the marker. The statement below matches the key version and quarantine timestamp from the list. If a reconnect occurred after that list, the statement leaves the row unchanged. Without these conditions, it would clear the credential that the reconnect wrote. The version increase prevents a concurrently loaded row from restoring the old blob.

1. Copy the key version and quarantine timestamp from the list into the statement:

   UPDATE connection
   SET credentials_encrypted = NULL,
       credentials_alg = NULL,
       credentials_key_version = NULL,
       credentials_rotation_failed_at = NULL,
       version = version + 1,
       updated_at = now()
   WHERE id = <app connection id>
     AND config ->> 'type' = 'GITHUB_APP'
     AND credentials_key_version = <credentials_key_version from the listing>
     AND credentials_rotation_failed_at = '<credentials_rotation_failed_at from the listing>'
   RETURNING id, workspace_id;

2. If the statement returns no row, list the connection again. The connection changed after the previous list.

3. If the statement returns no row, read the new row before you decide whether to clear it.

A credential the running key cannot read

Outside rotation, Hephaestus attempts to record the same marker when a request needs a credential that the configured keys cannot decrypt. Later reads repeat the attempt. This can occur after a key change or a host setup with a new key over an existing database.

The request returns 409 Conflict instead of a bare server error. Its problem detail names the connection and states what clears the marker. The workspace's integration pages show a notice that the stored token cannot be read. A GitHub App connection uses its installation and the app's own key, not its stored blob. Thus, Hephaestus never reports it this way.

There are two recovery paths:

1. If the key changed by mistake, restore the key that encrypted the credential. The next read succeeds and attempts to clear the marker. Later reads also repeat this attempt.
2. Alternatively, replace the credential through the paths above. A new credential clears the marker.

A credential encrypted under a key version with no configured key is a different case. Hephaestus refuses the read as a configuration fault. If the ciphertext is well-formed, it does not mark the row. The solution is to configure that key, not to change the row.