Contain an incident
Stop external writes, model spend, reviews, conversations or the whole server, and replace leaked credentials and sessions.
Contain an incident when Hephaestus sends wrong feedback, spends without control, or loses a credential. Use the smallest switch that stops the harm. No switch recalls a write that a provider already accepted.
Before you start
- You are an instance admin.
- For the chart switches, you have your site values and Helm access to the release namespace.
- Record the instance URL, the Helm release, the incident owner, and the start time in your incident record. Keep credentials and feedback text out of the record.
Each switch stops one kind of work:
| Switch | What it stops | What continues |
|---|---|---|
| Silent mode, in Instance administration → Instance settings | New writes to GitHub, GitLab and Slack, and all email. This includes comments, edits, approvals, reactions, Hephaestus replies, digests and their retries. | Reviews, model calls, sync, feedback in the web app, sign-in, webhook registration and GitHub redelivery |
| A budget of $0, or a connection turned off | Model calls that the budget pays for or that go through the connection, also the calls of running reviews | Sync, provider writes, and other budgets and connections |
sandbox.enabled: false | Practice reviews and conversation turns. The chart sets AGENT_ENABLED from this value. | The API, sync and provider deliveries |
sandbox.conversationTurns: 0 | New conversation turns | Practice reviews |
config.slack.enabled: false | Slack deliveries, conversations in Slack, channel reading and Slack messages | GitHub, GitLab and Outline |
server.stopped: true | The server: the API, provider deliveries, reviews and every write | The web app and PostgreSQL |
For one workspace, a workspace admin can pause sending or set the feedback mode to Off, as Feedback mode, review scope and delivery describes. Silent mode does not stop model calls or spending.
Stop external writes
-
Open Instance administration → Instance settings.
-
Select Turn on silent mode….
-
Enter the incident reference as the reason.
-
Select Turn on silent mode.
The Silent mode card shows On. On-work feedback that a review sends while silent mode is on stays withheld.
Tip
A script can turn on silent mode with PATCH /api/admin/settings/silent-mode and the body {"engaged":true,"reason":"<incident reference>"}.
Call the API gives the access token.
Stop model spend
A budget stops the model calls of one workspace. The model proxy also refuses the next call of a review that already runs.
- For the shared models of a workspace, open Instance administration → AI usage. In the row of the workspace, select Set budget or Change budget, and set the budget to $0.
- For a workspace that uses its own provider, a workspace admin sets the provider cap to $0. The cap is under AI usage → Set provider cap in the workspace administration.
- To stop every model on one connection, open Instance administration → AI models. Turn off Active for the connection, and select Turn off connection.
- To stop new reviews in a workspace, a workspace admin turns off Review settings → Automated review → Start practice reviews. Reviews that already run can finish.
To stop all model calls at once, stop reviews and conversations, as the next section describes.
Stop reviews and conversations
Stop the server and its sandboxes first.
sandbox.enabled: false also removes the network policies of the sandbox namespace, so no sandbox may run when you apply it.
-
In your site values, set
server.stopped: true. -
Run
helm upgradewith your site values. -
Remove the sandboxes that still run, as step 4 of Planned node maintenance describes.
-
Wait until no server Pod and no sandbox Pod runs.
-
In your site values, set
sandbox.enabled: falseandserver.stopped: false. -
Run
helm upgradewith your site values.The server starts again and runs no review and no conversation turn.
To stop only conversations, set sandbox.conversationTurns: 0 and run helm upgrade.
To stop only Slack, set config.slack.enabled: false and run helm upgrade.
Stop the server
Stop the server when every write must stop at once, or when you cannot trust the server.
- In your site values, set
server.stopped: true. - Run
helm upgradewith your site values.
Provider deliveries fail while no server runs. After the server starts again, recover them, as Which providers deliver again describes.
Replace a leaked credential
Revoke the leaked value at its provider, then give Hephaestus a new one.
| Leaked credential | Replace it |
|---|---|
| GitHub App private key or client secret | GitHub App keys |
| Slack signing secret or client secret | Slack secrets |
WEBHOOK_SECRET | Webhook secret |
| Client secret of a login provider | Login provider secrets |
| Database password | Database passwords |
| GitHub or GitLab token of a workspace | Revoke it at the provider. A workspace admin enters a new token, as Replace a refused token describes. |
| API key of an AI provider connection | Revoke it at the provider. Enter the new key in the API key field of the connection (Rotate the API key). |
| The database, a backup, or many session cookies | End every session on the instance |
End every session on the instance
End every session after a leak of the database, the JWT signing key, or many session cookies. Every person and every installed client must then sign in again.
Run this transaction in the bundled database:
kubectl --namespace <release namespace> exec -i <release>-postgresql-0 -c postgresql -- \
sh -c 'exec psql -v ON_ERROR_STOP=1 -U "$POSTGRES_USER" -d "$POSTGRES_DB"' <<'SQL'
BEGIN;
LOCK TABLE account IN EXCLUSIVE MODE;
UPDATE client_sign_in_handoff SET consumed_at = now() WHERE consumed_at IS NULL;
UPDATE client_session SET revoked_at = now(), revoked_reason = 'ADMIN_REVOKE' WHERE revoked_at IS NULL;
UPDATE issued_jwt SET revoked_at = now(), revoked_reason = 'ADMIN_REVOKE' WHERE revoked_at IS NULL;
COMMIT;
SQLReplace <release namespace> and <release> with the namespace and the name of your Helm release.
The lock waits for sign-ins in progress, so the updates include their sessions.
The server checks each request against issued_jwt, so every session ends at once.
A leaked JWT signing key lets its holder sign new sessions. If it leaked, make a new signing key, as Signing key describes. If the system key leaked together with the database or a backup, rotate the system key, as System key describes.
Record the action in your incident record.
Check that it worked
- Open Instance administration → Instance settings. The Silent mode card shows On.
- If you set a budget of $0, open Instance administration → AI usage. The row of the workspace shows Paused.
- If you stopped reviews, run
kubectl --namespace <sandbox namespace> get pods. The command lists no Pod. - If you stopped the server, run
kubectl --namespace <release namespace> get deployment <release>-server. The Deployment shows0/0ready replicas. - If you ended every session, sign in again. The web app asks for a new sign-in.
Undo
-
Before you turn off silent mode, fix the cause and check that the next review gives correct feedback.
-
Open Instance administration → Instance settings.
-
Select Turn off silent mode….
-
Enter
turn off. -
Select Turn off silent mode.
Feedback that was withheld while silent mode was on stays withheld. The next completed review posts its feedback, in every workspace.
-
Restore the budgets, provider caps, connections and review settings that you changed.
-
In your site values, restore
sandbox.enabled,sandbox.conversationTurns,config.slack.enabledandserver.stopped. -
Run
helm upgradewith your site values.
Ended sessions cannot be restored. If the instance must restart from a backup, follow Back up and restore.