Production operations runbook

This runbook is for the instance operator.

Before changing state, record the instance URL, Compose project, signed release lock, incident owner, and rollback point.
Keep credentials and feedback bodies out of tickets.

CAUTION: Before broad or unattended delivery

The operator must execute and sign off three drills on isolated, representative infrastructure. The failed-migration drill is on the Upgrade page. The broker and un-delivery drills are below. An operator-owned silent run requires the silent deployment checklist first.

First response and kill-switch map

If external feedback is wrong, immediately turn on silent mode in Instance admin → Instance settings.
If the control plane is unavailable, block provider egress at the host/network boundary.
Apply the same block if a request already in progress may still write.
Under either condition, stop all application servers, workers, and active sandboxes.
To preserve a recovery point, also stop ingress.

A switch cannot recall a request that a provider already accepted.

Control and location	What it stops	What it does not stop
Silent mode, instance settings	New SCM, Slack, and email writes through the egress gateway. This includes edits, inline comments, approvals, reactions, Hephaestus replies, weekly digest notes, and retries. It covers all email from the notification module. Withheld email is final and does not send when silent mode is turned off.	Review compute, persistence, reads/sync, in-app feedback, or declared OAuth, webhook-registration, webhook-redelivery, and operator-alert exemptions
`AGENT_ENABLED=false`, deployment environment on submitting servers and claiming workers	Queued review runtime. Recreate affected containers to apply it	Already-running sandboxes, conversation turns, sync, or every external write. Not a runtime global AI stop
Workspace Review settings controls: Start from work activity, Start requested reviews	Their respective review entry points	Other workspaces, existing work, or external delivery already in progress
Workspace/group/practice Feedback mode Off	New feedback at all destinations	Eligible reviews and stored results
Workspace Sending feedback → Send feedback off	External practice feedback and Hephaestus delivery for that workspace	Measurement and in-app feedback. Not an instance restore boundary
Recipient Comments on your work off	Covered feedback comments for that recipient	All other recipients, integration plumbing, or all Hephaestus activity
Every Conversations row turned off under the workspace's AI models. `HEPHAESTUS_INTEGRATION_SLACK_ENABLED=false` on applicable roles	Respective mentor/Slack features	SCM writes, other AI work, or restored database credentials
Model/connection disabled. Each funding source's budget set to zero	Work using that model/connection or budget	Other funding sources. Already-incurred costs. Budgets are not strict reservations

Practice review defines workspace and recipient controls. There is no independent per-destination toggle matrix. For an immediate all-AI stop, stop server/worker/sandbox processes and deny model-provider egress. Silent mode does not stop compute.

Turn silent mode on, verify it, and turn it off

The authenticated instance-admin API uses GET /api/admin/settings and PATCH /api/admin/settings/silent-mode. The UI handles the preconditions below.

Turn on silent mode with {"engaged":true,"reason":"incident reference"}.
Read back silentModeEngaged=true.
Retain the audit event and current ETag.
To turn it off, use {"engaged":false,"reason":"approved release reference"} with the current If-Match ETag.
If the response is 412, read again.
After a 412, reassess turning it off.
Do not retry turning it off blindly.

428 means the required precondition is missing. Restored settings can contain silent mode off. The default does not override an explicit decision to turn silent mode off. Turning silent mode off affects all workspaces and never flushes suppressed feedback.

Silent deployment checklist

Contain. For the running instance, turn on silent mode.
For the running instance, read back silent mode.
For a clone, use the pre-boot isolation below instead.
Record exceptions. Zero content delivery is not zero network traffic.
Capture rollback. Stop ingress.
Stop application writers, including remote workers and active sandboxes.
Save a restorable database backup with matching secrets, configuration, and signed release lock.
Verify the archive.
Verify the checksum.
Verify the off-host copy.
Record the recovery point and permitted outage/data loss.
Validate then deploy. Complete the staging migration.
Prepare the tested release.
Keep AGENT_ENABLED=false on server and worker roles.
Keep Slack disabled and public ingress blocked.
From /opt/hephaestus/docker/self-host, run the startup command below.

The command starts infrastructure and the application server with its required Compose dependencies.
```
docker compose --env-file .env --env-file release-lock.env up -d --wait postgres nats-server application-server
```
Verify migration completion.
Before starting webhook-server or enabling reviews, read back silent mode.

Compose health alone does not verify the brake.
Prove silence. After readback, keep silent mode on.
Set AGENT_ENABLED=true on the submitting server and claiming worker roles.
Enable the intended review/Slack settings.
Recreate affected containers.
Keep restore-clone network isolation.
Use controlled providers for clone drills.
Run bounded reviews on controlled GitHub/GitLab work and Slack conversations for each enabled lane.
Inspect practice results and WITHHELD / INSTANCE_SILENCED outcomes.
Correlate provider-side object inventories and egress observations, including retries and edits to existing feedback.
For Outline, verify no unplanned document writes or webhook registration.

Outline is a source integration.
Inspect and decide. Inspect capture failures, withheld feedback, policy traces, queue age, and spend under small budgets.
Record known integration/readiness failures caused by isolation.
Remain silent if writes are unexplained, migrations fail, or spend is unaccounted for.
If necessary, use recovery below.
Turn off silent mode deliberately. Only after the rollout gates pass, review all workspace review scope, account consent, Hephaestus settings, and feedback mode.
After the rollout gates pass, turn silent mode off with the current ETag.
Prefer Approve first for a bounded pilot.
Prove that old suppressed work stays suppressed.
Prove that only a fresh eligible event can deliver.
If either check fails, turn silent mode on again immediately.

Upgrade and failed-migration recovery

Upgrade owns the staging rehearsal, the upgrade, and failed-migration recovery. The silent deployment checklist above owns startup.

Backup, restore and PostgreSQL × JetStream

Backup & restore owns the data inventory and logical dump commands. A dump restores one snapshot, not an arbitrary point in time. Use pgBackRest for scheduled backups and WAL archiving (backup tooling decision). Require off-host backup freshness alerts and a verified restore before unattended delivery.

The self-host Compose implementation and its schedule are in Backup & restore. The shared redacted backup status required by Readiness, Operations, and System Information does not exist yet. Those consumers must show unknown. They must not infer health from a configured timer or a passing pgbackrest verify. A passing backup is not a restore result or release gate. Application metrics do not report backup/WAL freshness or restore-rehearsal outcomes.

Until the status publisher and operator alert service exist:

Monitor failed systemd units, WAL archive lag, and disk space.
Monitor pgBackRest info/verify and the last actual drill record.
Configure host or backup-tool alerts for stale backup/WAL, failed attempts, and expired or failed restore rehearsals.

For backup and recovery:

Set explicit RPO/RTO, backup schedule, retention, owner, off-host destination, and restore-test frequency.
Monitor the last successful backup/WAL archive and restore drill.
Test loss of the primary host. On the Helm chart, retrieve the required keys without that host, as Secret custody describes.
Test backup failure alerts.
For PITR, validate the base backup.
For PITR, validate continuous WAL coverage to the selected timestamp/timeline.
Where exact replay is required, include evidence storage.
If the backup overlaps a key rotation, preserve active and prior credential-encryption keys and versions.
Disable rotation during recovery.

CAUTION: Broker snapshot consistency
Never treat a live filesystem copy as a consistent broker backup. Per-stream snapshots are not atomic with each other, PostgreSQL, or external provider writes. Preserve them for gap analysis, not automatic replay after PITR.
For incident forensics, take broker-supported snapshots or a cold copy of the stopped JetStream store.
Use an authenticated CLI on the isolated broker network.

For each stream, run the NATS backup command:

nats stream backup --check STREAM /restricted/incident/STREAM

Record start/end time, stream sequence, consumer ACK positions, CLI version, and each snapshot's checksum.
Encrypt the snapshots.
Move them off-host under the database's access and expiry policy.

Snapshots include message payloads and, by default, consumers.

Pre-boot isolation — mandatory for every restore

The supported self-host clone needs a separate host. Its Compose project name and network/subnet are fixed. A different directory or -p alone does not provide isolation. Silent mode permits model calls and control-plane traffic. A disabled Hephaestus row alone is never an isolation boundary.

Stop ingress, all server/worker replicas, and active sandboxes.
Suspend automatic restarts/reconcilers.
Use a separate host for the supported self-host clone.
Give it a separate database, NATS store, domain, and credentials.
Do not connect it to production volumes, NATS, or provider webhooks.
Before any application starts, deny external traffic from application and sandbox networks.
Allow only required isolated database/broker/admin access.
Set AGENT_ENABLED=false on server/worker roles.
Set HEPHAESTUS_INTEGRATION_SLACK_ENABLED=false on Slack-capable roles.
Withhold SCM write credentials, including app signing keys, PATs, and credentials encrypted inside the restored database.

Removal of .env seeds does not remove database-backed credentials.
Keep network denial while inspecting those credentials.
Before interactive access, turn off all conversation model rows. The restore-clone lockdown does this.
Arrange admin authentication under isolation before the drill.
If admin authentication is unavailable, stop.
Review effective Compose configuration privately because it contains secrets.
Verify that the variables reach the containers.
Boot only infrastructure.
Before any application role, run the restore-clone lockdown against the recovered database.

The restore overlay refuses direct server, worker, and webhook starts. Application-health inspection needs an operator-owned isolated override. The server submits and recovers reviews. The worker runs them.
Keep AGENT_ENABLED=false on server and worker.
Before review execution, read back silent mode and the workspace delivery pause.
Record provider-readiness failures caused by isolation without restoring live write access.

Releases before silent mode require the environment, mentor/Slack shutdown, credential withholding, and network barrier throughout. These controls permit restore inspection, not a full silent review run.
Before such a run, upgrade.
Before such a silent review run, validate the egress brake.

Choose the broker recovery strategy before starting consumers

Restoring PostgreSQL to T-1 while JetStream retains T can replay work whose local processing/delivery records disappeared. Broker publish deduplication and consumer ACKs are not external-delivery idempotency. A provider comment written after T-1 can still exist with no restored ledger entry.

Default: quarantine and recreate the instance's broker state. Accept and record event loss rather than silently replaying a divergent buffer. The self-host broker publishes no host port.

Stop publishers and consumers.
Use an authenticated NATS CLI context for the intended instance on its isolated Compose network.
Record each stream and consumer's configuration, sequence/ACK positions, and pending counts.

Use these commands:

nats stream report
nats stream info STREAM
nats consumer info STREAM CONSUMER

Stop NATS.
Identify the exact nats-server data mount through container inspection and Compose project labels.
Preserve a cold snapshot and configuration in restricted incident storage.

CAUTION: Remove only confirmed instance storage
Never use down -v, a wildcard, or the first volume name ending in nats-data. On a shared broker, do not remove the broker volume. Have its owner export this instance's streams and consumers first. Have the owner delete/recreate only those streams and consumers.
For a dedicated self-host broker, remove only that instance's nats-server container and confirmed nats-data volume.
For a dedicated self-host broker, recreate NATS with an empty store.
Run the isolated database migration first, then start the application server.
Turn on silent mode and read it back.
Start the isolated webhook role to recreate streams.

Its Compose dependency requires the completed database migration.
Keep public ingress blocked.
Verify that server consumers attach after streams exist.
If necessary, restart only the isolated server.
Before provider ingress returns, check fresh stream/consumer positions and no old pending deliveries.
Keep silent mode on during reconciliation.
Inventory the gap.
Run supported provider reconciliation within its configured monitoring window.
Observe that reconciliation.
Explicitly record unrecoverable intervals.

Reconciliation can restore current source state, not all historical events or deleted sources. GitHub, GitLab, Slack, and Outline have different recovery limits.

Replay is an exception requiring a separate drill. The send recovery contract covers interrupted writes. It does not establish that identities derived from lost jobs survive a database rewind.

Preserve the retained stream.
Record the exact consumer start sequence/time.
Recreate only the chosen consumer under isolation.
Do not merely reset an ACK cursor and reconnect production writers.
Exercise provider-marker reconciliation with the specific target release and all enabled destinations.

Do not release replay externally under any of these conditions:
- A stable identity cannot be reconstructed.
- Provider lookup is inconclusive.
- Any live copy is unaccounted for.
Under those conditions, leave delivery suppressed.
Under those same conditions, reconcile manually.

Drill

Create a controlled delivery after the chosen restore point.
Retain newer broker events.
Restore PostgreSQL.
Exercise the chosen reset/replay strategy.
Include duplicate delivery and provider success with lost local acknowledgment.
Include a partially written summary/inline package and a provider lookup outage.
Verify no duplicate provider objects.
Verify no historical suppressed work was released.
Verify the recorded data-loss window.
Before release, also check restored PostgreSQL jobs/proposals and reviews created by source sync.

An empty broker does not remove those jobs/proposals or prevent source sync from creating new reviews.

Access and erasure for one person

Use Instance administration → Person data. Only an instance admin with a recent sign-in can create a preview, download an export, or start erasure. Requests and replies go through the privacy contact in the instance notice. This page is not a data-subject request portal.

Verify the requester's identity and authority under your privacy procedure.
Enter the account ID or exact provider instance and provider user ID.
For Slack, also enter the native Slack workspace ID.
Add all verified identities that belong to the request.
Do not match names, logins, or email addresses.

A conflicting account link stops resolution. Cached Slack-to-SCM actor stamps are not identity proof. The page does not infer an identity from those stamps.
Before another preview, repair the verified link or correct the supplied exact IDs.
If collected work has a cached actor stamp, include verified exact keys for both providers.
If stamped work has a conflicting association, alternatively repair that association.
Select Preview data.
Check resolved identities and store counts.

Stores with no rows appear under stores hold no rows. The preview expires after 48 hours. Export and erasure use its frozen row keys. Changed data needs a new preview, not an SQL supplement.
For an access request, select Download JSON export.
For an access request, transfer the file securely.
For an access request, handle the file as personal data.

The export has a schema version and a stores object keyed by store name. It excludes credentials and session capabilities. The completed erasure receipt is not an export archive.
For erasure, follow Un-deliver external feedback, then purge for each provider locator in the preview.
For erasure of an unconfirmed write without a comment ID, inspect the linked reviewed work for summary and inline feedback.
For that erasure inspection, include preserved discussions.

The application does not automatically delete provider comments already posted.
For erasure, select Erase data….
For erasure, in the dialog, confirm removal of the listed provider copies.
For erasure, type ERASE.
For erasure, select Erase data.
For erasure, keep the request URL for access after a reload.

The page follows the job until it finishes. The job records progress one store at a time. Worker evidence owners must acknowledge removal of each attempt and copy before database evidence clears. An unavailable owner is a failure, not proof of erasure.
For erasure, if the job stops, correct the operational failure.
For erasure, after correcting a stopped job's failure, select Resume erasure… on the same request.
For erasure, do not create a second request to bypass a failed job.
For erasure, before replying, check Erasure complete and the per-store receipt.

The receipt retains the acting instance admin, operational state, times, and counts. It retains neither erased content nor selected subject keys. Completion clears working request keys. Expiry clears abandoned previews. Failed jobs retain the working keys needed to resume.

Review-package provider requests hold capture admission until their outcomes have been recorded. Erasure cannot pass an in-flight provider request. A queued call cannot recreate feedback after its review has entered erasure or its job capability has been revoked.

The preview freezes delivery facts, not only row IDs. If a provider write starts or its result changes after inspection, export and erasure reject the stale preview. An unconfirmed write or a migrated comment without an exact inspection locator stops the request and identifies its send or review. The system does not treat it as never delivered.

If the preview is stale, create a new preview before confirming removal again.
If the preview is stale, inspect the current locators before confirming removal again.

Practice results and feedback belong to the developer they describe. The person's own results and feedback are exported and erased, including feedback copies in another person's conversation. Another developer's results and feedback remain, even when the person commented on that work. Instead, erasure removes the review's captured inputs and Hephaestus's hidden memory in affected workspaces.

Exact provider-key suppression is separate from the audit receipt. It prevents further processing of the erased identity. It is not a reversible workspace consent choice. Provider sync can still mirror upstream records.
Tell the requester what remains outside the active application's control:
- Original provider records.
- Provider comments not yet removed.
- Delivered mailbox copies.
- Third-party retention and backups.
After a restore, follow isolation and reconciliation above before processing resumes.
Do not claim selective deletion in backups.

The personal-data map owns store-specific export, removal and residual retention. The operator's record of processing owns the legal record, and Data retention the periods.

Exact source bindings for person requests

Connected GitHub, GitLab, Slack, and Outline sources appear in the person-request provider list, even for a person without an account. A connected source cannot change to a different provider instance. Equivalent host case and default-port spellings remain the same instance and keep processing controls. The upgrade registers existing source instances. Names, logins, email addresses, and document text are not binding proof.

Use the source's exact native user ID.
For Slack, also use the native workspace ID.
To change the provider instance, create a new connection.
If upgrade reports mirrored Outline documents without a provider instance, restore verified state with that exact binding before upgrading.

Visible document content remains.

An evidence store is offline

Person erasure records a deletion request for each mounted store that captured the person's data. The owning worker cancels the affected attempt and removes the whole job folder, including failed rendering folders and retries. It confirms deletion only after the runtime releases its folder. A server that has no such folder cannot confirm deletion for another worker's volume.

If the mounted owner does not confirm deletion, the request fails and can resume. The store identity stays on the volume across a worker restart.

Restore the same worker volume.
Let its deletion poll run.
Resume the request in instance administration.
Do not replace the volume with an empty one and report the old copy as erased.
Keep a disconnected old volume under access controls until removal of its copy and confirmation of deletion.

This does not change the normal job-folder contract:

Admission removes the admitted folder.
Ended attempts have a one-hour grace period.
Restart cleanup removes old attempts.

Git history and repository owners

The JSON export's git_repository_history section lists repositories whose history may contain the person's commits. It uses exact stored provider identities and contributor references. The list does not prove each author string in raw Git history. Some Git attribution has no native provider person ID.

Do not infer ownership from names, logins, or email addresses.

Raw Git objects in mirrors and job folders cache the upstream repository controlled by the workspace's organization. Hephaestus does not rewrite them or refuse capture. A rewrite changes commit IDs and breaks citations. A later fetch reverses the rewrite. Sync can mirror records that still exist upstream.
If Git authorship must be erased, ask the source repository's owner to remove it there.
Ask that owner to apply the repository retention procedure.

Hephaestus erases what it derives and records about the person. The erasure job cancels affected active review attempts. It removes their evidence folders through the evidence-erasure hook. It must not report completion without proof of removal for affected copies. No unfiltered Git object archive enters a person export.

Normal job-folder retention stays unchanged:

Admission removes the verified folder.
Ended attempts have a one-hour cleanup grace.
Restart cleanup removes abandoned folders.

Un-deliver external feedback, then purge

Deleting in-app data does not delete a provider comment. If external copies must be scrubbed, complete un-deliver before workspace purge, disconnect or person erasure removes their locator records.

Turn on silent mode.
Stop active writers/sandboxes.
Preserve their send state.
Pause sending for the affected workspace. This prevents content recreation when silent mode is later turned off.
Locate the piece of feedback from the workspace review trace.
With authorized read-only database access, inventory all copies, not only the summary.

Run the query below with psql variables set to verified workspace and feedback IDs.

The query deliberately omits bodies.

SELECT f.id, f.review_id, f.item_id, f.destination,
       f.state, f.replaces_id, p.id AS copy_id,
       p.kind, p.state AS copy_state, p.write_started_at, p.write_marker,
       p.external_ref, p.external_url, p.mentor_entry_id, p.mentor_turn_id,
       p.path, p.start_line, p.end_line, p.side, p.posted_at
FROM feedback f
LEFT JOIN feedback_copy p
  ON p.feedback_id = f.id AND p.workspace_id = f.workspace_id
WHERE f.workspace_id = :'workspace_id'::bigint
  AND f.id = :'feedback_id'::uuid;

For conversation copies, follow mentor_entry_id to the mentor_entry row of the Hephaestus message that showed the feedback.

mentor_turn_id is the turn that claimed the feedback. A copy with mentor_turn_id and no mentor_entry_id has no stored Hephaestus message that showed it. For that copy, find the turn's question as described below. A copy from before the upgrade to one conversation per developer has neither value.

Read the message's conversation_id, turn_id, and surface. A message with the surface APP has no Slack copy. For a thread, follow mentor_conversation.parent_conversation_id to the main conversation. The main conversation's slack_team_id, slack_user_id, and slack_channel_id identify the Slack DM.

One Slack message shows all Hephaestus messages of a turn. The question of the turn records the state of that Slack reply: it is the mentor_entry row with the same turn_id and the kind USER. Read its slack_delivery and slack_stream_ts. Then read slack_ts of the last HEPH row of the turn.

If slack_delivery is DELIVERED, that slack_ts is the ts of the Slack reply. A reply in a Slack thread is a stream, and its ts is also in slack_stream_ts. An UNCERTAIN or PENDING reply can have a Slack copy without a recorded slack_ts. A FAILED reply in a Slack thread can show part of a stream. Thus, the ledger query alone cannot establish complete Slack removal.
Use the provider's paginated conversation replies API to find each exact bot-authored reply and its ts.
If provider inventory is inaccessible or ambiguous, record the gap.
If provider inventory is inaccessible or ambiguous, keep removal incomplete.
Follow replacement history.
Check the feedback_send row for the feedback: action, state, attempts, and last_error.
Treat each copy in WRITE_STARTED state and each UNCERTAIN send as a possible provider object, even without an external_ref. If the copy has a write_marker, search the provider for it. The application appends it to the posted body.
Before deletion, verify repository/project, item, provider account, author, and content against the provider.
Never infer an API object type from a numeric ID alone.
Use a separate authorized operator session to delete each confirmed provider object.
Do not turn off application silent mode for cleanup.

Select the endpoint for the actual copy:

Copy	Provider API
GitHub issue or pull-request summary comment	`DELETE /repos/{owner}/{repo}/issues/comments/{comment_id}` — issue comments
GitHub inline review comment	`DELETE /repos/{owner}/{repo}/pulls/comments/{comment_id}` — review comments
GitLab issue or merge-request note	`DELETE /projects/:id/issues/:issue_iid/notes/:note_id` or `DELETE /projects/:id/merge_requests/:merge_request_iid/notes/:note_id` — notes
GitLab discussion/diff note	Use its discussion and note IDs — discussions
Slack feedback or Hephaestus answer	`chat.delete` with the exact channel and message `ts`, using a token permitted to delete that author's message — Slack API
Outline	No current external feedback destination. Do not delete source documents as a substitute for removing feedback

A submitted GitHub review/approval is not an issue comment. A 404 can mean hidden/inaccessible, not deleted. A timeout is ambiguous. External deletion does not mark the immutable ledger as retracted. No admin removal API exists. Complete in-product retraction requires the retraction workflow.

For a submitted GitHub review/approval, use the provider-supported review dismissal/remediation flow.
Report any retained content.
For all providers, obey rate limits.
Verify the response. Slack requires ok: true.
Read back absence with credentials known to have access.
After a timeout, read before retrying.
Record each ledger ID/provider reference, deletion result, verification time, and operator in the restricted incident record.
Include partial failures.
Do not substitute REPLACED, REJECTED, recipient resolution, or edited write fences for retraction.
Before delivery resumes, verify no live copies or unresolved sends can recreate the content.
If you cannot prove this, keep the workspace paused.
Only then purge, if requested.
Retain minimal lawful removal evidence under the incident retention policy.

Deletion cannot recall email notifications, screenshots, or third-party archives.

Drill

Remove a summary, inline comments, and a Slack reply on controlled fixtures.
Include permission denial and timeout recovery.
Run the inventory again.
Exercise restart/retry under containment.
Verify that the incident record matches provider state.

Alert response

Use Scrape metrics for the private endpoint. Alert rules gives the shipped rules and their thresholds. Use the instrumented metrics in Practice review operations and Webhook ingestion operations. Configure thresholds against your traffic and recovery objectives. Assign an owner and test the alert delivery route. The signal names the alerts of the shipped rules that it answers.

Signal	Contain and diagnose	Recovery proof
Poison events / repeated consumer failures (`HephaestusConsumerPoison`)	Preserve stream sequence, kind, reason and delivery count, not raw payloads in logs. Current poison handling ACKs at the retry limit. It does not move data to a durable DLQ. NATS maximum-delivery advisories are not a DLQ either. Recover the retained event before retention expires if authorized. Never raise the retry cap to hide a poison loop.	Reproduce with a sanitized fixture, fix the cause, replay one controlled event under silence, then reconcile the bounded gap. Healthy subsequent events continue.
Dead-letter alert from deployment tooling	Identify which system owns the queue. Do not assume an application replay endpoint exists. Preserve IDs/configuration and restrict payload access.	Redrive only after fixing the cause, with a bounded batch and verified delivery idempotency. Record irrecoverable events.
Rising oldest queue age, stale sampler, failed jobs (`HephaestusAgentQueueStalled`, `HephaestusAgentJobsFailing`)	Check eligible/held/running separately, worker availability, readiness, model assignments, database and provider health. Do not delete jobs or reset leases manually.	Oldest age falls and new work completes. Held work is explained rather than silently counted as healthy.
Review sandboxes failing (`HephaestusReviewSandboxFailing`)	Read the worker log for the sandbox failure and the initializer's exit. On Kubernetes, check the sandbox namespace's quota and events, the agent image pull, and the nodes (Troubleshooting). Do not raise the retry cap.	New attempts start their sandboxes, and `agent.job.infra.retry.requeued` with `cause=sandbox` stops increasing.
Review sandboxes cannot reach the server or the provider (`HephaestusReviewUnreachable`)	For `server_unreachable`, check the network policy between sandboxes and the worker gateway, and the worker's readiness. For `provider_unreachable`, check the provider's status and `networkPolicy.externalEgress`.	The retries with that cause stop increasing, and reviews complete.
Review preparation failing (`HephaestusReviewPreparationFailing`)	Read the worker log line `Review preparation failed`: it names the stage, the kind and the reason. Check provider sync for the reviewed sources, and the database.	The retries with `cause=preparation` stop increasing.
Reviews failed by lost workers (`HephaestusReviewWorkerLost`)	Find why workers stop during attempts: out-of-memory kills, evictions, lost nodes, or drains that end before the attempt. Compare the worker restarts with the failed review's attempts.	Workers run without restarts, and no review fails as an orphan.
`UNCERTAIN` / failed send	Turn on silent mode if content safety is in doubt. Restore provider lookup/credentials. Never clear the write fence or create a new send to bypass uncertainty.	Existing provider objects are reconciled without duplicate creates, including inline copies.
Budget breach / unpriced usage (`HephaestusBudgetExhausted`, `HephaestusBudgetBlocked`)	Identify shared versus workspace-owned funding. Pause each affected budget/model. For runaway global spend stop all AI processes/egress. Silent mode does not stop spending.	Validate usage ledger against provider billing, explain unpriced buckets and in-flight overshoot. Observe a bounded fresh run before restoring caps. Budget-held work can resume automatically.
Backup stale / restore failed (`HephaestusBackupStale`)	Preserve the last good backup, check destination credentials, disk, WAL coverage and encryption-key recovery. On Kubernetes, read the log of the latest backup Job (Troubleshooting). A failed WAL archive also stops the backups. Escalate at the agreed RPO, not after the next deploy.	A new off-host backup and isolated restore succeed within measured objectives.
Webhook publish failures / unrecovered GitHub events (`HephaestusWebhookPublishFailing`, `HephaestusWebhookEventsUnrecovered`)	Check broker health and disk latency first: the receiver fails a delivery that the broker does not acknowledge in time. GitHub deliveries are redelivered for up to 80 minutes after their first failure. Do not restart the receiver to clear the alert.	`webhook.publish` failures stop and `github.webhook.unrecovered` returns to zero. Redeliver older GitHub deliveries in the App settings within three days, or let reconciliation recover the rest.
GitHub redelivery not running (`HephaestusWebhookRedeliveryNotRunning`)	Check that the GitHub App private key and App ID are current, and that the App can read its webhook deliveries. Then read the server log for the refused requests.	`github.webhook.redelivery` reports no `refused` or `unavailable` outcome for 30 minutes. See Failed GitHub deliveries.
Many GitHub redeliveries requested (`HephaestusWebhookRedeliveriesRequested`)	GitHub recorded no successful delivery for these events, so the cause can be before the receiver: the Gateway, its timeout, or an unready receiver. Read the App's Advanced → Recent Deliveries for the status of the failed deliveries. Then check the receiver's readiness and `webhook.publish` failures.	`github.webhook.redelivery` with `outcome=requested` stops increasing.
No available webhook receiver (`HephaestusWebhookReceiverDown`)	Every provider delivery fails now. Check the webhook Deployment's events, and its pods' readiness probe and log. A pod that waits for the schema needs the migration Job. GitHub deliveries are redelivered for up to 80 minutes after their first failure. Other providers need reconciliation.	The webhook Deployment has an available replica, and `webhook.publish` counts successes.
Unavailable webhook receiver replica (`HephaestusWebhookReceiverUnready`)	The other receiver replicas serve. Read the unavailable pod's events and log, and delete it if it does not recover.	The webhook Deployment has no unavailable replica.
Broker storage exhaustion (`HephaestusJetStreamVolumeFilling`)	Follow the documented broker/receiver restart order after freeing space. Keep ingress loss visible. Do not purge streams merely to clear an alert.	Receiver durable acknowledgments and consumption recover. Inventory missing events and verify configured bounds.
JetStream backlog (`HephaestusStreamBacklog`)	Check that the server runs and is ready, and read its consumer log. `webhook.stream.consumers.without.pull.requests` above zero shows a consumer from which nothing fetches. A rising `ack.pending` with poison events shows a retry loop. Do not purge the stream or delete its consumers.	Pending and ack-pending return to zero. If the stream reached its bound meanwhile, let reconciliation recover what it removed (Webhook ingestion operations).
Stream near its byte bound (`HephaestusWebhookStreamNearByteBound`)	The stream's age bound is too long for its volume. Check the consumer backlog first: a large backlog keeps messages that no age removes. Lower `HEPHAESTUS_WEBHOOK_STREAM_MAX_AGE` (Webhook ingestion operations). Do not purge the stream.	`webhook.stream.bytes.utilization` stays below 0.5.
Stale or missing stream gauges (`HephaestusStreamPollStale`, `HephaestusStreamPollMissing`)	The receiver cannot read the broker, so all stream gauges hold old values. Check broker health and the receiver's warning log.	The poll age falls below the threshold for each stream.
Metrics target down (`HephaestusMetricsUnavailable`)	Check that the pod or container runs and is ready, and that the network policy admits the scraper. The other application alerts of that target cannot fire meanwhile.	The target's `up` is 1.
WAL archive failing (`HephaestusWalArchiveFailing`)	Read the PostgreSQL log for `archive-push` errors. Check the bucket's credentials, its reachability and its free space. After a change of the bucket's credentials, restart PostgreSQL. If `database_wal_archive_sample_failures_total` increases, the server cannot read the archiver: check the server's database connection first. Do not turn `archive_mode` off to clear the alert.	`database_wal_archive_failing` is 0, and a new backup succeeds. The point-in-time recovery has a gap from the first failure until the archive works again.
PostgreSQL volume filling (`HephaestusPostgresVolumeFilling`)	Check the WAL archive first: a failed archive keeps the WAL on the volume. Read the PostgreSQL log for `archive-push` errors and the log of the latest backup Job. Do not delete files from the WAL directory. Expand the volume only after the archive works, if its storage class permits expansion.	The archive pushes again, free space rises, and a new backup succeeds.
Node disk filling (`HephaestusNodeDiskFilling`)	Find what fills the node: Git mirrors (`server.gitCache.sizeLimit`, `worker.gitCache.sizeLimit`), review sandboxes, which have no disk limit, or images. Drain the node to move the worker when a sandbox fills it.	Free space stays above the threshold, and no pod on the node is evicted for disk pressure.
Certificate expiring (`HephaestusCertificateExpiring`)	Read the status and events of the Certificate and its latest CertificateRequest. Fix the issuer or the challenge. PostgreSQL reads a renewed certificate when it restarts (Database maintenance).	cert-manager issued a new certificate, its expiry moved ahead, and the roles connect.
Host does not follow its channel (`HephaestusDeployReconcileFailing`, `HephaestusDeployReconcileStale`, `HephaestusDeployMetricsMissing`, `HephaestusDeployToolingPending`)	Read `journalctl -u hephaestus-reconcile.service` for the failed run. A chart that refuses the host's values names the value. Change the site values, or wait for the fix of the release. For a stale run, check `systemctl status hephaestus-reconcile.timer`. Do not apply a release by hand around the reconciler (Pull-based deployment).	A run ends with `hephaestus_deploy_reconcile_success` 1, and `hephaestus_deploy_info` names the channel's release.
Sandbox isolation failed (`HephaestusSandboxIsolationFailed`)	The worker starts no sandbox and reports itself not ready. Read the worker log: it names each failed check (Troubleshooting). Do not widen the network policy or the Pod Security level to pass the probe.	The next probe passes, `sandbox.isolation.passed` is 1, and the worker is ready.

Model pricing update

Prices are runtime catalog data, not a source-code pricing table. Instance prices are superseded with history. Workspace-owned models use inline prices. Missing cache rates can leave the entire call unpriced.

Follow AI provider pricing.
Record the vendor/contract rate source, upstream model ID, currency, effective date, and all token buckets.
Update the instance catalog or workspace-owned catalog as applicable.
Confirm USD per million tokens, including cache read/write rates.
Never mark an unknown model NO_CHARGE to bypass a refused call.
For a new/unpriced model, create it disabled.
For a new/unpriced model, price it.
For a new/unpriced model, enable it.
For a new/unpriced model, assign it to a purpose.
For repricing, use the catalog UI.
Verify that historical usage was not billed again.
Verify that a controlled new call uses the intended rate snapshot.
Compare a hand-calculated token-cost example and provider billing.
If the rate is wrong, correct the current catalog.
If the rate is wrong, keep affected models disabled.
If the rate is wrong, document the accounting gap instead of rewriting historical usage.

Decommission

CAUTION: Shared integrations and retained data

Do not delete a shared app that another instance still uses. Destroy data only when retention permits.

Turn on silent mode.
Stop writers, workers, sandboxes, and deployment automation.
If required, complete external removal before erasure.
Take only final backups required by retention.
Deregister webhooks.
Uninstall/revoke relevant GitHub App installations.
Revoke relevant GitLab tokens/hooks.
Remove relevant Slack app subscriptions and tokens.
Revoke relevant Outline app/API credentials and hooks.
Revoke OAuth and LLM credentials.
Remove DNS/TLS and alert routes.
When retention permits, destroy volumes, secrets, and off-host backups.
Verify that provider subscriptions and recurring AI or storage charges have stopped.
Record retained legal evidence and its deletion owner/date.

Operator evidence and authoritative guidance

Keep drill evidence in the restricted deployment record. It must include these facts:

Operator/reviewer and UTC date.
Release locks and isolated URL/project.
Provider fixtures, backup checksum, and recovery target.
Broker strategy/positions and measured recovery point/time.
Provider counts before/after.
Failures and follow-up owner.

Repeat drills after changes to migrations, write identity, provider APIs, broker configuration, or backups.

Email delivery operations

SENT means relay acceptance, not inbox delivery. SMTP health must not be a liveness or readiness dependency. A broken email path cannot reliably report its own failure by email. Sustained unavailable, rate_limited, rejected, or expired outcomes require investigation.

Use Instance settings → Email → Send test email to distinguish missing configuration, silent mode, exhausted capacity, and relay failures.
For application, database, NATS, scheduler, or SMTP outages, page the operator through an external monitoring channel.
Monitor the notification.email.delivery counter by bounded kind and outcome.
Inspect the publication backlog without selecting serialized_event:
```
SELECT status, count(*) AS publications, min(publication_date) AS oldest_publication
FROM event_publication
GROUP BY status;
```
Retry batches prioritize publications least recently retried. One failed batch therefore does not indefinitely block later work. This fairness does not promise delivery within five minutes. Capacity limits and backlog size still apply. Each listener rechecks current eligibility. silent mode completes withheld mail rather than retaining a flood for later.
Alert when pending/failed publications age beyond the expected five-minute retry interval.
Investigate before notification expiry.
Use a separate database-monitoring identity with least privilege.
Do not replay a backlog blindly.

Hephaestus records synchronous SMTP rejection. It does not claim to ingest asynchronous delivery-status notifications or provider complaints.

Before inviting a large audience:

Confirm the relay's quota and SPF/DKIM/DMARC alignment.
Confirm the monitored reply address and bounce/complaint handling.
Use a relay that suppresses hard bounces and complaints.
Monitor that suppression list.
Keep optional bulk mail disabled until this operational path works.
Never re-enable a suppressed address without resolving the reason.

See email configuration.

Unsubscribe URLs contain narrowly scoped bearer capabilities. A local Mailpit MIME test alone cannot prove DNS or DKIM configuration.

Redact /notifications/unsubscribe/* in reverse-proxy access logs.
Never retain the public confirmation page's token query.
With a real message, validate the externally reachable HTTPS POST path.
Validate DKIM coverage of both unsubscribe headers.

Workspace connection alerts cover recorded Slack credential revocation and verified GitHub app suspension, with paired recovery. Intentional disconnection, routine token refresh, and first-time connection are not incidents. GitLab and Outline do not yet expose equivalent durable incident signals. Monitor their existing health/configuration surfaces rather than assuming email coverage.

First response and kill-switch map​

Turn silent mode on, verify it, and turn it off​

Silent deployment checklist​

Upgrade and failed-migration recovery​

Backup, restore and PostgreSQL × JetStream​

Pre-boot isolation — mandatory for every restore​

Choose the broker recovery strategy before starting consumers​

Access and erasure for one person​

Exact source bindings for person requests​

An evidence store is offline​

Git history and repository owners​

Un-deliver external feedback, then purge​

Alert response​

Model pricing update​

Decommission​

Operator evidence and authoritative guidance​

Email delivery operations​