Reapply erasures after a restore
Erase again each disconnect, purge, erased collection, deleted message and person that a database restore brought back, before the server processes data.
A database restore brings back the data that Hephaestus erased after the recovery point. That is disconnected connections, purged workspaces, erased Outline collections, deleted conversation messages and erased people. Reapply these erasures before the restored server processes data or opens access.
The server's erasure-recovery startup profile turns background work off: no scheduled task, claimer or inbox consumer runs.
Startup stops if a setting turns background work on beside the profile.
The profile then reapplies each erasure in a manifest, in this order: disconnects, workspace purges, erased collections, deleted messages, then people.
Each step is idempotent, so a restart with the same manifest is safe.
A missing or invalid manifest, a manifest of version 1 or 2, an unknown provider instance, or an incomplete erasure stops startup.
Before you start
- You restored the database and completed its migration with
server.stopped=true, as steps 1 to 6 of Recover the live installation describe. - External egress stays denied, as step 3 of the same procedure describes.
- You have the restricted incident record that lists each erasure after the recovery point.
- You have
kubectlaccess to the release namespace, andjq.
Write the manifest
- From the incident record, write a JSON manifest of each erasure after the recovery point.
- List each disconnected connection under
disconnects, with its workspace ID and connection ID. Include each disconnect that a provider caused, such as a Slack app uninstall. - List each purged workspace under
workspacePurges, with its workspace ID. The server log eventsconnection.erasedandworkspace.purgedcarry these IDs, so they cross-check the record. - List each Outline collection that an admin erased under
collectionErasures. Give its workspace ID, connection ID and collection ID. The log eventoutline.collection_erasedcarries these IDs asworkspace.id,connection.idandoutline.collection.id. - List each conversation message that a developer deleted under
messageDeletions, with its workspace ID and entry ID. The log eventconversation.message_deletedcarries both IDs asworkspace.idandconversation.entry.id. It never carries the words. - List each erased person under
people, including an account that its owner deleted. The log eventaccount.deletednames each account that its owner deleted, by itsaccount.id. That account ID alone identifies the person. Include the account ID of each person, when you know it, and all known exact native identities. - For GitHub and GitLab, use the numeric user ID. For Slack, use the user ID and the team ID. For Outline, use the user UUID. Never use a login, a display name or an email address in place of these IDs.
- Give the exact provider type and server URL from the restored provider inventory.
- Set
administratorAccountIdto an administrator who is active and has instance-admin rights in the restored database. Onlypeopleneeds the administrator. Another administrator must authorize the erasure of that administrator. - Set
externalCopiesRemovedtotrueonly after you confirm that the provider copies are gone, as Remove feedback from a provider requires. The recovery does not delete provider-hosted copies.
The manifest has this structure:
{
"version": 3,
"restorePoint": "2026-11-01T00:00:00Z",
"administratorAccountId": 10,
"noLaterErasures": false,
"disconnects": [
{"workspaceId": 3, "connectionId": 17}
],
"workspacePurges": [
{"workspaceId": 5}
],
"collectionErasures": [
{"workspaceId": 3, "connectionId": 21, "collectionId": "6a1f3b52-8e0d-4b7c-9f26-3d4e5a6b7c8d"}
],
"messageDeletions": [
{"workspaceId": 3, "entryId": "0f2d6c1e-5b7a-4c3e-9a41-2b8d7e6f1a90"}
],
"people": [
{
"accountId": 42,
"identities": [
{"providerType": "GITHUB", "serverUrl": "https://github.com", "subject": "123456"},
{"providerType": "GITLAB", "serverUrl": "https://gitlab.example.test", "subject": "234567"},
{"providerType": "SLACK", "serverUrl": "https://slack.com", "subject": "U123456", "teamId": "T123456"},
{"providerType": "OUTLINE", "serverUrl": "https://outline.example.test", "subject": "00000000-0000-4000-8000-000000000042"}
],
"externalCopiesRemoved": true
}
]
}What each entry does:
- A disconnect erases the local data of the connection and releases its credential. It also records the removal of what the connection installed at the provider. The server tries that removal at once. It fails while provider egress is closed, and the server tries it again after egress opens. The erasure does not wait for it.
- A purge deletes the data of the workspace, as the original purge did.
- A collection erasure deletes the collection's mirrored documents and its registration, as the admin's erase did. No sync mirrors it again.
- A message deletion erases the message's words and every summary that read them, as the developer's delete did.
- An entry needs nothing more if the restored database does not have it, or if it is erased already.
If no erasure occurred after the recovery point, declare it:
{
"version": 3,
"restorePoint": "2026-11-01T00:00:00Z",
"noLaterErasures": true,
"disconnects": [],
"workspacePurges": [],
"collectionErasures": [],
"messageDeletions": [],
"people": []
}Empty lists without noLaterErasures: true stop startup.
Redo the repository removals
The manifest does not list repository removals. A restore brings back each repository that an admin removed after the recovery point, and its synced work. Redo these removals after the recovery and before egress opens:
- From the incident record, list each repository that an admin removed from a workspace after the recovery point.
- For a GitLab project, stop it again, then remove its kept work in the workspace's repository settings.
- For another repository, remove it from the workspace's monitor list again.
Keep the manifest outside the database, and give access only to the recovery operators. Do not put it in Helm values, a ConfigMap, a command-line argument or shared logs. Hephaestus does not store or log the manifest. Keep it for the same period as each backup that can restore erased data, as Backup retention describes.
Start the server with the recovery profile
The chart starts the server with the prod profile and a startup probe.
A large manifest can take longer than the startup probe permits.
Thus, this procedure patches the server Deployment for the recovery.
The patch mounts the manifest from a Secret and adds the erasure-recovery profile.
It sets AGENT_ENABLED=false, so the server starts no sandbox.
It keeps the readiness probe, removes the startup and liveness probes, and sets the rollout progress deadline to one day.
The Secret holds private identities, so give namespace access only to the recovery operators. A Secret holds less than 1 MiB. For a larger manifest, mount a protected read-only volume at the same path.
While the recovery Pod is not ready, HephaestusServerDown can fire.
Silence it in your Alertmanager for the time of the recovery.
-
Keep retained server evidence volumes under access control until the recovery is complete. Their servers must reconcile their folders against the recovered receipts before you use the volumes again. Follow A server with evidence is offline.
-
Set
NAMESPACE,SERVER_DEPLOYMENTandERASURE_MANIFESTto the namespace, the server Deployment and the manifest file. -
Save the Deployment, create the Secret, and apply the recovery patch:
umask 077 server_recovery_dir=$(mktemp -d) erasure_secret="${SERVER_DEPLOYMENT}-erasure-recovery" kubectl -n "$NAMESPACE" get deployment "$SERVER_DEPLOYMENT" -o json > "$server_recovery_dir/before.json" kubectl -n "$NAMESPACE" create secret generic "$erasure_secret" --from-file="manifest.json=$ERASURE_MANIFEST" server_container=$(jq -r '.spec.template.spec.containers[0].name' "$server_recovery_dir/before.json") jq -n --arg container "$server_container" --arg secret "$erasure_secret" ' {spec: {replicas: 1, progressDeadlineSeconds: 86400, template: {spec: { containers: [{name: $container, startupProbe: null, livenessProbe: null, env: [ {name: "SPRING_PROFILES_ACTIVE", value: "prod,erasure-recovery"}, {name: "HEPHAESTUS_PRIVACY_RECOVERY_MANIFEST", value: "/var/run/hephaestus/erasure-recovery/manifest.json"}, {name: "AGENT_ENABLED", value: "false"}], volumeMounts: [{name: "erasure-recovery", mountPath: "/var/run/hephaestus/erasure-recovery", readOnly: true}]}], volumes: [{name: "erasure-recovery", secret: {secretName: $secret, defaultMode: 288}}] }}}}' > "$server_recovery_dir/recovery-patch.json" kubectl -n "$NAMESPACE" patch deployment "$SERVER_DEPLOYMENT" --type=strategic --patch-file="$server_recovery_dir/recovery-patch.json" -
Wait until the recovery Pod is ready:
kubectl -n "$NAMESPACE" rollout status deployment "$SERVER_DEPLOYMENT" --timeout=60mA timeout of this command does not stop the Pod. If the recovery takes more than one hour, run the command again. If the recovery approaches one day, extend the progress deadline before it expires. After the deadline, wait for the Pod with
kubectl wait --for=condition=Ready pod <pod-name> --timeout=60m.
Return to normal operation
-
Restore the saved Deployment specification. It has zero replicas from the maintenance stop:
jq '{spec: .spec}' "$server_recovery_dir/before.json" > "$server_recovery_dir/restore-patch.json" kubectl -n "$NAMESPACE" patch deployment "$SERVER_DEPLOYMENT" --type=merge --patch-file="$server_recovery_dir/restore-patch.json" -
Wait until the recovery Pod stops, then remove the Secret:
server_selector=$(jq -r '.spec.selector.matchLabels | to_entries | map(.key + "=" + .value) | join(",")' "$server_recovery_dir/before.json") kubectl -n "$NAMESPACE" wait --for=delete pod -l "$server_selector" --timeout=5m kubectl -n "$NAMESPACE" delete secret "$erasure_secret" -
Remove the silence of
HephaestusServerDown. -
Continue with step 8 of Recover the live installation. That step starts the normal server with Helm.
Check that it worked
- The recovery Pod became ready in step 4 of the recovery start. Its startup applied every erasure in the manifest.
- The recovery Pod log has no
person_data.recovery.failedrecord. - The server Deployment has no
erasure-recoveryprofile and no manifest mount.
Troubleshooting
The recovery Pod logs person_data.recovery.failed
The startup stopped before it applied every erasure.
The record has a closed person_data.recovery.reason code.
It does not quote the manifest or the identities.
- Keep the other Deployments stopped and external egress denied.
- Correct the cause that the reason names:
- For
MANIFEST_VERSION, write the manifest as version 3. - For
BACKGROUND_WORK, remove the setting that turns background work on beside the profile, such asHEPHAESTUS_BACKGROUND_WORK_ENABLED. - For
ERASURE_INCOMPLETEon a disconnect, turn on the integration of that connection for the recovery run. Without it, nothing can erase its data. - For
ERASURE_INCOMPLETEon a message deletion, check that the entry ID names a message that a developer wrote. A manifest that names another entry stops the recovery. - For another reason, correct the manifest, or complete the unfinished erasure.
- For
- Replace the Secret with the corrected manifest.
- Delete the recovery Pod, so that the Deployment starts it again with the corrected manifest.
- Wait for the recovery Pod, as step 4 of the recovery start describes.
Do not remove the recovery profile to continue past the failure.