# Upgrade to 1.0 1.0 starts the [1.x upgrade guarantee](/next/admin/compatibility-policy.md#upgrade-guarantee). The cutover to 1.0.0 is a stopped upgrade: [ADR 0070](https://github.com/hephaestus-build/Hephaestus/blob/main/docs/decisions/0070-1-0-is-a-cutover-release.md) records why. No role of the previous release runs beside a 1.0 role. No role of the previous release runs on the 1.0 schema. Plan for one outage, from the first stopped role to the last passed check. The migration entry *Stop conversation work before enabling digest runs* requires matching runtime-contract-19 server and agent images. Drain and stop every server, take a verified backup, and remove the stopped release's conversation SandboxClaims. Then start the servers and run the sandbox isolation gate at once: its test sandboxes must reach the gateway, and the server serves the gateway. If a check fails, scale the servers to 0 before work resumes. A count of zero for conversation turns stops both requested turns and proactive digests. 1.0.0 carries the source and images of the last qualified release candidate. A host that already runs that candidate changes only its release metadata. It still stops every role, and it still needs the verified recovery point and the checks below. The target release's [migration guide](https://github.com/hephaestus-build/Hephaestus/blob/main/MIGRATION.md) owns each required action. This page orders the actions of the migration entries after `1.0.0-rc.5` and names the entry of each. TODO(K5) This page states no sizing number for 1.0, because the K5 measurement is not done. [Sizing and scaling](/next/admin/sizing.md) has no measurement of the sandbox load of 1.0 yet. Do not size nodes for 1.0 from this page. ## Before you start 1. Find the installed release in **Instance administration → Overview**. 2. Select the required path in [Database baseline prerequisites](/next/admin/compatibility-policy.md#database-baseline-prerequisites). 3. Read every migration entry between the installed release and the target release. Each release candidate has its own `### v1.0.0-rc.N` section in the migration guide. 4. Preserve the current signed release lock, chart package, site values, and required keys. 5. Verify a restorable database backup with its matching keys before you upgrade. [Backup policy for 1.0](/next/admin/backup-restore.md#backup-policy-for-10) states the release scope. Each upgrade still requires a verified recovery point. If you have no verified recovery point, stop before the migration. ## Prepare before the maintenance window Complete these items before the maintenance window. The window then holds only the cutover. ### The sandbox platform The chart installs a sandbox template and a warm pool for each kind of sandbox. They need the agent-sandbox controller and its resource definitions. Thus, the controller must exist before the chart. 1. **Controller.** The platform bundle of the release pins the agent-sandbox controller in `k3s-agent-sandbox.yaml`. The one-host installer applies it. On another cluster, apply it for each release with cluster-admin rights. The migration entry *Install the agent-sandbox controller before the chart* gives the commands. Without it, `helm upgrade` stops because it cannot find the kind `SandboxTemplate`. 2. **Sandbox runtime.** Review and conversation Pods use `sandbox.templates.runtimeClassName`, or the cluster's default runtime. [Sandbox prerequisites](/next/admin/sandboxes.md#sandbox-prerequisites) states the requirements for each runtime. The one-host platform installs [gVisor](/next/admin/k3s.md#the-platform-files) for the chart's sandbox templates (`sandbox.templates.runtimeClassName`). Both practice reviews and conversation turns claim these templates. The installer downloads the pinned gVisor release from `storage.googleapis.com`, so the host must reach that address. The first run with gVisor restarts k3s once, which stops the running Pods for a short time. Do this during the window, as the cutover steps below describe. 3. **NetworkPolicy enforcement.** The network plugin must enforce NetworkPolicy for egress and ingress. The platform isolation gate qualifies that boundary before work starts. [Sandbox prerequisites](/next/admin/sandboxes.md#sandbox-prerequisites) lists its checks. The [sandbox isolation gate](/next/admin/sandboxes.md#sandbox-templates) is a script of the repository. Run it from a checkout of the release tag. It starts two sandboxes from a template, and it fails when a sandbox reaches more than the gateway. Thus, it fails on a network plugin that does not enforce NetworkPolicy. A one-host installation that another tool deploys, such as [pull-based deployment](/next/admin/pull-based-deployment.md), updates its platform with the installer's `--platform-only` mode. [Install only the platform](/next/admin/k3s.md#install-only-the-platform) gives what it installs, its checks and the procedure. ### Values and keys to move The chart refuses an old name. Phase 1 of the window still runs the current release with your current site file and Secret, so change neither before the window. Before the window, prepare a copy of your site file for the target release, as step 1 of [Upgrade the host](/next/admin/k3s.md#upgrade-the-host) describes. Change the Secret in step 7, after phase 1. The migration guide owns each change. This table gives the names. | Where | Before | After | Migration entry | | ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------- | | Site file | `config.monitoring` | `config.sync`, with the same four child settings | *Rename scheduled sync settings* | | Site file | `config.auth.bootstrapAdmins`, `config.auth.browserExtensionIds`, `config.auth.passkeys.allowedOrigins`, `config.outline.allowedOrigins` as strings | The same keys as YAML arrays | *Convert chart list values to arrays* | | Site file | `conversationLoop.turns` | `sandbox.conversationTurns`, a count for the whole installation | *Stop every role before moving conversation turns to sandboxes* | | Site file | `worker.enabled`, `worker.sandbox.*`, `worker.runtime`, `worker.sandboxGateway.maxConnections`, `networkPolicy.workerGatewayPorts`, and the worker's drain, pool, git cache and resources | `sandbox.enabled`, `sandbox.*`, `sandbox.runtime`, `sandbox.gateway.maxConnections`, `networkPolicy.modelGatewayPorts`, and the same settings under `server` | *Move the worker values to the server and the sandboxes* | | Site file | `worker.replicas`, `worker.rolloutStrategy` | No key. `server.replicas` counts the servers. | *Move the worker values to the server and the sandboxes* | | Site file | `server.rolloutStrategy` | Remove the key. The server always uses a rolling update. | *Reserve room for rolling server updates* | | Site file | `webhook.replicas`, `webhook.resources` | None: the server receives provider webhooks | *Route provider webhooks to the server* | | Profile | `small-16gib` in `HEPHAESTUS_CHART_PROFILE` and `profiles/small-16gib.yaml` | `small` and `profiles/small.yaml` | *Use the small profile on every single-node install* | | Server settings | `hephaestus.agent.max-retries` | `hephaestus.agent.capacity.review-retries`. `AGENT_MAX_RETRIES` still works. | *Stop every role before moving review leases to agent runs* | | Secret | `DATABASE_USERNAME`, `DATABASE_PASSWORD` | `DATABASE_ADMIN_USERNAME` and `DATABASE_ADMIN_PASSWORD` for the bundled PostgreSQL, and the new `DATABASE_OWNER_PASSWORD` and `DATABASE_APP_PASSWORD` | *Give each database purpose its own login* | | Secret | `HEPHAESTUS_CONVERSATION_LOOP_SECRET` | Remove the key. Each sandbox receives its own credential. | *Stop every role before moving conversation turns to sandboxes* | | Image outside the chart | `HEPHAESTUS_WORKER_CAPACITY_MENTOR_MAX` | `HEPHAESTUS_MENTOR_TURN_CAPACITY` on the server | *Stop every role before moving conversation turns to sandboxes* | | Image outside the chart | `HEPHAESTUS_RUNTIME_WORKER_ENABLED`, the `worker` Spring profile and `HEPHAESTUS_WORKER_DRAIN_TIMEOUT` | `AGENT_ENABLED=true`, `HEPHAESTUS_AGENT_DRAIN_TIMEOUT` and the sandbox settings on the server | *Move the worker values to the server and the sandboxes* | | Custom agent image | An image from an earlier release | An image that you rebuilt from this release | *Rebuild custom agent images* and *Rebuild a custom agent image* | The table covers the changes after the release candidate `1.0.0-rc.5`. A host on an earlier candidate also needs the entries of the later candidates. Eight more checks need a read before the window: * API callers of the team routes move to the new contract: *Update team API callers*. * Custom API clients read error responses in the problem format: *Read problem responses from the top level*. * The first daily run erases old feedback and results. See *Choose the feedback retention before you upgrade*. * A production server needs a root HTTPS issuer. See *Use a root HTTPS API issuer in production*. * Slack connections need the canonical callback on the Slack app: *Use the canonical Slack workspace callback*. * Each chat model needs an output limit. The server reads missing limits from the provider when it starts. Check that no model in **AI providers** is marked **Limits missing** ([Model limits](/next/admin/ai-providers.md#model-limits)). See *Stop every role before moving model usage to one row for each call*. * *Stop every role before the repository enumeration upgrade*: every replica must record repository access events in the same connection revision. Stop all servers before starting the new release. * Duplicate repository monitor names stop the migration: run the query of *Stop every role before enforcing unique repository monitor names*. * A practice identifier can collide with a reserved route name: run the preflight of *Check practice identifiers before upgrading*. These migration entries also apply to a host that upgrades to 1.0: * *Stop every role before changing role-notice delivery*: use the procedure in that entry. * *Stop every role before replacing live Slack replies*: stop all replicas and keep a verified pre-upgrade backup. Existing replies with an unknown Slack outcome stay unconfirmed. * *Refresh the internal PostgreSQL certificate*: on one host, restart PostgreSQL after cert-manager issues the new certificate. * *Stop every role before correcting unknown issue comment counts*: the migration changes stored issue comment counts of zero to unknown. * *Use the stopped procedure for one-host release changes*: the one-host installer refuses a release change that does not follow it. * *Let the one-host installer reach the gVisor download*: allow `storage.googleapis.com` in the host's egress rules. * *Update custom practice script imports*: remove the imports that the release deletes from your custom scripts. * *Remove broker consumer settings*: remove `config.durableConsumerName` and `config.consumerInactiveThreshold` from every values file. * *Remove the Git checkout switch*: remove `config.gitCheckoutEnabled` from every values file. * *Move the GitHub webhook byte bound into the stream map*: remove the whole `webhook` block from every values file. This chart has no `webhook` key and refuses one. * *GitHub synchronization is incremental*: remove `HEPHAESTUS_SYNC_INCREMENTAL_ENABLED` from your configuration. * *Change the model call queries that read `class` or `queue`*: your own queries and alerts read the `band` label. * *Decide whether GitHub App installations need an instance admin's approval*: by default, an instance admin adopts each new installation. * *Use the canonical webhook receiver base*: remove `WEBHOOK_EXTERNAL_URL` and `hephaestus.webhook.external-url`. * *Remove existing sandbox claims before starting the new workers*. Delete the remaining sandbox workloads before the new release starts. * *Public activity pages count contributors without an account instead of naming them*. To keep their names, turn on **Show contributors without an account**. * *Stop every role before the models for practice scripts are deleted*. The upgrade deletes the Decisions, Embeddings and Rerank connections. * *Remove the review poll settings and the review Job metrics*. Remove `AGENT_POLL_INTERVAL` and `AGENT_CLAIM_BATCH_SIZE`. Remove your own alerts on these metrics. * *Stop every role before moving conversation turn leases to agent runs*. The previous release cannot read the new turn leases. * *Product feedback drafts use server egress*: allow the selected instance model's provider through the server firewall. Keep a verified pre-upgrade database backup. * *Stop every role for the conversation loop*. This earlier entry introduced the loop. The sandbox cutover below replaces it. * *Stop every role before moving conversation turns to sandboxes*. Stop the old loop before migration, remove the old workloads, and run the template isolation gate. * *Log records name their trace in `trace.id` and `span.id`*: update your log queries, saved searches and collector rules that read `context.traceId` or `context.spanId`. * *Log context uses consistent field names*: update your log queries that read `context.installation_id` or `context.proxy.apiProtocol`. * *Stop every role before clearing personal logins from GitHub App installation records*. The previous server cannot read an installation record without a login. ## Cut over to 1.0.0 The migration entry *Stop every role to cut over to 1.0.0* points to this section. CAUTION: Changed schema Do not use automatic Helm rollback for a stopped schema upgrade. An older role can corrupt data or fail against the changed schema. Keep every role stopped after a migration failure. Use [Failed-migration recovery](/next/admin/upgrade.md#if-migration-fails). Follow these steps in this order. Each step names the page that owns it. 1. Complete [Validate the actual production data](/next/admin/upgrade.md#validate-the-actual-production-data) on an isolated copy. 2. Pause [pull-based deployment](/next/admin/pull-based-deployment.md#operating-it), if enabled. Wait for an active run to finish. 3. Verify the target's [signed release assets](/next/admin/release-image-lock.md#independent-verification) and [chart](/next/admin/kubernetes.md#verify-the-release-chart). Record the current chart, profile, site values and replica counts. 4. Stop every role. Stop the old conversation loop first, if present. Do phase 1 of [Stopped schema upgrades](/next/admin/upgrade.md#stopped-schema-upgrades) with the source release: `server.replicas`, `worker.replicas` and `webhook.replicas` are 0. Wait until every role Pod, sandbox Pod and Job is gone. Foreground-delete the old claims and legacy Jobs, as *Stop every role before moving conversation turns to sandboxes* describes. 5. Back up the database. Verify that the backup restores, as [Backup and restore](/next/admin/backup-restore.md#back-up-the-database) describes. Export the keys and keep them with the backup and the current signed release lock. You need this recovery point to roll back. 6. Install the sandbox platform of the target release. On a one-host installation, run `--check-platform` and then `--platform-only` from the target bundle, as [Install only the platform](/next/admin/k3s.md#install-only-the-platform) describes. On another cluster, apply `k3s-agent-sandbox.yaml` and wait for the controller: ``` kubectl apply --server-side -f k3s-agent-sandbox.yaml kubectl -n agent-sandbox-system rollout status deployment/agent-sandbox-controller ``` If this step restarts k3s, wait until PostgreSQL are ready before the next step. On a one-host installation, the platform files also renew the internal PostgreSQL certificate: do *Refresh the internal PostgreSQL certificate* before you go on. 7. Change the Secret. First move the database logins, as step 3 of *Give each database purpose its own login* describes. `hephaestus-keys.sh` stops while the Secret still holds `DATABASE_USERNAME` or `DATABASE_PASSWORD`. Then run the target release's `hephaestus-keys.sh` so that the Secret holds every new key. Export the keys again and keep that copy with the restore point. 8. Migrate. Do phase 2 of [Stopped schema upgrades](/next/admin/upgrade.md#stopped-schema-upgrades) with the target chart and your changed site values. Save the migration Job's log. Require its `Complete` condition and no `ERROR` line in the log. If the Job fails, keep every role stopped and go to [Roll back](#roll-back). 9. Check the sandbox prerequisites on the migrated release: * The controller is ready and the chart's `SandboxTemplate` and `SandboxWarmPool` objects exist in the sandbox namespace. * The pre-pull DaemonSet is ready on every sandbox node: `kubectl --namespace hephaestus rollout status daemonset/hephaestus-agent-image`. It keeps the agent image on each node, because a cold pull takes longer than the start of a sandbox. 10. Start the server. Do phase 3 of [Stopped schema upgrades](/next/admin/upgrade.md#stopped-schema-upgrades) with the recorded server count. A server starts no sandbox until its isolation probe passes. The gate needs a running server, because a sandbox must reach the gateway. Thus, run the gate for each template now, as [Sandbox templates](/next/admin/sandboxes.md#sandbox-templates) describes. If it fails, fix the policy enforcement of the network plugin before you go on. 11. On a one-host installation, run the target release's installer with `--version`, as [Upgrade the host](/next/admin/k3s.md#upgrade-the-host) describes. Keep deployment automation paused until it finishes. 12. If pull-based deployment is enabled, make its channel name the installed release, then resume reconciliation. [Stopped major upgrades](/next/admin/pull-based-deployment.md#stopped-major-upgrades) gives the reconciler's rules for a cutover. 13. Load the alert rules of the target release into your Prometheus or vmalert. A chart upgrade brings them, but your own Prometheus does not read them until you load them ([Alert rules](/next/admin/alert-rules.md)). 14. Do [Check that it worked](#check-that-it-worked). What happens to a review that a server still held when it stopped is in *Stop every role before moving review leases to agent runs*. Provider reception stops during the window. [Reception during a restart](/next/admin/runtime-roles.md#reception-during-a-restart) explains provider recovery. Keep the window short. ## Roll back 1.0 has no mixed-version window. An older image or `helm rollback` does not undo the migration. 1. Keep every role stopped. 2. Restore the verified backup from step 5 into the database. Restore its keys into the Secret. [If migration fails](/next/admin/upgrade.md#if-migration-fails) gives the procedure. Use the previous release's signed lock. 3. Install the previous release. For a pull-based host, [Recover a stopped cutover](/next/admin/pull-based-deployment.md#recover-a-stopped-cutover) gives the Helm and applied-record steps. 4. Start the roles of the previous release only on the restored database. 5. Report the recovery point and any lost writes. The migration entry of a stopped change can give its own rollback notes. ## Reach the database baseline from 0.x A release before 1.0.0-rc.1 must first reach that database baseline, as [Prepare an older database for the current release](/next/admin/liquibase-baseline-runbook.md) describes. Then follow the cutover above. ## Check that it worked 1. Check the migration Job's log and the readiness of each enabled role. 2. Open **Instance administration → Overview** and read **Configuration readiness**. 3. Resolve each item under **Action required**. 4. Verify an authenticated read through the public gateway. 5. Run the sandbox isolation gate against each template and require every check to pass. 6. Run one practice review in **Approve first** mode, as [First steps § 7](/next/admin/first-steps.md#7-run-the-first-practice-review) describes. Watch it end with a result. 7. Ask one question in Hephaestus and verify that the answer completes. 8. Read the dashboards **Overview** and **Reviews and feedback**, as [Dashboards](/next/admin/monitoring.md#open-the-dashboards) describes. Reviews leave the queue, and no role is missing from the scrape. 9. Check these alerts for the first hours: `HephaestusAgentQueueStalled`, `HephaestusAgentRunReaperFailing`, `HephaestusAgentSandboxReaperFailing`, `HephaestusReviewSandboxFailing`, `HephaestusSandboxGatewayRefusing`. [Alert response](/next/admin/production-operations-runbook.md#alert-response) gives the action for each. 10. Did the upgrade cross *Stop every role before correcting unknown issue comment counts*? Then ask the administrators of each affected workspace to run its backfill. Until it finishes, those issues show an unknown comment count. Check [Sizing and scaling](/next/admin/sizing.md) before increasing capacity. Its measurements state their workload and limits. A successful upgrade does not qualify throughput or recovery time. ## Drain the webhook streams before the upgrade Follow the [provider event inbox cutover](/next/admin/webhook-ingestion-operations.md) before you start 1.0. Use the provider inbox cutover procedure before starting the new server. ## Stop every role before the provider resource repair upgrade Stop every server before the migration, then deploy the matching release. The old server cannot write the required authorization binding columns. Pending OAuth requests must restart after this upgrade. [Integration administration](/next/admin/integrations.md#confirm-a-provider-removal) describes the removal review and instance-admin release.