Skip to main content

Upgrade to 1.0

1.0 starts the 1.x upgrade guarantee. The cutover to 1.0.0 is a stopped upgrade: ADR 0070 records why. No role of the previous release runs beside a 1.0 role. No role of the previous release runs on the 1.0 schema. Plan for one outage, from the first stopped role to the last passed check.

The migration entry Stop conversation work before enabling digest runs requires matching runtime-contract-19 server and agent images. Drain and stop every server, take a verified backup, and remove the stopped release's conversation SandboxClaims. Then start the servers and run the sandbox isolation gate at once: its test sandboxes must reach the gateway, and the server serves the gateway. If a check fails, scale the servers to 0 before work resumes. A count of zero for conversation turns stops both requested turns and proactive digests.

1.0.0 carries the source and images of the last qualified release candidate. A host that already runs that candidate changes only its release metadata. It still stops every role, and it still needs the verified recovery point and the checks below.

The target release's migration guide owns each required action. This page orders the actions of the migration entries after 1.0.0-rc.5 and names the entry of each.

TODO(K5)

This page states no sizing number for 1.0, because the K5 measurement is not done. Sizing and scaling has no measurement of the sandbox load of 1.0 yet. Do not size nodes for 1.0 from this page.

Before you start​

  1. Find the installed release in Instance administration → Overview.
  2. Select the required path in Database baseline prerequisites.
  3. Read every migration entry between the installed release and the target release. Each release candidate has its own ### v1.0.0-rc.N section in the migration guide.
  4. Preserve the current signed release lock, chart package, site values, and required keys.
  5. Verify a restorable database backup with its matching keys before you upgrade.

Backup policy for 1.0 states the release scope. Each upgrade still requires a verified recovery point. If you have no verified recovery point, stop before the migration.

Prepare before the maintenance window​

Complete these items before the maintenance window. The window then holds only the cutover.

The sandbox platform​

The chart installs a sandbox template and a warm pool for each kind of sandbox. They need the agent-sandbox controller and its resource definitions. Thus, the controller must exist before the chart.

  1. Controller. The platform bundle of the release pins the agent-sandbox controller in k3s-agent-sandbox.yaml. The one-host installer applies it. On another cluster, apply it for each release with cluster-admin rights. The migration entry Install the agent-sandbox controller before the chart gives the commands. Without it, helm upgrade stops because it cannot find the kind SandboxTemplate.
  2. Sandbox runtime. Review and conversation Pods use sandbox.templates.runtimeClassName, or the cluster's default runtime. Sandbox prerequisites states the requirements for each runtime. The one-host platform installs gVisor for the chart's sandbox templates (sandbox.templates.runtimeClassName). Both practice reviews and conversation turns claim these templates. The installer downloads the pinned gVisor release from storage.googleapis.com, so the host must reach that address. The first run with gVisor restarts k3s once, which stops the running Pods for a short time. Do this during the window, as the cutover steps below describe.
  3. NetworkPolicy enforcement. The network plugin must enforce NetworkPolicy for egress and ingress. The platform isolation gate qualifies that boundary before work starts. Sandbox prerequisites lists its checks. The sandbox isolation gate is a script of the repository. Run it from a checkout of the release tag. It starts two sandboxes from a template, and it fails when a sandbox reaches more than the gateway. Thus, it fails on a network plugin that does not enforce NetworkPolicy.

A one-host installation that another tool deploys, such as pull-based deployment, updates its platform with the installer's --platform-only mode. Install only the platform gives what it installs, its checks and the procedure.

Values and keys to move​

The chart refuses an old name. Phase 1 of the window still runs the current release with your current site file and Secret, so change neither before the window. Before the window, prepare a copy of your site file for the target release, as step 1 of Upgrade the host describes. Change the Secret in step 7, after phase 1. The migration guide owns each change. This table gives the names.

WhereBeforeAfterMigration entry
Site fileconfig.monitoringconfig.sync, with the same four child settingsRename scheduled sync settings
Site fileconfig.auth.bootstrapAdmins, config.auth.browserExtensionIds, config.auth.passkeys.allowedOrigins, config.outline.allowedOrigins as stringsThe same keys as YAML arraysConvert chart list values to arrays
Site fileconversationLoop.turnssandbox.conversationTurns, a count for the whole installationStop every role before moving conversation turns to sandboxes
Site fileworker.enabled, worker.sandbox.*, worker.runtime, worker.sandboxGateway.maxConnections, networkPolicy.workerGatewayPorts, and the worker's drain, pool, git cache and resourcessandbox.enabled, sandbox.*, sandbox.runtime, sandbox.gateway.maxConnections, networkPolicy.modelGatewayPorts, and the same settings under serverMove the worker values to the server and the sandboxes
Site fileworker.replicas, worker.rolloutStrategyNo key. server.replicas counts the servers.Move the worker values to the server and the sandboxes
Site fileserver.rolloutStrategyRemove the key. The server always uses a rolling update.Reserve room for rolling server updates
Site filewebhook.replicas, webhook.resourcesNone: the server receives provider webhooksRoute provider webhooks to the server
Profilesmall-16gib in HEPHAESTUS_CHART_PROFILE and profiles/small-16gib.yamlsmall and profiles/small.yamlUse the small profile on every single-node install
Server settingshephaestus.agent.max-retrieshephaestus.agent.capacity.review-retries. AGENT_MAX_RETRIES still works.Stop every role before moving review leases to agent runs
SecretDATABASE_USERNAME, DATABASE_PASSWORDDATABASE_ADMIN_USERNAME and DATABASE_ADMIN_PASSWORD for the bundled PostgreSQL, and the new DATABASE_OWNER_PASSWORD and DATABASE_APP_PASSWORDGive each database purpose its own login
SecretHEPHAESTUS_CONVERSATION_LOOP_SECRETRemove the key. Each sandbox receives its own credential.Stop every role before moving conversation turns to sandboxes
Image outside the chartHEPHAESTUS_WORKER_CAPACITY_MENTOR_MAXHEPHAESTUS_MENTOR_TURN_CAPACITY on the serverStop every role before moving conversation turns to sandboxes
Image outside the chartHEPHAESTUS_RUNTIME_WORKER_ENABLED, the worker Spring profile and HEPHAESTUS_WORKER_DRAIN_TIMEOUTAGENT_ENABLED=true, HEPHAESTUS_AGENT_DRAIN_TIMEOUT and the sandbox settings on the serverMove the worker values to the server and the sandboxes
Custom agent imageAn image from an earlier releaseAn image that you rebuilt from this releaseRebuild custom agent images and Rebuild a custom agent image

The table covers the changes after the release candidate 1.0.0-rc.5. A host on an earlier candidate also needs the entries of the later candidates.

Eight more checks need a read before the window:

  • API callers of the team routes move to the new contract: Update team API callers.
  • Custom API clients read error responses in the problem format: Read problem responses from the top level.
  • The first daily run erases old feedback and results. See Choose the feedback retention before you upgrade.
  • A production server needs a root HTTPS issuer. See Use a root HTTPS API issuer in production.
  • Slack connections need the canonical callback on the Slack app: Use the canonical Slack workspace callback.
  • Each chat model needs an output limit. The server reads missing limits from the provider when it starts. Check that no model in AI providers is marked Limits missing (Model limits). See Stop every role before moving model usage to one row for each call.
  • Stop every role before the repository enumeration upgrade: every replica must record repository access events in the same connection revision. Stop all servers before starting the new release.
  • Duplicate repository monitor names stop the migration: run the query of Stop every role before enforcing unique repository monitor names.
  • A practice identifier can collide with a reserved route name: run the preflight of Check practice identifiers before upgrading.

These migration entries also apply to a host that upgrades to 1.0:

  • Stop every role before changing role-notice delivery: use the procedure in that entry.
  • Stop every role before replacing live Slack replies: stop all replicas and keep a verified pre-upgrade backup. Existing replies with an unknown Slack outcome stay unconfirmed.
  • Refresh the internal PostgreSQL certificate: on one host, restart PostgreSQL after cert-manager issues the new certificate.
  • Stop every role before correcting unknown issue comment counts: the migration changes stored issue comment counts of zero to unknown.
  • Use the stopped procedure for one-host release changes: the one-host installer refuses a release change that does not follow it.
  • Let the one-host installer reach the gVisor download: allow storage.googleapis.com in the host's egress rules.
  • Update custom practice script imports: remove the imports that the release deletes from your custom scripts.
  • Remove broker consumer settings: remove config.durableConsumerName and config.consumerInactiveThreshold from every values file.
  • Remove the Git checkout switch: remove config.gitCheckoutEnabled from every values file.
  • Move the GitHub webhook byte bound into the stream map: remove the whole webhook block from every values file. This chart has no webhook key and refuses one.
  • GitHub synchronization is incremental: remove HEPHAESTUS_SYNC_INCREMENTAL_ENABLED from your configuration.
  • Change the model call queries that read class or queue: your own queries and alerts read the band label.
  • Decide whether GitHub App installations need an instance admin's approval: by default, an instance admin adopts each new installation.
  • Use the canonical webhook receiver base: remove WEBHOOK_EXTERNAL_URL and hephaestus.webhook.external-url.
  • Remove existing sandbox claims before starting the new workers. Delete the remaining sandbox workloads before the new release starts.
  • Public activity pages count contributors without an account instead of naming them. To keep their names, turn on Show contributors without an account.
  • Stop every role before the models for practice scripts are deleted. The upgrade deletes the Decisions, Embeddings and Rerank connections.
  • Remove the review poll settings and the review Job metrics. Remove AGENT_POLL_INTERVAL and AGENT_CLAIM_BATCH_SIZE. Remove your own alerts on these metrics.
  • Stop every role before moving conversation turn leases to agent runs. The previous release cannot read the new turn leases.
  • Product feedback drafts use server egress: allow the selected instance model's provider through the server firewall. Keep a verified pre-upgrade database backup.
  • Stop every role for the conversation loop. This earlier entry introduced the loop. The sandbox cutover below replaces it.
  • Stop every role before moving conversation turns to sandboxes. Stop the old loop before migration, remove the old workloads, and run the template isolation gate.
  • Log records name their trace in trace.id and span.id: update your log queries, saved searches and collector rules that read context.traceId or context.spanId.
  • Log context uses consistent field names: update your log queries that read context.installation_id or context.proxy.apiProtocol.
  • Stop every role before clearing personal logins from GitHub App installation records. The previous server cannot read an installation record without a login.

Cut over to 1.0.0​

The migration entry Stop every role to cut over to 1.0.0 points to this section.

CAUTION: Changed schema

Do not use automatic Helm rollback for a stopped schema upgrade. An older role can corrupt data or fail against the changed schema. Keep every role stopped after a migration failure. Use Failed-migration recovery.

Follow these steps in this order. Each step names the page that owns it.

  1. Complete Validate the actual production data on an isolated copy.

  2. Pause pull-based deployment, if enabled. Wait for an active run to finish.

  3. Verify the target's signed release assets and chart. Record the current chart, profile, site values and replica counts.

  4. Stop every role. Stop the old conversation loop first, if present. Do phase 1 of Stopped schema upgrades with the source release: server.replicas, worker.replicas and webhook.replicas are 0. Wait until every role Pod, sandbox Pod and Job is gone. Foreground-delete the old claims and legacy Jobs, as Stop every role before moving conversation turns to sandboxes describes.

  5. Back up the database. Verify that the backup restores, as Backup and restore describes. Export the keys and keep them with the backup and the current signed release lock. You need this recovery point to roll back.

  6. Install the sandbox platform of the target release. On a one-host installation, run --check-platform and then --platform-only from the target bundle, as Install only the platform describes. On another cluster, apply k3s-agent-sandbox.yaml and wait for the controller:

    kubectl apply --server-side -f k3s-agent-sandbox.yaml
    kubectl -n agent-sandbox-system rollout status deployment/agent-sandbox-controller

    If this step restarts k3s, wait until PostgreSQL are ready before the next step. On a one-host installation, the platform files also renew the internal PostgreSQL certificate: do Refresh the internal PostgreSQL certificate before you go on.

  7. Change the Secret. First move the database logins, as step 3 of Give each database purpose its own login describes. hephaestus-keys.sh stops while the Secret still holds DATABASE_USERNAME or DATABASE_PASSWORD. Then run the target release's hephaestus-keys.sh so that the Secret holds every new key. Export the keys again and keep that copy with the restore point.

  8. Migrate. Do phase 2 of Stopped schema upgrades with the target chart and your changed site values. Save the migration Job's log. Require its Complete condition and no ERROR line in the log. If the Job fails, keep every role stopped and go to Roll back.

  9. Check the sandbox prerequisites on the migrated release:

    • The controller is ready and the chart's SandboxTemplate and SandboxWarmPool objects exist in the sandbox namespace.
    • The pre-pull DaemonSet is ready on every sandbox node: kubectl --namespace hephaestus rollout status daemonset/hephaestus-agent-image. It keeps the agent image on each node, because a cold pull takes longer than the start of a sandbox.

  10. Start the server. Do phase 3 of Stopped schema upgrades with the recorded server count. A server starts no sandbox until its isolation probe passes. The gate needs a running server, because a sandbox must reach the gateway. Thus, run the gate for each template now, as Sandbox templates describes. If it fails, fix the policy enforcement of the network plugin before you go on.

  11. On a one-host installation, run the target release's installer with --version, as Upgrade the host describes. Keep deployment automation paused until it finishes.

  12. If pull-based deployment is enabled, make its channel name the installed release, then resume reconciliation. Stopped major upgrades gives the reconciler's rules for a cutover.

  13. Load the alert rules of the target release into your Prometheus or vmalert. A chart upgrade brings them, but your own Prometheus does not read them until you load them (Alert rules).

  14. Do Check that it worked.

What happens to a review that a server still held when it stopped is in Stop every role before moving review leases to agent runs.

Provider reception stops during the window. Reception during a restart explains provider recovery. Keep the window short.

Roll back​

1.0 has no mixed-version window. An older image or helm rollback does not undo the migration.

  1. Keep every role stopped.
  2. Restore the verified backup from step 5 into the database. Restore its keys into the Secret. If migration fails gives the procedure. Use the previous release's signed lock.
  3. Install the previous release. For a pull-based host, Recover a stopped cutover gives the Helm and applied-record steps.
  4. Start the roles of the previous release only on the restored database.
  5. Report the recovery point and any lost writes.

The migration entry of a stopped change can give its own rollback notes.

Reach the database baseline from 0.x​

A release before 1.0.0-rc.1 must first reach that database baseline, as Prepare an older database for the current release describes. Then follow the cutover above.

Check that it worked​

  1. Check the migration Job's log and the readiness of each enabled role.
  2. Open Instance administration → Overview and read Configuration readiness.
  3. Resolve each item under Action required.
  4. Verify an authenticated read through the public gateway.
  5. Run the sandbox isolation gate against each template and require every check to pass.
  6. Run one practice review in Approve first mode, as First steps § 7 describes. Watch it end with a result.
  7. Ask one question in Hephaestus and verify that the answer completes.
  8. Read the dashboards Overview and Reviews and feedback, as Dashboards describes. Reviews leave the queue, and no role is missing from the scrape.
  9. Check these alerts for the first hours: HephaestusAgentQueueStalled, HephaestusAgentRunReaperFailing, HephaestusAgentSandboxReaperFailing, HephaestusReviewSandboxFailing, HephaestusSandboxGatewayRefusing. Alert response gives the action for each.
  10. Did the upgrade cross Stop every role before correcting unknown issue comment counts? Then ask the administrators of each affected workspace to run its backfill. Until it finishes, those issues show an unknown comment count.

Check Sizing and scaling before increasing capacity. Its measurements state their workload and limits. A successful upgrade does not qualify throughput or recovery time.

Drain the webhook streams before the upgrade​

Follow the provider event inbox cutover before you start 1.0. Use the provider inbox cutover procedure before starting the new server.

Stop every role before the provider resource repair upgrade​

Stop every server before the migration, then deploy the matching release. The old server cannot write the required authorization binding columns. Pending OAuth requests must restart after this upgrade. Integration administration describes the removal review and instance-admin release.