# Kubernetes The Helm chart installs Hephaestus on a Kubernetes cluster. A single host runs the chart on k3s. From 1.0, the chart is the supported deployment: [ADR 0058](https://github.com/hephaestus-build/Hephaestus/blob/main/docs/decisions/0058-the-helm-chart-is-the-only-supported-deployment-from-1-0.md) records the decision, and the [compatibility policy](/admin/compatibility-policy.md#support-for-1x) gives the terms. The worker runs practice reviews and conversation turns. The `small.yaml` profile enables it, and [Install](#install) ends when it is ready. On one host, k3s runs the cluster: [Install on one host (k3s)](#install-on-one-host-k3s) gives the complete procedure. Each release publishes the chart as a signed OCI artifact, `oci://ghcr.io/hephaestus-build/charts/hephaestus`. The chart source is in `deploy/chart/hephaestus/` of the repository. It uses the standard Helm chart format: `Chart.yaml`, `values.yaml`, `values.schema.json`, and `templates/`. It uses only stable Kubernetes APIs and the Gateway API `HTTPRoute` resource of version `v1`. ## Supported clusters Hephaestus supports the chart on a cluster with these items: * A [conformant](https://www.cncf.io/training/certification/software-conformance/) Kubernetes cluster at a minor version that the [Kubernetes project supports](https://kubernetes.io/releases/) when the Hephaestus release ships. The chart refuses a cluster below 1.35. A cluster at an older minor, for example 1.30 or 1.33, must be upgraded before Hephaestus runs on it ([ADR 0060](https://github.com/hephaestus-build/Hephaestus/blob/main/docs/decisions/0060-the-kubernetes-floor-follows-upstream-support.md)). * A network plugin that enforces NetworkPolicy for egress and for ingress. The worker's isolation probe stops the worker on a plugin that does not ([Sandbox prerequisites](#sandbox-prerequisites)). * A Gateway API controller for `HTTPRoute` `v1` with the four features in [Gateway routes](#gateway-routes). * A storage class for `ReadWriteOnce` volumes, for the bundled PostgreSQL and NATS. CI installs and upgrades the chart on Kubernetes 1.35, the lowest version that the chart takes, and on the latest minor version, 1.37, on `amd64`. It also runs 1.35 on `arm64`. A pull request runs only the latest minor version on `amd64`. When a change touches the chart or the sandbox runtime, the merge queue runs all of them before the commit reaches `main`. A nightly run repeats them on `main`. One more run uses three nodes, Calico for network policy, and an external PostgreSQL. Each run starts two workers with the `dev.yaml` profile, and each worker passes its isolation probe on that cluster. In the merge queue, the 1.35 run on `amd64` uses `small.yaml` and `small-16gib.yaml` instead, with one worker, on a runner of 16 GiB. Each upgrade must keep the PostgreSQL and NATS Pods: only a change to their own image or settings restarts them. On the latest minor version, the `amd64` run also backs up to an S3 server and restores to a point in time, as [Backup and restore](#backup-and-restore) describes. [Install on one host (k3s)](#install-on-one-host-k3s) states what the release workflow tests before it publishes a release. When a Kubernetes minor version reaches its end of life upstream, the next Hephaestus release can drop it. ## What the chart runs | Component | Kind | Role or image | | ------------------- | ----------------------------------------------- | ------------------------------------------------------------------------------------------------- | | `server` | Deployment, 1 replica, or 0 while stopped | Server role. | | `migration` | Job, one for each Helm revision | The server image in its migrate run mode. It applies the release's database migrations and exits. | | `webhook` | Deployment, 1 to 3 replicas, or 0 while stopped | Webhook role, as in [Runtime roles](/admin/runtime-roles.md). | | `webapp` | Deployment | The SPA, from the non-root webapp image. | | `worker` | Deployment, 2 replicas by default, optional | Worker role: practice reviews and conversations, in sandboxes ([Worker](#worker)). | | `postgresql` | StatefulSet, optional | The release PostgreSQL image, with `pg_partman` and pgBackRest. | | `postgresql-backup` | CronJobs, optional | Full and differential pgBackRest backups ([Backup and restore](#backup-and-restore)). | | `nats` | StatefulSet, optional | One NATS JetStream server. | | `route` | HTTPRoute | `/`, `/api`, `/webhooks`, and Slack interactivity. | | `priority` | PriorityClasses, cluster-wide | The priorities of the role Pods and of the sandboxes ([Worker § Priorities](#priorities)). | The PriorityClasses are the only cluster-wide objects of the chart. The account that installs the chart therefore needs permission to create PriorityClasses. Their names contain the release's namespace and name, so two releases never share one. On a shared cluster that does not allow this, use classes that exist ([Worker § Priorities](#priorities)). The migration Job is the only migration owner. No role runs Liquibase: each new Pod waits for the schema before it starts, for up to 10 minutes, while the previous release serves. The Job is a plain part of the release and not a Helm hook, because a hook would run before the bundled PostgreSQL exists. [Runtime roles](/admin/runtime-roles.md) describes the run mode and the wait. The server has one replica and the `Recreate` update strategy. Servers do not yet process provider events in order across replicas. Two server versions never serve the same schema at the same time. The `Recreate` strategy applies only to the server. The worker also connects to the database, and its rollout does not wait for the server. The webhook role has no database. [Database maintenance](#database-maintenance) gives the order of the roles for a change of the database or of a schema that needs every role stopped. ## Verify the release chart The chart package of a release names the release version, the source commit, and the release's five images by digest. Thus, you give Helm only your own values. The chart version and its `appVersion` equal the release version without the leading `v`. A stable promotion keeps the candidate's images and packages the chart with stable release metadata. The signed joint lock records the resulting chart digest. The chart source in the repository names no image, tag, or registry. You need [Cosign](https://docs.sigstore.dev/cosign/system_config/installation/) 3.1 or later, the Helm CLI, `curl`, and `jq`. The release signs the chart with Cosign 3.1 in the Sigstore bundle format. Cosign 2 reads that format only from version 2.6 and only with `--new-bundle-format`, and CI does not test it. You do not need Node.js or a checkout of the repository. Use the Helm version that [`.github/actions/setup-helm`](https://github.com/hephaestus-build/Hephaestus/blob/main/.github/actions/setup-helm/action.yml) pins. CI and the release workflow install that version. Contributors also need it: see the [local development prerequisites](/contributor/local-development.md#prerequisites). The pin is a build tool, not a qualified Helm version for an installation. No Helm plugin is necessary. Run these commands in the directory of your `site.yaml`. If one command fails, stop: do not install the chart. ``` VERSION=X.Y.Z CHART=ghcr.io/hephaestus-build/charts/hephaestus # 1. The chart digest, from the joint lock of the release. DIGEST=$(curl -fsSL "https://github.com/hephaestus-build/Hephaestus/releases/download/v$VERSION/release-v$VERSION.chart.json" | jq -er .chart.digest) # 2. The release workflow signed that digest. cosign verify "$CHART@$DIGEST" \ --certificate-identity 'https://github.com/hephaestus-build/Hephaestus/.github/workflows/release.yml@refs/heads/main' \ --certificate-oidc-issuer https://token.actions.githubusercontent.com # 3. The one pull: Helm gets the chart by that digest, never by a tag. helm pull "oci://$CHART@$DIGEST" --untar --untardir "chart-$VERSION" # 4. The signed chart is the chart of this release. helm show chart "chart-$VERSION/hephaestus" | grep -x "version: $VERSION" >/dev/null ``` The joint lock in step 1 is only a lookup for the chart digest. You do not verify it. Step 2 refuses every digest that the release workflow did not sign. That identity is the same for every release, so step 4 binds the chart to the release that you asked for. A version tag can move, and it is not a release signal. The `chart-$VERSION/hephaestus` directory is the verified chart, with its `profiles/`. Step 3 stops if that directory already exists. [Install](#install) and [Upgrades](#upgrades) use it. [Release assets](/admin/release-image-lock.md#release-assets) gives the purpose of each signed asset of a release. ## Create the Secret The chart reads secret values only from one Secret in the release namespace. It never writes a secret value. Set `secret.name` to the name of that Secret. The chart reads fixed key names from it. [Secret custody and rotation](/admin/secret-custody.md#inventory) lists each key, the roles that read it, and the command that generates it. That page also tells you which keys to keep outside the cluster for a restore, and how to replace each key. ### PostgreSQL TLS Secret Each role connects to PostgreSQL with `sslmode=verify-full`. The role verifies the server certificate against a CA, and it checks that the certificate names the host that the role connects to. Set `postgresql.tls.existingSecret` to an existing Secret in the release namespace. The chart refuses to render without it. The chart creates no certificate, no key, and no CA. | Key | Bundled PostgreSQL | External PostgreSQL | Mounted in | | -------------------- | ------------------ | ------------------- | ----------------------- | | `ca.crt` | Yes | Yes | Server, webhook, worker | | `tls.crt`, `tls.key` | Yes | No | Bundled PostgreSQL only | The server, the webhook role, and the worker receive only `ca.crt`, read-only, at `/etc/hephaestus/postgresql-ca`. No application role receives a private key. Backup Jobs use a separate client Secret, `postgresql.backup.clientTlsSecret`. It holds `ca.crt`, `tls.crt` and `tls.key` for the common name `hephaestus-backup`. The PostgreSQL CA signs it. The chart mounts its certificate and key as `backup.crt` and `backup.key`. Do not put the private key of the CA in this Secret. For the bundled PostgreSQL, the certificate in `tls.crt` must name `-postgresql..svc`. `-postgresql` follows the object name rule in [Install](#install). The roles connect to exactly this name, without a cluster domain. The chart mounts `tls.crt` and `tls.key` into the PostgreSQL pod with the mode `0440`. The files belong to root and to the group 999, and PostgreSQL accepts a key with these permissions. [TLS certificates](/admin/secret-custody.md#tls-certificates) gives the renewal of each certificate. A new CA needs the [database maintenance](#database-maintenance) procedure. ## Gateway routes The Gateway API is a platform prerequisite of the chart. The platform operator installs and operates these items: * The Gateway API resource definitions and a Gateway controller. * The Gateway, its TLS certificate, and its HTTP to HTTPS redirect. The chart installs none of them. It attaches one `HTTPRoute` to the Gateways in `route.parentRefs`. `route.parentRefs` must name at least one Gateway. | Request path | Backend | Path change | | ------------------------------- | --------- | ---------------------------------------------------- | | `/webhooks/slack/interactivity` | `server` | None | | `/webhooks` and below | `webhook` | None | | `/api` and below | `server` | `/api` is removed | | All other paths | `webapp` | None. The webapp serves the SPA for an unknown path. | The server serves the API without the `/api` prefix, as behind Traefik in Compose. A standard `Ingress` cannot remove a path prefix, so the chart does not use one. The Gateway controller must support these `HTTPRoute` features: * `URLRewrite` with `ReplacePrefixMatch`, for `/api`. * `ResponseHeaderModifier`, for `Strict-Transport-Security` on each rule. * `HTTPRouteRequestTimeout`, for `timeouts.request` on `/api`. * `HTTPRouteBackendTimeout`, for `timeouts.backendRequest` on `/api`. If the controller does not support a feature, the route does not work as described. The chart cannot detect this. Confirm all four features for the controller that you select. The `/api` rule sets both `timeouts.request` and `timeouts.backendRequest` to `0s`. The Hephaestus live stream stays open for much longer than one request. The two timeouts are separate: the request timeout covers the client request, and the backend timeout covers the full backend response. If a rule omits one of them, the controller selects that behavior. A zero value asks the controller to disable the timeout. A controller that cannot disable it can use its longest possible value instead. Thus, the `0s` values do not guarantee a stream without a time limit. The other three rules set no timeout and no retry. Set `trustedProxies` to a Tomcat regular expression that matches only the Gateway data plane addresses. The server does not start in production with a blank value. The Gateway API has no portable request body limit or idle timeout. Configure these values on the Gateway: * A request body limit of 25 MiB on `/webhooks` and `/api`. * An idle timeout of more than 20 seconds for the Hephaestus live stream. An idle timeout of more than 20 seconds does not replace the request and backend timeouts. No Gateway controller or version is qualified yet. A qualification must include these tests on the selected controller and its idle settings: * A long Hephaestus live stream. * A long WebSocket connection through the Gateway. * A cancellation by the client during a stream. ## Network policy Network policy is always on. No value turns it off. Each pod of the release starts with no ingress and no egress. The chart then opens only these paths: | From | To | Port | | ------------------------------------------------------------------- | --------------------------------------------------------- | ---------------------------------- | | `networkPolicy.gateway` pods | `server`, `webhook`, `webapp` | 8080 | | `server`, `webhook`, `worker` | PostgreSQL | 5432, or the external port | | `server`, `webhook` | NATS | 4222, or the external port | | `server`, `worker` | `networkPolicy.externalEgress` | 443 | | `server`, if `config.email.host` is set | `networkPolicy.externalEgress` | `config.email.port` | | `worker` | The Kubernetes API, `networkPolicy.kubernetesApi` | Its port | | `worker`, if `networkPolicy.workerGatewayPorts` is set | `networkPolicy.gateway` pods | `networkPolicy.workerGatewayPorts` | | `server`, `worker`, for each item of `networkPolicy.modelEndpoints` | Its `cidr` | Its `ports` | | Review sandboxes and conversation hosts | `worker` | 8081, the sandbox gateway | | PostgreSQL and the backup Jobs, with backups on | `postgresql.backup.s3`, or `networkPolicy.externalEgress` | `postgresql.backup.s3.port` | | The backup Jobs, with backups on | PostgreSQL | 8432, the pgBackRest TLS server | | All pods | `networkPolicy.dns` pods | 53 | | `networkPolicy.metrics` pods | All application roles | 9090 | Sandboxes are in the sandbox namespace, where every pod also starts with no ingress and no egress. Turn sandboxes get no other path. A model endpoint that this cluster's own Gateway serves resolves to a public address, but the connection ends at a Gateway Pod. That private address is outside `networkPolicy.externalEgress`, so the worker cannot reach the endpoint. Set `networkPolicy.workerGatewayPorts` to the container ports of the Gateway Pods, for example `[8443, 8000]` for the Traefik of the one-host platform. `networkPolicy.gateway` must name the namespace and at least one pod label of the Gateway data plane. A namespace without labels would admit every pod in it, so the chart refuses it. The same rule applies to `networkPolicy.metrics` when you set its namespace. ### A model endpoint on the cluster's own ingress The worker reaches AI providers through `networkPolicy.externalEgress`, which leaves out private address ranges. If the host name of a model endpoint resolves to the cluster's own ingress, the connection ends at an ingress Pod with a private address, and the policy refuses it. An example is a model proxy that the same cluster serves. Add a NetworkPolicy that lets the worker Pods reach the ingress Pods. NetworkPolicies add up, so this policy does not change the chart's policies. Use the container ports of the ingress Pods, not the Service ports: the policy applies after address translation. For Traefik in the namespace `traefik`, with container ports 8000 and 8443: ``` apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: worker-to-ingress namespace: hephaestus spec: podSelector: matchLabels: app.kubernetes.io/name: hephaestus app.kubernetes.io/component: worker policyTypes: ["Egress"] egress: - to: - namespaceSelector: matchLabels: kubernetes.io/metadata.name: traefik podSelector: matchLabels: app.kubernetes.io/name: traefik ports: - { port: 8443, protocol: TCP } - { port: 8000, protocol: TCP } ``` ### A model endpoint on a private network A model endpoint outside the cluster can have a private address, for example a vLLM server on the LAN. The application and the network policy both refuse private addresses by default, so you must allow the endpoint in both places: ``` config: # The application: host names, IP addresses, or CIDR ranges that a provider base URL can use. llmPrivateHosts: [vllm.lab.example] # Only if the endpoint serves plain HTTP. llmPrivateHostsAllowHttp: true networkPolicy: # The network: the address range and the ports of the endpoint. modelEndpoints: - cidr: 10.20.0.0/16 ports: [8000] ``` If only `config.llmPrivateHosts` names the endpoint, a connection saves, and then the request times out at the network. The chart cannot compare the two lists, because a host name is not an address range. [Connect an AI provider](/admin/ai-providers.md#model-endpoints-on-your-own-network) tells what the application allows. A `cidr` must be in 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 100.64.0.0/10, or fc00::/7, and `ports` must name at least one port. Thus, no item can open the link-local range of the cloud metadata service. Use `modelEndpoints` only for addresses outside the cluster. Do not list a range in `config.llmPrivateHosts` that covers the Pod or Service range of the cluster. A range that covers the Pod's own address stops the server or the worker at startup, but the server cannot detect the Service range. A NetworkPolicy `ipBlock` does not reliably match Pod addresses. For an endpoint in this cluster, use `networkPolicy.workerGatewayPorts` or a policy with a `podSelector`, as the previous section shows. The management listener binds to the pod address. The kubelet probes it, and the metrics scraper can read it. No Service and no route expose it. The external egress rule excludes private, link-local, and shared address ranges by default. An external database or broker in a private range needs `postgresql.external.cidr` or `nats.external.cidr`. A rendered policy does not show enforcement. Each network plugin needs its own positive and negative traffic tests before the chart is supported. Network policy controls which pods can connect. It does not encrypt traffic. The connections to PostgreSQL use TLS ([PostgreSQL TLS Secret](#postgresql-tls-secret)). These internal channels send plain text: * The connections from the server and the webhook role to the bundled NATS. * The connection from the worker to the server. * The connections from sandboxes to the worker gateway. The bundled PostgreSQL still accepts a connection without TLS. The roles never use one, and network policy admits only the roles. On a cluster with several nodes, these channels cross the node network. Use a network plugin that encrypts traffic between nodes, for example WireGuard in Calico or Cilium, when that network is not private to the cluster. ## Storage and retention | Data | Bound | Setting | | ------------------ | ---------------------------------------------------------------------------------------------------- | ------------------------------------------ | | PostgreSQL | Volume size | `postgresql.persistence.size` | | JetStream | Broker `max_file` and stream storage budget | `nats.jetstream.maxFileBytes` | | JetStream volume | `max_file` plus 25 % | Derived from `nats.jetstream.maxFileBytes` | | Webhook streams | Age and size for each stream, and one start that may delete stored messages to apply a smaller bound | `webhook.stream.*` | | Server Git mirrors | `emptyDir` size limit on the node | `server.gitCache.sizeLimit` | | Worker Git mirrors | `emptyDir` size limit on the node, per worker | `worker.gitCache.sizeLimit` | | AI spend ledger | Retention | `config.llmUsageRetention` | | Temporary files | `emptyDir` size limit | Fixed in the templates | One value sets both the broker `max_file` and the webhook role's stream storage budget. The webhook role refuses to start when the stream bounds add up to more than that budget. [Webhook ingestion operations](/admin/webhook-ingestion-operations.md) gives the durability guarantee. The `small.yaml` profile sets GitHub to 4 GiB and each GitLab, Slack and Outline stream to 512 MiB. All four streams keep deliveries for at most seven days. Their byte bounds total 5.5 GiB. The broker file budget is 8 GiB. A fresh bundled broker claims 10 GiB, including the 25 % storage allowance. The profile overrides the shorter Slack and Outline ages through `webhook.stream.maxAgeByStream`. It retains 30-day durable expiry through `config.consumerInactiveThreshold`. A byte bound can remove messages before seven days. Measure peak traffic before you select these bounds. For an existing installation, keep its current `nats.jetstream.maxFileBytes` in your site values before you apply this profile. Do not shrink an existing PVC or change the StatefulSet claim template to adopt the smaller broker budget. Do not lower the broker budget below its stored data. Before you shorten stream limits, verify an off-host backup and test its restore. Then use the [guarded stream-limit procedure](/admin/webhook-ingestion-operations.md#changing-a-bound-on-a-stream-that-already-exists). A higher limit cannot recover deleted messages. The server updates expiry only on durables it uses. Review orphan durables separately before you change or delete them. The Git mirrors are caches. A restarted server or worker clones them again, so they use node storage and no volume claim. The node needs that much free ephemeral storage, and the container limits include it. The bundled NATS is one JetStream server, and every stream has one replica. The application has no setting for more replicas. A lost NATS volume loses each provider delivery that the broker accepted and the server had not yet processed. That window is usually seconds, and it grows while the server is stopped or behind. Use a storage class whose volumes survive the loss of a node, and back up the volume as [Backup and restore](/admin/backup-restore.md) describes for Compose. ### Disk on one host Each upgrade pulls the images of the new release, and the chart removes no image. The kubelet removes unused images when the image filesystem is fuller than `imageGCHighThresholdPercent` (85 % by default), until it is below `imageGCLowThresholdPercent` (80 %). On one host, set lower thresholds and a maximum age, so that the images of old releases go before the disk fills. On k3s, put a KubeletConfiguration file whose name ends in `.conf` into `/var/lib/rancher/k3s/agent/etc/kubelet.conf.d/`, then run `sudo systemctl restart k3s`: ``` apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration imageGCHighThresholdPercent: 75 imageGCLowThresholdPercent: 65 imageMaximumGCAge: 168h ``` The thresholds count the whole filesystem, also data that is not an image. If other data keeps the disk above the low threshold, the kubelet logs that image garbage collection failed, and it removes every unused image. The kubelet never removes an image that a Pod uses. ## Pod security Each pod meets the Kubernetes Restricted Pod Security Standard: * A non-root user, `RuntimeDefault` seccomp, and no privilege escalation. * All capabilities dropped and a read-only root filesystem. * No host path, host network, or container runtime socket. * No service account token, except on the worker. Its Role reaches only the sandbox namespace. The webapp runs as user and group 101 and listens on port 8080. Its only writable path is a 16 MiB volume at `/tmp/hephaestus-webapp`. The image writes its generated files, its nginx PID file, and its temporary directories there. `/tmp` itself and the assets stay read-only. The bundled PostgreSQL runs as the image user 999 and owns its volume through `fsGroup`. The volume is at `/var/lib/postgresql`, as [ADR 0038](https://github.com/hephaestus-build/Hephaestus/blob/main/docs/decisions/0038-postgresql-18-release-baseline.md) decides. The image keeps its versioned data directory below it, and the chart does not set `PGDATA`. The image's entrypoint starts PostgreSQL with `ssl=on` and the certificate and key from the TLS Secret, read-only at `/etc/postgresql/tls`. No real volume driver has shown this configuration yet. First initialization, restart, and restore need a start of the release image on a volume before this path counts as working. Every container of the release has CPU, memory, and ephemeral storage requests and limits. The schema refuses a zero value. Sandbox containers have CPU, memory, and disk limits ([Sandbox sizes](#sandbox-sizes)). They request the memory that their kind of sandbox was measured to use, below their limit ([Sandbox sizes](#sandbox-sizes)). They request a tenth of their CPU limit, because the CPU limit is a share of CPU time, not a reserved core ([Sizing](/admin/sizing.md#review-sandbox-limits)). ## Profiles Four values files in `profiles/` set capacity for one use. All four use the same templates. Each release chart contains them in `chart-$VERSION/hephaestus/profiles/` after you [verify the release chart](#verify-the-release-chart). Give the profile as the first `-f` file, before your own values, as [Install](#install) shows. | Profile | Use | PostgreSQL and NATS | | ------------------------------------------ | ------------------------------------------- | ------------------------ | | `dev.yaml` | One workstation cluster | Bundled, small volumes | | `preview.yaml` | One pull request preview | Bundled, small volumes | | `small.yaml` | One small cluster, for example one k3s node | Bundled, bounded streams | | `small-16gib.yaml`, on top of `small.yaml` | One k3s node of 16 GiB | Bundled, bounded streams | `small.yaml` fits one node of 4 vCPU, 32 GiB and 150 GiB of disk, with one worker that runs one review and up to 8 conversation turns at a time. A chart test adds up what that node holds during an upgrade, with backups on: | Part | Memory | CPU requests | Disk | | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------- | ------------ | ----------------------------------------------------------------------- | | Server, webhook role, worker, webapp, PostgreSQL with pgBackRest, NATS | 9.8 GiB | 1.25 | The volumes of PostgreSQL and NATS, and bounded caches | | An upgrade: the new worker, webhook and webapp Pods beside the old ones, the migration Job, the stanza Job and one backup | 5.8 GiB | 1.00 | The new Pods' caches | | The sandbox quotas during the upgrade: in the reviews tier one review and the isolation probes of both workers, in the conversations tier the conversation host, its two pooled turn sandboxes and 8 running turns of each worker | 5.8 GiB | 1.00 | The disk requests of the sandbox Pods ([Sandbox sizes](#sandbox-sizes)) | | Total, against the node less platform reservations and the hard eviction floor | 21.4 GiB of 30.25 GiB | 3.25 of 3.5 | 80 GiB of the 110 GiB left beside images and review workspaces | The worker upgrades by a rolling update with a surge of one. The new worker passes its isolation probe and opens its conversation host's command channel before the old worker drains. Hephaestus keeps answering during the upgrade. The old worker finishes its running review within its drain of up to 60 minutes. The new worker's review waits for that review's room in the reviews quota. A turn sandbox requests 25m of CPU, so a second conversation host and turn pool fit beside the old ones. The table gives the measured resident memory of each role and the chart's sizes. Each JVM role requests all of its memory limit, because the heap is sized from the limit, and the limit is the measurement plus headroom. PostgreSQL and its backup containers also request their full memory limits. CPU remains burstable, so these Pods keep the Burstable quality-of-service class. NATS requests less than its limit, because JetStream's caches grow above the request. | Role | Measured | Request | Limit | | -------------------------------------------------- | -------------- | -------- | -------- | | Server | 1.5 GiB | 2.5 GiB | 2.5 GiB | | Worker | 0.8 to 1.4 GiB | 2 GiB | 2 GiB | | Webhook role | 1.0 GiB | 1.25 GiB | 1.25 GiB | | PostgreSQL, at rest in the earlier staging capture | 0.7 GiB | 2 GiB | 2 GiB | | pgBackRest | Not measured | 512 MiB | 512 MiB | | NATS, with JetStream's memory and file caches | 1.3 to 2.7 GiB | 1.5 GiB | 3 GiB | The server's and the worker's caches have a bound of 10 GiB each, so that the node does not run out of disk and evict a draining worker. On a node of 16 GiB, the Pods of an upgrade with `small.yaml` alone do not fit beside a running review: they stay pending, the previous release keeps serving, and the upgrade rolls back when its timeout ends. For one node of 16 GiB, apply `small-16gib.yaml` after `small.yaml`: ``` --values deploy/chart/hephaestus/profiles/small.yaml \ --values deploy/chart/hephaestus/profiles/small-16gib.yaml ``` The overlay lowers the NATS request to 512 MiB and keeps its limit at 3 GiB. It also changes worker updates to `Recreate`, with a drain of at most 10 minutes. The worker still runs one review and up to 8 conversation turns at a time, as in `small.yaml`. A second worker's conversation host and turn pool do not fit in 16 GiB beside the first: the upgrade stops reviews and conversation turns for the drain, the new worker's start and its isolation probe. The webhook role keeps its rolling update: it receives provider deliveries during an upgrade. A chart test adds up what one node of 16 GiB holds, with backups on: | State | Memory | CPU requests | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------- | ------------ | | Steady: every role, one backup, one review, the conversation host, two pooled turn sandboxes and 8 running turns | 11.8 GiB of 13.9 GiB | 1.75 of 3.5 | | The same, with the isolation probe | 12.5 GiB | — | | An upgrade: the old worker gone, the webhook and webapp surge, the migration and stanza Jobs, the new worker's probe, and the old worker's pooled turn sandboxes | 13.2 GiB of 13.9 GiB | 2.40 of 3.5 | The table counts requests. Its node budget uses the recorded Node capacity of 16 372 036 KiB, less the current platform reservations and hard eviction floor. It does not qualify the peak memory of running work. Sandbox Pods may use more than they request, up to their limits. If every sandbox Pod used its limit at the same time, the node would be full. A chart test derives the conversation turns of both single-node profiles: 8, the worker's automatic number, as long as one more turn sandbox would still fit. An earlier staging capture recorded these values on one node of 16 GiB with k3s 1.37. Its request figures precede the PostgreSQL memory-request change: * At rest, the release requests 10.3 GiB of 15.6 GiB and uses about 5.9 GiB on the host: the server 2.0 GiB, the worker 1.3 GiB, the webhook role 0.8 GiB, PostgreSQL 0.7 GiB and NATS 60 MiB. * An upgrade that recreated the worker and restarted PostgreSQL stopped reviews for about 4 minutes, with the isolation probe. The webhook role kept receiving. * A point-in-time restore of a database of 1.8 GB took 152 s. The release served again 5 minutes 41 seconds after the restore started. * The peak during a review and a conversation turn is not measured yet. For a cluster with several nodes, start from `small.yaml`, and set the workers, their reviews, their quota and the external services in your own values file. [ADR 0035](https://github.com/hephaestus-build/Hephaestus/blob/main/docs/decisions/0035-pull-request-previews-are-label-gated.md) owns what a preview can run. The preview profile sets only capacity. ## External PostgreSQL and NATS Set `postgresql.bundled=false` to use your PostgreSQL server. That server must have the `pg_partman` extension. The chart connects to `postgresql.external.host` and `postgresql.external.port` with `sslmode=verify-full`. The server certificate must name that host. The Secret in `postgresql.tls.existingSecret` needs only `ca.crt`, the CA that signed the server certificate. The chart has no setting for a connection without certificate verification or without TLS. Set `nats.bundled=false` to use your NATS JetStream broker. Put the broker on a private network that only this release can reach. The released NATS client verifies the certificate chain of a `tls://` broker, but not the host name in the certificate. Thus, TLS alone does not prevent a connection to another broker that has a certificate from the same CA. The bundled broker stays inside the release namespace, and network policy admits only the server and the webhook role. The application has no setting for the stream replica count. A three-server JetStream cluster therefore keeps one replica of each stream. ## Install on one host (k3s) This procedure installs Hephaestus on one Linux host with [k3s](https://docs.k3s.io/) and the Helm chart. One command, `k3s-install.sh`, does the install. It checks the host, installs the pinned k3s and the platform, and keeps the release Secret. Then it verifies the release chart and installs it. It uses the `small.yaml` profile, the bundled PostgreSQL and NATS, and one worker that runs reviews. A second run of the same release changes nothing that the first run made. A run with the files of a newer release upgrades the host. Before it publishes a release, the release workflow runs `k3s-install.sh` on Ubuntu 24.04 on `amd64` and on `arm64`. It runs the command twice, then an upgrade that restarts every role except PostgreSQL and NATS, whose Pods must stay. That run differs from your install in these points: * A local CA issues the Gateway certificate, because `Let's Encrypt` cannot reach a CI runner. CI does not test the HTTP-01 challenge of the `Let's Encrypt` issuer. * It skips the DNS check, because a runner has no public host name. * It skips the clock check: a hosted runner takes its time from the hypervisor and can report no NTP sync. * A runner has 16 GiB of memory, so the installer adds the `small-16gib.yaml` profile. * It checks the shutdown settings that the kubelet and logind use. It does not reboot the host. ### The platform files Each release publishes the files of the one-host install as signed assets. | Asset | What it is | | ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | | `k3s-install.sh` | The installer. It is a POSIX shell script, and it needs no Node.js and no checkout of the repository. | | `k3s-platform.yaml` | The pins and the platform add-ons. It goes to the k3s manifests directory. | | `k3s-certificates.yaml` | The `Let's Encrypt` issuer, the Gateway certificate, and the PostgreSQL CA and server certificate. The installer writes your host name and address into it. | | `k3s-config.yaml` | The k3s configuration: the kubelet configuration file, and no bundled Traefik. | | `k3s-kubelet.yaml` | A limit of 4096 processes for each Pod, the shutdown grace period, the memory and CPU reserved for k3s and the host, and the eviction thresholds. | | `k3s-logind.conf` | How long logind lets the kubelet hold a shutdown. | | `k3s-journald.conf` | Host journal limits: 512 MiB on disk, 128 MiB in memory, and seven days. | | `k3s-values.yaml` | The chart values of every one-host install: the Gateway, the Traefik Pods, the Secret names, the sandbox namespace, and reviews with repository checkouts. | `k3s-platform.yaml` contains these add-ons: * Traefik as the Gateway controller on the host's ports 80 and 443. Its chart creates the Gateway `traefik/traefik-gateway` and a redirect from HTTP to HTTPS. * cert-manager, with Gateway API support. The `pin: k3s` line of `k3s-platform.yaml` sets the k3s release, and its two `HelmChart` objects set the versions of Traefik and cert-manager. The file pins each chart by version and each image by digest. The Gateway API resource definitions come from the `gateway-api-crd` chart that k3s includes, so no second owner installs them. k3s deploys each file in `/var/lib/rancher/k3s/server/manifests/`, and it applies a file again every 15 seconds until the apply succeeds. Thus, cert-manager and the Gateway start some minutes after k3s. The installer does not run the application itself as a k3s `HelmChart`. The k3s Helm controller does not wait for the migration Job, and it does not roll back a failed upgrade. The limit of 4096 processes applies to each Pod on the host, the server and the worker included. On Compose, the server and the worker had a limit of 512 each, so a lower limit for every Pod could stop them. CI's clusters run with the same 4096. When the host shuts down, the kubelet stops the Pods first. The configured budget is 5 minutes: 4 minutes for the application Pods, then 1 minute for Traefik and CoreDNS. PostgreSQL stops within its termination grace period of 120 seconds, inside the first 4 minutes. Ubuntu sets the logind limit to 30 seconds, so `k3s-logind.conf` raises it to 5 minutes. At the end of the budget, a worker stops the reviews that it runs. They run again after the start. Pods cannot request all of the host's memory. `k3s-kubelet.yaml` reserves 1 GiB and 250m CPU for k3s, which runs the API server, the kubelet and containerd in one process of about 1 GB. It reserves 512 MiB and 250m CPU for the host's own services. The kubelet evicts Pods when less than 256 MiB of memory is available, before the kernel's out-of-memory killer stops a process on the host. For a Node API capacity of exactly 16 GiB and 4 vCPU, this leaves 14 592 MiB (14.25 GiB) and 3.5 CPUs for Pods. At the recorded capacity of 16 372 036 KiB used in the profile table, Pods get about 13.9 GiB. The installer restarts k3s when this file changes. After the node becomes ready, it reads the kubelet configuration endpoint and Node allocatable. It stops before the application upgrade if the configured policy is absent or overridden. Allocatable must equal measured capacity less the reservations and the hard memory floor. The soft threshold does not reduce allocatable again. Only Pod allocatable is enforced. The reservations do not impose hard memory caps on the host or k3s system slices. To repeat these read-only cluster checks with your current kubeconfig, run `sh k3s-install.sh --verify-kubelet`. The command uses Helm locally to parse the policy; it does not change the cluster. After a restart, verification allows 31 checks with two-second pauses for the Node status and configuration endpoint to converge. Soft eviction is disabled until stateful-service requests and admission protection are qualified. For memory eviction, usage above requests ranks before Pod priority. Thus, a higher priority cannot protect PostgreSQL or NATS above their requests from a review below its request. The existing hard thresholds remain emergency recovery boundaries. The kubelet targets an additional 256 MiB of reclaim and holds a pressure condition for five minutes. Application admission must stop new background work before those boundaries. Container logs rotate at 10 MiB with at most five files per container. Image garbage collection starts at 75 percent disk use and targets 65 percent. It does not reclaim live volumes or database files. The installer bounds the host journal and restarts journald only when its policy changes. Host swap must be disabled, including at boot. The installer rejects active swap, and the kubelet requires `failSwapOn=true` and `NoSwap`. These policies do not qualify a smaller container memory limit or a higher review count. Before enabling either, measure synchronized node peaks, pressure, queue waits, and OOM counters under the reference load. The verifier accepts the declared Ubuntu 24.04/kernel 6.8 GA baseline and rejects unknown OS or kernel families. The optional `--allow-unqualified-host` flag permits an explicitly authorized rehearsal on another family. It preserves all effective-policy and allocatable checks. A matching host family alone does not prove installation, pressure behavior, or loaded capacity. The current proof consists of policy/parser fixtures, read-only Node API observations, and rendered chart checks. The release host smoke targets Ubuntu 24.04 on both architectures; this change has not run a full host installation. Before target-host rollout, the release lane must rehearse the exact assets and pinned k3s release on disposable Ubuntu 26.04/kernel 7 with 16 GiB. Use the qualification override for that rehearsal and record the actual OS, kernel, CPU count, capacity, and allocatable. Run the installer, then repeat it and verify stateful Pod identity and service continuity. The second run must preserve unchanged configuration and avoid unnecessary k3s or journald restarts. Verify disabled swap and its boot activation settings. Apply pressure with PostgreSQL and NATS above their requests and review/turn Pods below theirs. Require application backoff and recovery before hard eviction, with no database/broker loss or OOM. Observe container and journal rollover, image GC at 75/65, and image pull/start latency. Retain the command logs, effective configuration, pressure/OOM counters, and synchronized footprint samples as release evidence. ### Requirements * A host of 4 vCPU, 32 GiB of memory, and 150 GiB of disk, as [Profiles](#profiles) gives for `small.yaml`. On a host of 16 GiB, the installer adds `small-16gib.yaml`, and each upgrade stops reviews and conversations for a while. * Ubuntu 24.04 with its 6.8 GA kernel on `amd64` or `arm64`. Other OS or kernel families require explicit qualification with `--allow-unqualified-host`. * A DNS A record for your host name that points to the public address of the host. * Ports 80 and 443 open from the internet, for `Let's Encrypt`, OAuth callbacks, and provider webhooks. Keep port 6443, the Kubernetes API, and port 10250, the kubelet, closed to the internet. * If `ufw` is active, rules that let in ports 80 and 443 and the k3s networks: `sudo ufw allow 80,443/tcp`, `sudo ufw allow from 10.42.0.0/16`, and `sudo ufw allow from 10.43.0.0/16`. * Outbound HTTPS to the destinations in [Network egress](/admin/network-egress.md#one-host-on-k3s). * `curl`, `jq`, `openssl`, `python3`, Cosign, and the Helm CLI, as [Verify the release chart](#verify-the-release-chart) describes. Ubuntu includes `python3`. * A clock that NTP keeps synchronized, as [Clocks](/admin/install.md#clocks) describes. * A GitHub OAuth app for sign-in, as [First steps § 1](/admin/first-steps.md#1-configure-a-login-provider) describes. Its callback URL is `https:///api/login/oauth2/code/github`. ### What the installer checks The installer stops before it changes the host when one of these checks fails. Each message tells you what to change. | Check | Condition | | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Input | The host name, the address, the OAuth client ID, and the admin list have a strict format. | | Size | The host has 4 CPUs and at least 15 GiB of memory. Below 30 GiB, the installer adds the `small-16gib.yaml` profile. | | Clock | NTP synchronizes the clock. | | Swap | No swap device or file is active. The kubelet also refuses swap. | | Ports | No other program listens on port 80 or 443. The installer checks this before k3s runs. | | Host firewall | If `ufw` is active, it allows `80/tcp` and `443/tcp` by a port rule or by the `Nginx`, `Apache`, or `WWW` profile. If `firewalld` is active, the zone of the default route allows `http` and `https`, or ports 80 and 443. Otherwise the installer stops and gives the command to run. | | DNS | The host name resolves to this host. Before k3s runs, the installer serves a random token on port 80 for a moment, and it fetches the token through the host name. | The DNS check proves only that the host name leads to this host. The request can stay on the host, so it does not prove that the internet reaches port 80. A cloud firewall or a security group of your provider is yours to open, and the installer cannot check it. cert-manager needs port 80 from the internet to get the certificate. The installer waits for the certificate for 10 minutes, and then it stops with the command that shows the reason. If your network cannot reach its own public address, the DNS check fails. Then run the installer with `--skip-check dns`. ### 1. Get the release files 1. Make a directory for the release files. The installer keeps its state there: `site.yaml`, the verified chart, and a `kubeconfig` file. ``` VERSION=0.95.0 # the release, without the leading "v" mkdir -p ~/hephaestus && cd ~/hephaestus ``` 2. Download each asset and its signature, and verify the signature. Cosign stops with an error if a file does not come from the release workflow: ``` release="https://github.com/hephaestus-build/Hephaestus/releases/download/v$VERSION" for file in k3s-install.sh k3s-platform.yaml k3s-certificates.yaml k3s-config.yaml k3s-kubelet.yaml k3s-logind.conf k3s-journald.conf k3s-values.yaml; do curl -fsSLO "$release/$file" curl -fsSLO "$release/$file.sigstore.json" cosign verify-blob --bundle "$file.sigstore.json" \ --certificate-identity https://github.com/hephaestus-build/Hephaestus/.github/workflows/release.yml@refs/heads/main \ --certificate-oidc-issuer https://token.actions.githubusercontent.com \ "$file" done ``` ### 2. Run the installer 1. Type the client secret of your GitHub OAuth app. The shell does not show it: ``` read -rs GITHUB_OAUTH_CLIENT_SECRET && export GITHUB_OAUTH_CLIENT_SECRET ``` 2. Run the installer. Give your host name, and the address that gets the expiry notices of `Let's Encrypt`. Also give your OAuth client ID and your numeric GitHub user ID: ``` sh k3s-install.sh --version "$VERSION" --host hephaestus.example.com --email ops@example.com \ --github-client-id --bootstrap-admins github: ``` For a first try, add `--staging`: the `Let's Encrypt` staging service has higher rate limits, and no browser trusts its certificate. Later, run the installer again without `--staging`, with the explicit certificate choice described below. Delete the Secret `traefik/hephaestus-tls` to get a trusted certificate. [First steps § 2](/admin/first-steps.md#2-name-the-first-admin) explains the admin list. 3. Keep the required keys of the release Secret outside the host, as [Keep the required keys outside the cluster](/admin/secret-custody.md#keep-the-required-keys-outside-the-cluster) describes. For `kubectl` on this host, run `export KUBECONFIG=~/hephaestus/kubeconfig` first. 4. Open `https://` and sign in as the bootstrap admin. 5. Continue with [First steps § 4](/admin/first-steps.md#4-connect-repository-data). The installer keeps every key of the release Secret `hephaestus-app` and generates only the keys that it lacks. It generates each key as [Secret custody and rotation](/admin/secret-custody.md#generation-and-rotation) gives it. If the PostgreSQL volume exists and the Secret lacks the database login or an encryption key, the installer stops: restore the Secret from your backup. The installer writes `site.yaml` only when the file does not exist. It holds the host name, the Kubernetes API endpoint, the OAuth client ID, the admin list, and `config.gitCheckoutEnabled: true`. Change the file to change these values, and run the installer again. On a repeat run, select certificates explicitly: * For standard apex certificates, pass `--standard-certificates`. Omit `--staging` to use the production issuer. * For standard workspace certificates, repeat `--workspace-subdomains --dns01-issuer NAME`. * For custom certificates, including workspace hosts, pass the intended manifest with `--certificates FILE`. Keep its issuer and every required apex and wildcard name. This check prevents a repeat run from replacing wildcard TLS with an implicit apex-only certificate. `--render-certificates` previews the selected manifest without host changes and needs no repeat-install choice. The installed manifest is root-only, mode 0600. Before using it with `--certificates`, copy it to a file that you own: ``` sudo install -m 0600 -o "$(id -u)" -g "$(id -g)" \ /var/lib/rancher/k3s/server/manifests/hephaestus-certificates.yaml ./saved-certificates.yaml ``` Then pass `--certificates ./saved-certificates.yaml`. This preserves the existing issuer, including a staging issuer. A new `GITHUB_OAUTH_CLIENT_SECRET` replaces the stored one and restarts the server. ### Certificate renewal cert-manager renews the Gateway certificate and the PostgreSQL certificates before they expire. Traefik reads the renewed Gateway certificate without a restart. The PostgreSQL server certificate is valid for one year, and cert-manager renews it 30 days before it expires. PostgreSQL reads it only when it starts, as [TLS certificates](/admin/secret-custody.md#tls-certificates) describes. The CA of the PostgreSQL certificate is valid for ten years. 1. Find the time of the next renewal: ``` kubectl --namespace hephaestus get certificate hephaestus-postgresql-tls -o jsonpath='{.status.renewalTime}' ``` 2. After that time, restart PostgreSQL: ``` kubectl --namespace hephaestus rollout restart statefulset/hephaestus-postgresql ``` ### Upgrade the host 1. Get the files of the new release into the same directory, as [Get the release files](#1-get-the-release-files) describes. 2. Run the installer of the new release with the same arguments. It installs a new k3s pin, replaces the platform files, and runs the upgrade that [Upgrades](#upgrades) describes. ### Known limits of the one-host platform * Traefik has no request body limit on this platform. Its default idle timeout is 180 seconds. * The k3s default Pod network is `10.42.0.0/16`. `trustedProxies` in `k3s-values.yaml` matches it. If you change the network of k3s, change `trustedProxies` too. * The platform issues a separate backup client certificate, `hephaestus-postgresql-backup-tls`. Enable backups after you configure the off-host repository ([Backup and restore](#backup-and-restore)). * If the model endpoint resolves to this host's own Gateway, set `networkPolicy.workerGatewayPorts: [8443, 8000]` in `site.yaml`, as [Network policy](#network-policy) describes. * CI's clusters run no cert-manager. Only the release workflow's install tests the cert-manager pin. ## Install This procedure installs the chart on a cluster that you operate. For one host, use [Install on one host (k3s)](#install-on-one-host-k3s). 1. Create the release namespace and the sandbox namespace, and label both with the Restricted Pod Security level. Use the sandbox namespace for this release alone. ``` kubectl create namespace hephaestus kubectl create namespace hephaestus-sandboxes kubectl label namespace hephaestus hephaestus-sandboxes pod-security.kubernetes.io/enforce=restricted ``` 2. Make the OAuth App of your login provider, as [First steps § 1](/admin/first-steps.md#1-configure-a-login-provider) describes. Then create the [Secret](#create-the-secret) with its client secret, and the [PostgreSQL TLS Secret](#postgresql-tls-secret). The server copies the client secret into the database at its first start. 3. Write your values file, `site.yaml`, with `host`, `trustedProxies`, `secret.name`, `postgresql.tls.existingSecret`, `route.parentRefs`, and `networkPolicy.gateway` with its pod labels. Add `config.github.oauthClientId` and `config.auth.bootstrapAdmins`, as [First steps § 1 and § 2](/admin/first-steps.md) describe. Do not set `release` or `images`: the release chart sets them. For a public instance, create the ConfigMap with your legal pages and name it in `webapp.legalOverrides`, as [Legal pages](/admin/legal-pages.md#deployment-specific-overrides) describes. The webapp Pod waits for a ConfigMap that it names. 4. Enable reviews. `small.yaml` enables the worker, and the chart does not render it without the sandbox namespace and the Kubernetes API endpoint. Find the endpoint that the network plugin sees: ``` kubectl get endpointslices --namespace default --selector kubernetes.io/service-name=kubernetes ``` On a single k3s node, `PORTS` is `6443` and `ENDPOINTS` is the node's internal address, for example `192.168.5.15`. Add both values to `site.yaml`: ``` worker: sandbox: namespace: hephaestus-sandboxes networkPolicy: kubernetesApi: cidr: 192.168.5.15/32 port: 6443 ``` Use `/128` for an IPv6 address. If `ENDPOINTS` lists more than one address, one for each control plane node, set a CIDR that holds all of them. 5. [Verify the release chart](#verify-the-release-chart). 6. Install the verified chart with a [profile](#profiles) and your values: ``` helm upgrade --install hephaestus "chart-$VERSION/hephaestus" -n hephaestus \ -f "chart-$VERSION/hephaestus/profiles/small.yaml" -f site.yaml \ --rollback-on-failure --wait=watcher --wait-for-jobs --timeout 20m ``` 7. Follow the migration Job's log until the migrations are complete: `kubectl --namespace hephaestus logs job/hephaestus-migration-1`. 8. Wait until the worker is ready. A worker is ready only after its [isolation probe](#sandbox-prerequisites) passed and it watches the sandbox namespace. ``` kubectl --namespace hephaestus rollout status deployment/hephaestus-worker ``` If the worker does not become ready, read [Troubleshooting](#troubleshooting). 9. Continue with [First steps § 3. Sign in](/admin/first-steps.md#3-sign-in). The schema refuses an unknown key, an image tag, and a value outside its bounds. Each object name is `-`, with at most 52 characters. A StatefulSet revision label adds up to 11 characters to its name, and a label value has at most 63. If a name is longer, the chart keeps its start and adds 8 characters of a hash of the full name. The same release always gets the same names. The `app.kubernetes.io/instance` label always holds the full release name. `config.auth.assuranceEnforced` controls the passkey and recent sign-in checks for administration. Its default is `true`, as in the application. The chart never changes it from the environment name. To turn it off, for example on a staging instance, set it to `false` yourself. These values set the server's settings of the same meaning ([Production setup](/admin/production-setup.md)): | Value | Setting | Default | | -------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------- | | `config.researchOrganization` | `HEPHAESTUS_RESEARCH_ORGANIZATION` | Empty | | `config.publicActivity.enabled` | `HEPHAESTUS_PUBLIC_ACTIVITY_ENABLED` | `false` | | `config.publicActivity.repositoryVisibilityMaxAge` | `HEPHAESTUS_PUBLIC_ACTIVITY_REPOSITORY_VISIBILITY_MAX_AGE` | `48h` | | `config.llmDisplayCurrency` | `HEPHAESTUS_LLM_DISPLAY_CURRENCY`: empty or `EUR` | Empty | | `config.auth.passkeys.instanceAdminRequired` | `HEPHAESTUS_AUTH_PASSKEYS_INSTANCE_ADMIN_REQUIRED` | `true` | | `config.auth.passkeys.workspaceAdminRequired` | `HEPHAESTUS_AUTH_PASSKEYS_WORKSPACE_ADMIN_REQUIRED` | `false` | | `config.auth.passkeys.rpId` | `HEPHAESTUS_AUTH_PASSKEYS_RP_ID` | Empty: the host | | `config.auth.passkeys.allowedOrigins` | `HEPHAESTUS_AUTH_PASSKEYS_ALLOWED_ORIGINS` | Empty: the host | | `config.email.tls` | `starttls` sets `SPRING_MAIL_PROPERTIES_MAIL_SMTP_STARTTLS_ENABLE` and `_REQUIRED`; `implicit` sets `SPRING_MAIL_PROPERTIES_MAIL_SMTP_SSL_ENABLE`. `SPRING_MAIL_PROPERTIES_MAIL_SMTP_AUTH` is on when `config.email.username` is set. | `starttls` | | `config.monitoring.runOnStartup` | `MONITORING_RUN_ON_STARTUP` | `true` | | `config.monitoring.timeframeDays` | `MONITORING_TIMEFRAME` | `7` | | `config.monitoring.syncCron` | `MONITORING_SYNC_CRON`, a Spring cron expression with six fields | `0 0 * * * *` | | `config.monitoring.cooldownMinutes` | `MONITORING_SYNC_COOLDOWN_IN_MINUTES` | `60` | `webapp.securityContact` sets the webapp's `SECURITY_CONTACT`: with a value, `/.well-known/security.txt` names it as the contact of this deployment ([Production setup](/admin/production-setup.md)). Empty serves no file. ## Integration values Each integration takes its settings from `site.yaml` and its secrets from the [Secret](#create-the-secret). `values.schema.json` describes each value. [Integration settings](/admin/production-setup.md) explains each setting. | Integration | Values | Secret keys | | ----------------- | -------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- | | GitHub login | `config.github.oauthClientId` | `GITHUB_OAUTH_CLIENT_SECRET` | | GitHub App | `config.github.appId`, `config.github.appClientId`, `config.github.appInstallationUrl` | `GH_APP_PRIVATE_KEY`, `GH_APP_CLIENT_SECRET` | | GitLab | `config.gitlab.*` | `GITLAB_OAUTH_CLIENT_SECRET` | | Slack | `config.slack.*` | `HEPHAESTUS_INTEGRATION_SLACK_SIGNING_SECRET`, `HEPHAESTUS_INTEGRATION_SLACK_CLIENT_SECRET` | | Outline | `config.outline.*` | `OUTLINE_OAUTH_CLIENT_SECRET` | | Email | `config.email.*` | `SPRING_MAIL_PASSWORD` | | Provider webhooks | None | `WEBHOOK_SECRET`, `WEBHOOK_ROUTING_SECRET` | ## Worker The worker role runs every review sandbox, the conversation host, and every turn sandbox. The chart's default disables the worker, and `small.yaml` enables it. Prepare the [sandbox prerequisites](#sandbox-prerequisites). Set `worker.sandbox.namespace` and `networkPolicy.kubernetesApi` ([Install](#install), step 4). The chart refuses a worker without them. The chart runs the workers as one Deployment: * Two replicas by default, spread across nodes where the cluster has several. `worker.replicas` sets the count. * A rollout starts all new workers before an old one stops, and a PodDisruptionBudget lets a node drain take one worker at a time. `worker.rolloutStrategy=Recreate` stops the old workers first, for a node that cannot hold both ([Profiles](#profiles)). * A stopping worker claims no new work and serves its running attempts for `worker.drainTimeoutSeconds`, 3600 by default. A practice review usually takes 10 to 16 minutes. An attempt that outlasts the drain is tried again. * A worker on a lost node is replaced after 60 seconds. * `AGENT_ENABLED=true` on the workers and the server, so that the server submits practice reviews and the workers run them. Without a worker, the server submits none. * `worker.reviews` sets the reviews that one worker runs at once, and `worker.conversationTurns` the conversation turns (`HEPHAESTUS_WORKER_CAPACITY_MENTOR_MAX`, `auto` by default). Each running turn holds a turn sandbox beside the 2 pooled ones, and the conversations quota counts them ([Runtime roles § Conversation turn capacity](/admin/runtime-roles.md#conversation-turn-capacity)). `small.yaml` sets 8. * The settings `HEPHAESTUS_SANDBOX_RUNTIME=kubernetes`, the sandbox namespace, and the pod name and address from the Downward API ([Runtime roles § Sandbox runtime](/admin/runtime-roles.md#sandbox-runtime)). * The existing gateway setting `SANDBOX_API_PORT`, with the value 8081. The sandbox namespace is the one that you created for this release. The chart creates no namespace. It writes these objects into it: * One ResourceQuota for each PriorityClass tier: reviews, and conversations for the conversation host and turn sandboxes. The chart derives both from `sandbox-pod-requests.json` and from `worker.replicas`, `worker.reviews` and `worker.conversationTurns`. They count requests, not limits: sandbox Pods may use more than they request, up to their limits. Thus, Hephaestus's sandboxes never take a review's room, and reviews never take Hephaestus's. Each tier's quota counts Pods, CPU and memory. Kubernetes does not let a quota that selects a PriorityClass count disk. The quota without a scope counts the disk requests of both tiers together. A quota also counts the Pod overhead of a RuntimeClass, such as gVisor's on some clusters. For such a class, set `worker.runtime.podOverhead` to its `overhead.podFixed`: `cpuMillis` and `memoryMi`. The chart then adds the overhead to each Pod in the quotas. Set `worker.sandbox.quota.enabled=false` to manage the quotas yourself. `priorityClasses.reviews` and `priorityClasses.conversations` must name different classes, because each quota selects its Pods by class. During a rolling update, the chart's quotas hold the workers of the new and the old version. A worker that still drains during a second update can find no room: its sandbox then waits and is tried again. A worker claims a practice review only when every quota that counts review Pods has room for the review's Pod. An unscoped quota counts them, and so does a quota that selects the reviews' PriorityClass. It counts the reviews that it claimed but has not started yet. Otherwise the review stays queued. The worker's Role lets it list the quotas; its permissions are listed below. It reads the Pod overhead of the sandbox RuntimeClass from its own sandbox Pods. A quota with another scope does not hold a review back, and neither do quotas that the worker cannot read. Another worker can still take the room after the claim. The quota then refuses the review's Pod, and the review is tried again. * A deny-all policy for ingress and egress. * One more policy that lets review and host sandboxes reach port 8081 of the worker Pods, and nothing else. Those sandboxes carry the label `hephaestus.build/sandbox-kind` from the runtime. Turn sandboxes get no other policy, so they have no network. The worker policy admits the same sandboxes on the same port. * The worker's Role and RoleBinding. ### Priorities Under memory pressure, the kubelet evicts Pods of a lower priority first. The scheduler can also preempt Pods of a lower priority to place one of a higher priority. The chart gives every Pod one of three priorities, highest first: 1. The roles, PostgreSQL, NATS, and the migration and backup Jobs. They serve everyone. 2. The conversation host and the turn sandboxes. A person waits for the answer. 3. The review sandboxes and the isolation probe. A review is an attempt that runs again. It never preempts another Pod. A preempted or evicted review counts as an attempt that the worker lost, and the review runs again. By default, the chart creates the three PriorityClasses. If your account cannot create cluster-wide objects, ask the cluster's administrator for three classes, with values in the order above. Then set `priorityClasses.create=false` and name them: ``` priorityClasses: create: false roles: platform-high conversations: people-waiting reviews: batch-low ``` The chart refuses a name while `create` is true, and a missing name while it is false. The workers remove what no attempt and no worker uses in the sandbox namespace. Do not share it with another release or another application. ### When the sandbox quota is full A review waits in the queue while the quota has no room for its sandbox. Waiting behind running reviews is normal backlog, and it never ends a review. Two cases end queued reviews, with the stop reason `NO_SANDBOX_ROOM`: * A quota is smaller than one review sandbox. The worker cancels the queued reviews within 30 seconds, with "The sandbox quota is smaller than one review sandbox." It logs one warning. * Reviews have waited for an hour while the quota had no room for one and no review ran, so nothing frees room. Usually a quota of your own holds the room. The worker logs a warning when this starts. After the hour, it cancels the queued reviews with "Hephaestus had no room to run this review in time." A scan that loses a file's job this way ends with the same reason. It does not queue its next file. A review whose Pod the quota refuses after its claim also waits, and spends none of its retries. Its next attempt gets a new number, as every attempt does. The waits grow from 30 seconds to 5 minutes. The number of refusals never ends a review. Only the two cases above do. The metric `sandbox_start_failures_total{kind="review",reason}` counts these refusals and cancellations, with the reasons `quota_exceeded` and `quota_cannot_fit`. The gauge `sandbox_review_starved_seconds` shows how long no review could start while none ran. Alert on it well before an hour: the quota is too small for the sandboxes beside the reviews. The worker learns of the refusal from the events of the sandbox namespace, and reads its quotas to tell whether a sandbox can fit at all. It needs `get` and `list` on `events`, and `list` and `watch` on `resourcequotas` there. The chart's Role grants them. Without the verbs on `events`, the worker logs one warning, and a refused review waits 2 minutes and spends a retry. Without `list` and `watch` on `resourcequotas`, `pods` or `jobs`, the worker stays not ready ([Troubleshooting](#troubleshooting)). The worker watches the Pods, Jobs and quotas of the sandbox namespace. Thus, a claim poll and a readiness probe send no request to the API server. A sandbox's credential, not the network, binds it to the worker that started it. The worker reaches the Kubernetes API only at `networkPolicy.kubernetesApi`. Set the address and port that the selected network plugin sees after address translation. The plugins that CI runs, k3s's kube-router and Calico, see the endpoint of the `kubernetes` Service ([Install](#install), step 4). The chart has no default for them, because only the running cluster knows them. The worker's service account has these permissions in the sandbox namespace, and none elsewhere: * Pods: `get`, `list`, `watch`, `create`, `delete`, and `patch`. A cancellation or a timeout patches the deadline of the sandbox's Pod. * `pods/log`: `get`. * `pods/exec`: `get` and `create`. The client opens an exec as a WebSocket `GET`, and the API server also authorizes it as `create`. * Jobs: `get`, `list`, `watch`, `create`, and `delete`. * Events: `get` and `list`. A review Pod that the sandbox quota refuses waits for room: the Job controller reports the refusal as an event. * ResourceQuotas: `list` and `watch`. The quota tells how many reviews fit and whether a review can ever fit. A practice review attempt runs as one Job. The Job ends at the attempt's upload deadline plus two minutes, also when its worker is gone, and Kubernetes removes it 15 minutes after it ends. The review-ledger worker identity stays per process. The chart does not set `HEPHAESTUS_WORKER_ID`. ### Sandbox prerequisites The sandbox nodes need these items: 1. A network plugin that enforces NetworkPolicy, for egress as well as ingress. 2. The sandbox namespace with the label `pod-security.kubernetes.io/enforce=restricted`. 3. Encrypted node storage, for example LUKS or the disk encryption of the cloud provider. A sandbox Pod writes to the node's disk: the review workspace with the repository, `/tmp`, `~/.local` and a turn sandbox's workspace. Only `/run` is in memory. Encryption protects a stolen disk and a backup of it without the key. It does not protect against a compromised running node or a privileged workload on the node. The kubelet deletes a Pod's volumes with the Pod, but deletion does not erase the data from the disk. These settings are optional: * `worker.runtime.runtimeClassName`: a RuntimeClass for sandbox Pods. gVisor (`runsc`) is recommended for workloads from people that you do not trust. * `worker.runtime.podOverhead`: the RuntimeClass's `overhead.podFixed`, as `cpuMillis` and `memoryMi`. The sandbox quotas count it with each Pod. * `worker.runtime.seccompProfile`: the bundled `sandbox/agent-seccomp-profile.json` as a Localhost profile. Put the file below the kubelet's seccomp directory on every sandbox node, and set its relative path. Without it, sandbox Pods use `RuntimeDefault`. * `worker.runtime.nodeSelector` and `worker.runtime.tolerations`: the nodes of sandbox Pods. * A kubelet `podPidsLimit` on the sandbox nodes. Each worker runs the isolation probe in the sandbox namespace at its start, and then every `worker.runtime.probeInterval`, 15 minutes by default. The probe checks these items on the running cluster: * The API server refuses, in a dry run, a Pod that the Baseline standard admits and the Restricted standard refuses. * A Pod with the labels of a review sandbox reaches the worker's gateway port. It does not reach the worker's management port, the Kubernetes API, or the cluster DNS server. * A Pod of no sandbox kind does not reach the gateway port. Turn sandboxes have the same policy. * The probe Pods have no service account token, a read-only root filesystem, and a non-root user. * A named RuntimeClass exists: without it, no probe Pod runs. The worker starts no sandbox while one of these items fails. At its start, the worker then logs each failed item and stops, and Kubernetes starts it again. On a later probe, it logs each failed item, reports itself not ready, and starts no new sandbox until a probe passes again. Until a probe passes, at the start and after a failed later probe, the worker is not ready and claims no practice review and no conversation turn. They stay queued for a ready worker. Reviews and turns that already run continue. A later probe whose Pods cannot start, because the sandbox quota is full or the nodes have no room, proves nothing: the worker logs a warning and keeps its last passing proof. A Pod that the sandbox quota refuses delays that verdict by up to 2 minutes. The probe also records the isolation tier of the cluster, which is not a requirement: | Item | Effect | | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | | RuntimeClass | The runtime that isolates sandboxes from the node kernel, or the cluster's default. | | User namespaces | When a Pod with `hostUsers: false` runs in a user namespace of its own, every sandbox Pod runs that way. The root of a sandbox is then no user of the node. | | Process limit | The number of processes that a sandbox Pod could start before the node refused one, or no limit. | The worker's log and the metrics `sandbox.isolation.passed`, `sandbox.isolation.user_namespaces` and `sandbox.isolation.process_limit` show the tier. The probe Pods use the agent image and the sandbox configuration. Each probe Pod is the Pod of a Job, and a run takes 768 MiB of the sandbox quota on the cluster's default runtime: 512 MiB for the Pod that tries 1024 processes, and 128 MiB for each of the two others. The worker deletes the Jobs when the probe ends. If the worker stops during a run, the Job's deadline ends a Pod that still waits for a node, and Kubernetes deletes the finished Job with its Pod after 15 minutes. Some network plugins apply the policy of a new Pod a few seconds after the Pod starts. Each sandbox Pod therefore starts with an init container that waits, at most 60 seconds, until it cannot reach the Kubernetes API. #### Sandbox sizes The worker sizes each sandbox Pod from the measured use of that kind of sandbox. `SANDBOX_MEMORY_BYTES` applies to Docker only. | Sandbox Pod | CPU request | Memory request | Memory limit | Disk request | Size limit of each disk volume | Disk limit | | ----------------- | -------------------------- | ---------------------------------- | ------------------- | ------------------- | ------------------------------ | ------------------------------ | | Review | 100m | 640 MiB | 1,280 MiB | 1 GiB | 1 GiB, 7 volumes | 7 GiB and 64 MiB | | conversation host | 50m | 256 MiB | 1 GiB | 256 MiB | 1 GiB, 7 volumes | 7 GiB and 64 MiB | | Turn sandbox | 20m, and 5m for its stager | 128 MiB, and 32 MiB for its stager | 512 MiB, and 32 MiB | 256 MiB, and 16 MiB | 512 MiB, 3 volumes | 1.5 GiB and 64 MiB, and 16 MiB | A review and the conversation host have seven volumes on the node's disk: the workspace, `work`, `out`, `.pi`, `.sessions`, `/tmp` and `~/.local`. A turn sandbox has three: its workspace, `/tmp` and `~/.local`. The disk limit of the main container holds all the volumes full and 64 MiB for its logs. Each other container, such as the isolation init container and the stager, has a disk limit of 16 MiB for its logs. The stager and each probe Pod request a tenth of their CPU limit. The isolation init container requests 10m. The chart's sandbox quotas count the requests of this table, not the limits. Four large reviews at once peaked at 632 to 657 MiB each. One conversation used 136 MiB in the host, and a turn sandbox used 3 MiB. The host's Node heap stops at 768 MiB. `/run` is in memory, with 16 MiB. The kubelet evicts a Pod whose volume passes its size limit, or whose disk use passes its disk limit. Thus, no sandbox can fill the node's disk and cause the eviction of other Pods. The worker counts the eviction as a failure of the infrastructure, and the review spends a retry: a review whose sandbox keeps filling its disk ends at `AGENT_MAX_RETRIES`. The chart's sandbox quota without a scope counts the disk requests of all sandbox Pods. Kubernetes has no field for two Docker sandbox settings. `/tmp`, `/run` and a turn sandbox's `/workspace` are not `noexec`, and a sandbox Pod keeps `/dev/shm` in its private IPC namespace. [ADR 0041](https://github.com/hephaestus-build/Hephaestus/blob/main/docs/decisions/0041-compose-1x-kubernetes-2.md#update--2026-10-24-the-kubernetes-runtime) records why. Every container command and every exec starts through `prlimit` from the agent image. `prlimit` sets the `nofile` limit to 1024 and the `core` limit to 0, both soft and hard. A sandbox process cannot raise them. ## Backup and restore With `postgresql.backup.enabled=true`, the bundled PostgreSQL archives its WAL to one S3-compatible bucket with pgBackRest, as the [Compose backup overlay](/admin/backup-restore.md#automated-off-host-backup-self-host-compose) does. The repository is encrypted with `BACKUP_CIPHER_PASS`. A CronJob takes a full backup each Sunday at 02:15 UTC, and another takes a differential backup on the other days at the same time. `postgresql.backup.retentionFull` full backup chains stay, 2 by default. A Job of each release creates the repository's stanza, so that the WAL archive works from the install. The backup Jobs do not mount the PostgreSQL volume. They read the data directory through the pgBackRest TLS server in the PostgreSQL Pod. That server admits only a client certificate for the common name `hephaestus-backup` that `ca.crt` signed. To prepare backups: 1. Create the bucket for this instance alone, and an identity with read, write and delete rights in it. 2. Add `BACKUP_S3_KEY`, `BACKUP_S3_KEY_SECRET` and `BACKUP_CIPHER_PASS` to the release Secret. Generate the passphrase and keep a copy as [Secret custody](/admin/secret-custody.md#keep-the-required-keys-outside-the-cluster) describes. 3. Set `postgresql.backup.clientTlsSecret` to a TLS Secret for the common name `hephaestus-backup`. It needs `ca.crt`, `tls.crt` and `tls.key`, signed by the PostgreSQL CA. The one-host platform issues `hephaestus-postgresql-backup-tls` and sets this value. 4. Set `postgresql.backup.enabled=true` and `postgresql.backup.s3` with the bucket, the endpoint host name and the region. Set `cidr` and `port` when the endpoint is in a private address range. When a private CA signs the endpoint's certificate, add it to the PostgreSQL TLS Secret as `s3-ca.crt`, and set `postgresql.backup.s3.privateCa=true`. 5. Upgrade the release, and check that the stanza Job completed. [Backup bucket credentials](/admin/secret-custody.md#backup-bucket-credentials) and [Backup cipher passphrase](/admin/secret-custody.md#backup-cipher-passphrase) give the rotation of these values. A failed WAL archive keeps the WAL on the PostgreSQL volume until the volume is full. The [alert rules](/admin/observability.md#alert-rules) watch the WAL archive, the age of the latest backup and the free space of the volume. ### Restore A restore replaces the bundled database with a backup. It loses every change after the backup or the time that you select. The restored database is readable only with the keys that sealed its values. Keep them as [Secret custody](/admin/secret-custody.md#keep-the-required-keys-outside-the-cluster) describes. Restore the NATS JetStream data separately, as [Backup and restore](/admin/backup-restore.md) describes. #### Isolated restore drill Run the following procedure only on a disposable, isolated clone. It changes control settings with clone lockdown SQL. Use [Recover the live installation](#recover-the-live-installation) for a live recovery. 1. Upgrade the release with `postgresql.backup.restore.enabled=true`. Set `postgresql.backup.restore.time` to restore the database as it was at that time, for example `2026-10-24 18:00:00+00`. Leave it empty to restore the latest backup and every archived WAL segment. While the setting is on, the server, the webhook role, the workers and PostgreSQL stop, and the backups pause. 2. Wait for the Job `-postgresql-backup-restore-` to complete: ``` kubectl --namespace hephaestus wait --for=condition=complete \ job/hephaestus-postgresql-backup-restore- --timeout=30m ``` The Job retries while PostgreSQL still shuts down. 3. Keep the clone isolated before any application code starts. Deny provider and model egress independently of silent mode. 4. Upgrade with `postgresql.backup.restore.enabled=false`, `migration.enabled=false`, and server, webhook and worker replica counts of zero. PostgreSQL alone replays the WAL up to the target. Readiness does not prove that replay finished. Wait until this command prints `f`: ``` kubectl -n hephaestus exec hephaestus-postgresql-0 -c postgresql -- \ sh -c 'exec psql -Atc "SELECT pg_is_in_recovery()" -U "$POSTGRES_USER" -d "$POSTGRES_DB"' ``` 5. Apply the lockdown SQL from the verified chart, before starting migrations or application roles: ``` kubectl -n hephaestus exec -i hephaestus-postgresql-0 -c postgresql -- \ sh -c 'exec psql -v ON_ERROR_STOP=1 -U "$POSTGRES_USER" -d "$POSTGRES_DB"' \ < "chart-$VERSION/hephaestus/files/restore-clone-lockdown.sql" ``` 6. Read back silent mode, paused feedback and disabled conversation bindings. Keep external egress denied. 7. Restore JetStream with a recovery point consistent with PostgreSQL. Resolve provider write receipts before consumers start. 8. Set `migration.enabled=true` and start the server alone. Verify migrations, readiness and authenticated reads. 9. Restore the other replica counts only after the isolated checks pass. Follow [Prove silence](/admin/production-operations-runbook.md#silent-deployment-checklist) before opening any external destination. The restored database has the database password that it had at the backup. If `DATABASE_PASSWORD` changed since then, put the old value back in the Secret before step 3. Rehearse a restore on a copy of the instance before you need one. #### Recover the live installation 1. Close traffic and provider reception. Stop application roles and sandboxes, and pause reconciliation. 2. Keep a copy of the failed data and record the chosen recovery point and expected lost writes. 3. Check whether `DATABASE_PASSWORD` changed after the recovery point. If it changed, restore the matching value in the Secret before startup. Complete steps 1, 2 and 4 of the isolated drill with application replicas at zero. Keep provider and model egress denied. Wait for `pg_is_in_recovery()` to return `f`. 4. Do not apply `restore-clone-lockdown.sql` to the live recovery database. Verify silent mode, delivery status, review triggers and conversation bindings against the intended live settings. 5. Complete [Choose the broker recovery strategy](/admin/production-operations-runbook.md#choose-the-broker-recovery-strategy-before-starting-consumers) before consumers start. The default is to quarantine and recreate the instance's broker state, with recorded event loss. Snapshot replay needs a separate drill. A matching timestamp or recovery fence alone does not make it safe. 6. Enable migrations and start the server alone. Verify readiness, authenticated reads and encrypted values. 7. Before access or delivery resumes, reapply post-backup security and privacy decisions from the restricted incident record. [Before restarting services](/admin/backup-restore.md#before-restarting-services) gives the required suspension, session, consent and deletion checks. 8. Start the worker and require its isolation probe. Permit only the approved model endpoint for the controlled test while provider writes remain blocked. Test an approved practice review and conversation with delivery paused. 9. Complete **Prove silence**, **Inspect and decide**, and **Turn off silent mode deliberately** in the [silent checklist](/admin/production-operations-runbook.md#silent-deployment-checklist). Restore the intended live control settings, provider reception and egress. Start the remaining roles and reopen traffic after the checks pass. 10. Verify every connected provider and feedback destination. Resume reconciliation only after its release and apply mode match the recovered deployment. ## Upgrades An upgrade to a new release is one `helm upgrade`, with the new release's chart, its profile, and your own values. First, [verify the release chart](#verify-the-release-chart) of the new release. Then upgrade: ``` helm upgrade --install hephaestus "chart-$VERSION/hephaestus" -n hephaestus \ -f "chart-$VERSION/hephaestus/profiles/small.yaml" -f site.yaml \ --rollback-on-failure --wait=watcher --wait-for-jobs --timeout 20m ``` The migration Job of the new revision and the new Pods start together. The new Pods wait for the migrated schema, and the previous release serves until they are ready. `--wait-for-jobs` fails the upgrade when the migration fails, and `--rollback-on-failure` then returns the Deployments to the previous revision. The timeout is longer than the 10 minutes that a new Pod waits for the schema. The server stops before its new version starts (`Recreate`). The webhook role and the workers roll: new workers prove the cluster with the isolation probe before an old worker drains. An old worker drains for up to `worker.drainTimeoutSeconds`, 60 minutes by default, and Kubernetes no longer counts it while it terminates. Start the next upgrade only after the old worker is gone: `kubectl --namespace hephaestus get pods -l app.kubernetes.io/component=worker` shows no `Terminating` Pod. A review runs for up to 3 hours. When the drain ends first, the review stops: an attempt that was not yet admitted runs again on another worker, and an admitted one keeps its results. A drained attempt spends none of the review's retries. After 3 drains, the next drain cancels the review with the message "Workers stopped this review for updates too often. It is not tried again." The metric `agent_job_drained_total{outcome}` counts drained reviews that run again (`requeued`) and that are cancelled (`cancelled`). Read the release notes before the upgrade. If the release notes say **Stop every role**, use [Stopped schema upgrades](#stopped-schema-upgrades). For other database maintenance, use [Database maintenance](#database-maintenance). Roll forward to fix a failed upgrade. `helm rollback` and an older image do not undo a migration: restore the database from the backup taken before the upgrade ([Restore](#restore)). ### Stopped schema upgrades Use this procedure when the migration guide says **Stop every role**. It takes precedence over the normal upgrade and database-maintenance procedures. Do not use `--rollback-on-failure`: a rollback can restart old roles against a changed schema. Pause deployment automation that can change the release until this procedure is complete. Before the first phase, record the current chart, profile, site values and all three replica counts. Back up PostgreSQL and JetStream, verify the restore, and keep the matching signed release lock. Verify the new release's chart before you use it. Each phase is a separate `helm upgrade` with that phase's chart, profile and site values: | Phase | Chart | `migration.enabled` | `server.replicas` | `worker.replicas` | `webhook.replicas` | Before the next phase | | ---------------------- | --------------- | ------------------- | ----------------- | ----------------- | ------------------ | --------------------------------------------------------------------------------------- | | 1. Stop every role | Current release | `false` | 0 | 0 | 0 | Every old role Pod, including terminating Pods, and its sandbox Pods and Jobs are gone. | | 2. Migrate | New release | `true` | 0 | 0 | 0 | The migration Job of this revision completed successfully. | | 3. Start the new roles | New release | `true` | Recorded count | Recorded count | Recorded count | Readiness, authenticated reads, a practice review and a conversation pass. | Keep PostgreSQL and NATS running throughout these phases. Use `worker.replicas=0`, not `worker.enabled=false`, to stop the worker. Apply each row's values after your profile and site values, so that they take precedence. Use `--wait=watcher --wait-for-jobs --timeout 20m`, without automatic rollback. A worker can drain for up to its configured timeout. If the first phase times out, keep all counts at zero and wait for every old Pod and sandbox to stop before phase 2. In phase 2, identify the migration Job of the new Helm revision. Its default name is `hephaestus-migration-` for release `hephaestus`. Save its log with `kubectl --namespace hephaestus logs job/ -c migration > migration.log`. Require its `Complete` condition: `kubectl --namespace hephaestus wait --for=condition=complete job/ --timeout=20m`. A failed or timed-out Job does not permit phase 3. Keep all roles stopped, preserve the log and failed change set before the Job's 24-hour expiry, and follow [failed-migration recovery](/admin/upgrade.md#if-migration-fails). Phase 3 creates another migration Job for the same release; it has no new schema changes to apply. Resume deployment automation only after its release and replica counts match the verified installation. ### Database maintenance Use this procedure when the release notes require it, before you change PostgreSQL itself, or before you change the CA in its TLS Secret. The server and the workers are the database roles. The webhook role has no database, so it keeps receiving deliveries into NATS JetStream through every phase. Each phase is one `helm upgrade` with the replica counts of that phase: | Phase | `server.replicas` | `worker.replicas` | Before the next phase | | ---------------------- | ----------------- | ----------------- | -------------------------------------------------------------------------------------------------------------- | | 1. Stop the writers | 0 | 0 | The old pods of the two roles are gone. The worker's sandbox pods are gone. | | 2. Change the database | 0 | 0 | PostgreSQL is ready and accepts connections. | | 3. Migrate | 1 | 0 | The migration Job of the revision completed. The server is ready. | | 4. Restore | 1 | The prior count | The pods of the two roles are ready. The server consumes the deliveries that the receiver published meanwhile. | Phase 1 uses the verified chart and the values of the current release, with only the two counts changed. Phase 2 applies the change. The change can be the new release values, a new PostgreSQL configuration, or a new CA in the TLS Secret and a restart of PostgreSQL. Phase 3 runs the new release's migration Job and starts its server alone. Before phase 1, record the current `worker.replicas`, and back up the database and the NATS JetStream data. A count of 0 is a desired state: check on the cluster that the pods stopped, that the sandbox Pods and Jobs are gone, and that the migrations completed before the next phase. Set `worker.replicas=0` to stop the workers; `worker.enabled=false` removes the worker objects. A change of NATS itself stops reception: while NATS is down, the receiver answers deliveries with `503`, and each provider decides whether it retries. ## Uninstall 1. Delete the sandbox namespace first. This stops every sandbox at once. If you remove the release first, its network policies go, and the conversation hosts and turn sandboxes that are left run without them. ``` kubectl delete namespace hephaestus-sandboxes ``` 2. Remove the release. Running reviews stop. ``` helm uninstall hephaestus --namespace hephaestus ``` 3. These items stay: * The PostgreSQL and NATS volumes. * The Secrets that you created. * The backups in the S3 bucket. 4. To remove the data too, delete the release namespace: `kubectl delete namespace hephaestus`. A PersistentVolume with the `Retain` reclaim policy stays until you delete it. ## Troubleshooting | Symptom | Cause and check | | --------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | A worker stays unready, and its log says that a sandbox namespace informer failed and retries | The worker cannot watch the Pods, Jobs or ResourceQuotas of the sandbox namespace. The log gives the HTTP status. A `403` means that the worker's Role lacks `list` or `watch` on one of them ([Sandbox prerequisites](#sandbox-prerequisites)). The worker retries and becomes ready by itself once it can watch. | | A worker stays unready, and its log says that the cluster does not hold the sandbox boundary | The isolation probe failed. The log names each failed check. A failed egress check usually means that the network plugin does not enforce NetworkPolicy, or that `networkPolicy.kubernetesApi` names an address that the plugin does not see. | | A worker log says that the network policy did not filter a sandbox Pod in time | The network plugin applied the policy of a new Pod more than 60 seconds after the Pod started. The attempt is tried again. | | A review fails with "The sandbox quota is smaller than one review sandbox" | A quota of your own holds less than one review sandbox requests. Raise it. The chart's quotas always hold one. | | A sandbox Pod stays `Pending` with a quota error | Its tier's quota is full. The chart sizes each tier for its workers' capacity, so check a quota of your own, or `worker.reviews` and `worker.conversationTurns`. | | Reviews stay queued, and the worker log says that reviews wait for room in the sandbox quota | The reviews tier holds the running reviews and the isolation probe. The reviews start when a sandbox ends. To run more at once, raise `worker.reviews`. | | A new Pod restarts during an upgrade, and its log names a pending change set | It waited 10 minutes for the schema. Check the migration Job's log. A longer migration needs the [database maintenance](#database-maintenance) procedure. | | A backup Job fails with `unable to find primary cluster` | The Job cannot reach pgBackRest's TLS server on port 8432 of the PostgreSQL Service, or its client certificate is not for `hephaestus-backup` from `ca.crt`. | | PostgreSQL's volume fills | A failed WAL archive keeps the WAL on the volume. Check the latest backup Job's log and the bucket's credentials. | | A practice review fails with "The sandbox could not prepare its workspace." | The sandbox's initializer failed before any model call. The job runs again within its retry limit, so a failure that repeats on every attempt uses all of them. The worker log names the initializer's exit code. After the last attempt, read the exit code and the last lines of the initializer's log on the job: `kubectl --namespace hephaestus exec hephaestus-postgresql-0 -c postgresql -- psql -U "$DATABASE_USERNAME" -d hephaestus -c "SELECT capture->'initializationFailure' FROM review_job WHERE id=''"`. Known credential formats in those lines are masked. | `kubectl --namespace logs deployment/-worker` shows a worker's probe result and its isolation tier. The worker's `/actuator/prometheus` on its management port shows the same tier in the `sandbox_isolation_*` metrics. ## Known limits * NATS JetStream runs as one server with one replica of each stream. A lost NATS volume loses the provider deliveries that the server had not processed ([Storage and retention](#storage-and-retention)). * PostgreSQL runs as one instance. A lost node stops the release until PostgreSQL runs again; a lost volume needs a [restore](#restore). * The server runs as one replica until the server processes provider events in order across replicas. * The worker records the isolation tier in its log and metrics, not with each review. A record for each review needs a schema change. * Turn sandboxes take their commands through `pods/exec` until they pull them through the gateway, as the conversation host does. * The kubelet enforces sandbox disk limits by eviction after it measures the use, not when a process writes. A sandbox can pass its limit until the next measurement, for some seconds. Only file system project quotas (`LocalStorageCapacityIsolationFSQuotaMonitoring`) refuse the write itself. * Move a Compose install at the same release version. Back up PostgreSQL, JetStream and the encryption keys before the move. ## What CI does not show A successful install in CI does not show that your cluster isolates sandboxes. The worker's isolation probe checks that on your cluster, at each start and on its schedule. CI does not qualify your Gateway controller: check its four `HTTPRoute` features, its body limit and its idle timeout yourself, and rehearse a [restore](#restore) on your own cluster.