Kubernetes

The Helm chart installs Hephaestus on a Kubernetes cluster. A single host runs the chart on k3s. From 1.0, the chart is the supported deployment: ADR 0058 records the decision, and the compatibility policy gives the terms. The worker runs practice reviews and conversation turns. The small.yaml profile enables it, and Install ends when it is ready. On one host, k3s runs the cluster: Install on one host (k3s) gives the complete procedure.

Each release publishes the chart as a signed OCI artifact, oci://ghcr.io/hephaestus-build/charts/hephaestus. The chart source is in deploy/chart/hephaestus/ of the repository. It uses the standard Helm chart format: Chart.yaml, values.yaml, values.schema.json, and templates/. It uses only stable Kubernetes APIs and the Gateway API HTTPRoute resource of version v1.

Supported clusters

Hephaestus supports the chart on a cluster with these items:

• A conformant Kubernetes cluster at a minor version that the Kubernetes project supports when the Hephaestus release ships. The chart refuses a cluster below 1.35. A cluster at an older minor, for example 1.30 or 1.33, must be upgraded before Hephaestus runs on it (ADR 0060).
• A network plugin that enforces NetworkPolicy for egress and for ingress. The worker's isolation probe stops the worker on a plugin that does not (Sandbox prerequisites).
• A Gateway API controller for HTTPRoute v1 with the four features in Gateway routes.
• A storage class for ReadWriteOnce volumes, for the bundled PostgreSQL and NATS.

CI installs and upgrades the chart on Kubernetes 1.35, the lowest version that the chart takes, and on the latest minor version, 1.37, on amd64. It also runs 1.35 on arm64. A pull request runs only the latest minor version on amd64. When a change touches the chart or the sandbox runtime, the merge queue runs all of them before the commit reaches main. A nightly run repeats them on main. One more run uses three nodes, Calico for network policy, and an external PostgreSQL. Each run starts two workers with the dev.yaml profile, and each worker passes its isolation probe on that cluster. In the merge queue, the 1.35 run on amd64 uses small.yaml and small-16gib.yaml instead, with one worker, on a runner of 16 GiB. Each upgrade must keep the PostgreSQL and NATS Pods: only a change to their own image or settings restarts them. On the latest minor version, the amd64 run also backs up to an S3 server and restores to a point in time, as Backup and restore describes. Install on one host (k3s) states what the release workflow tests before it publishes a release. When a Kubernetes minor version reaches its end of life upstream, the next Hephaestus release can drop it.

What the chart runs

Component	Kind	Role or image
server	Deployment, 1 replica, or 0 while stopped	Server role.
migration	Job, one for each Helm revision	The server image in its migrate run mode. It applies the release's database migrations and exits.
webhook	Deployment, 1 to 3 replicas, or 0 while stopped	Webhook role, as in Runtime roles.
webapp	Deployment	The SPA, from the non-root webapp image.
worker	Deployment, 2 replicas by default, optional	Worker role: practice reviews and conversations, in sandboxes (Worker).
postgresql	StatefulSet, optional	The release PostgreSQL image, with pg_partman and pgBackRest.
postgresql-backup	CronJobs, optional	Full and differential pgBackRest backups (Backup and restore).
nats	StatefulSet, optional	One NATS JetStream server.
route	HTTPRoute	/, /api, /webhooks, and Slack interactivity.
priority	PriorityClasses, cluster-wide	The priorities of the role Pods and of the sandboxes (Worker § Priorities).

The PriorityClasses are the only cluster-wide objects of the chart. The account that installs the chart therefore needs permission to create PriorityClasses. Their names contain the release's namespace and name, so two releases never share one. On a shared cluster that does not allow this, use classes that exist (Worker § Priorities).

The migration Job is the only migration owner. No role runs Liquibase: each new Pod waits for the schema before it starts, for up to 10 minutes, while the previous release serves. The Job is a plain part of the release and not a Helm hook, because a hook would run before the bundled PostgreSQL exists. Runtime roles describes the run mode and the wait.

The server has one replica and the Recreate update strategy. Servers do not yet process provider events in order across replicas. Two server versions never serve the same schema at the same time.

The Recreate strategy applies only to the server. The worker also connects to the database, and its rollout does not wait for the server. The webhook role has no database. Database maintenance gives the order of the roles for a change of the database or of a schema that needs every role stopped.

Verify the release chart

The chart package of a release names the release version, the source commit, and the release's five images by digest. Thus, you give Helm only your own values. The chart version and its appVersion are the release version without the leading v. The chart source in the repository names no image, tag, or registry. No release has published the chart yet.

You need Cosign 3.1 or later, the Helm CLI, curl, and jq. The release signs the chart with Cosign 3.1 in the Sigstore bundle format. Cosign 2 reads that format only from version 2.6 and only with --new-bundle-format, and CI does not test it. You do not need Node.js or a checkout of the repository. Use the Helm version that .github/actions/setup-helm pins. CI and the release workflow install that version. Contributors also need it: see the local development prerequisites. The pin is a build tool, not a qualified Helm version for an installation. No Helm plugin is necessary.

Run these commands in the directory of your site.yaml. If one command fails, stop: do not install the chart.

VERSION=X.Y.Z
CHART=ghcr.io/hephaestus-build/charts/hephaestus
# 1. The chart digest, from the joint lock of the release.
DIGEST=$(curl -fsSL "https://github.com/hephaestus-build/Hephaestus/releases/download/v$VERSION/release-v$VERSION.chart.json" |
  jq -er .chart.digest)
# 2. The release workflow signed that digest.
cosign verify "$CHART@$DIGEST" \
  --certificate-identity 'https://github.com/hephaestus-build/Hephaestus/.github/workflows/release.yml@refs/heads/main' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com
# 3. The one pull: Helm gets the chart by that digest, never by a tag.
helm pull "oci://$CHART@$DIGEST" --untar --untardir "chart-$VERSION"
# 4. The signed chart is the chart of this release.
helm show chart "chart-$VERSION/hephaestus" | grep -x "version: $VERSION" >/dev/null

The joint lock in step 1 is only a lookup for the chart digest. You do not verify it. Step 2 refuses every digest that the release workflow did not sign. That identity is the same for every release, so step 4 binds the chart to the release that you asked for. A version tag can move, and it is not a release signal.

The chart-$VERSION/hephaestus directory is the verified chart, with its profiles/. Step 3 stops if that directory already exists. Install and Upgrades use it. Release assets gives the purpose of each signed asset of a release.

Create the Secret

The chart reads secret values only from one Secret in the release namespace. It never writes a secret value. Set secret.name to the name of that Secret. The chart reads fixed key names from it.

Secret custody and rotation lists each key, the roles that read it, and the command that generates it. That page also tells you which keys to keep outside the cluster for a restore, and how to replace each key.

PostgreSQL TLS Secret

Each role connects to PostgreSQL with sslmode=verify-full. The role verifies the server certificate against a CA, and it checks that the certificate names the host that the role connects to. Set postgresql.tls.existingSecret to an existing Secret in the release namespace. The chart refuses to render without it. The chart creates no certificate, no key, and no CA.

Key	Bundled PostgreSQL	External PostgreSQL	Mounted in
ca.crt	Yes	Yes	Server, webhook, worker
tls.crt, tls.key	Yes	No	Bundled PostgreSQL only

The server, the webhook role, and the worker receive only ca.crt, read-only, at /etc/hephaestus/postgresql-ca. No application role receives a private key. Backup Jobs use a separate client Secret, postgresql.backup.clientTlsSecret. It holds ca.crt, tls.crt and tls.key for the common name hephaestus-backup. The PostgreSQL CA signs it. The chart mounts its certificate and key as backup.crt and backup.key. Do not put the private key of the CA in this Secret.

For the bundled PostgreSQL, the certificate in tls.crt must name <release>-postgresql.<release namespace>.svc. <release>-postgresql follows the object name rule in Install. The roles connect to exactly this name, without a cluster domain. The chart mounts tls.crt and tls.key into the PostgreSQL pod with the mode 0440. The files belong to root and to the group 999, and PostgreSQL accepts a key with these permissions.

TLS certificates gives the renewal of each certificate. A new CA needs the database maintenance procedure.

Gateway routes

The Gateway API is a platform prerequisite of the chart. The platform operator installs and operates these items:

• The Gateway API resource definitions and a Gateway controller.
• The Gateway, its TLS certificate, and its HTTP to HTTPS redirect.

The chart installs none of them. It attaches one HTTPRoute to the Gateways in route.parentRefs. route.parentRefs must name at least one Gateway.

Request path	Backend	Path change
/webhooks/slack/interactivity	server	None
/webhooks and below	webhook	None
/api and below	server	/api is removed
All other paths	webapp	None. The webapp serves the SPA for an unknown path.

The server serves the API without the /api prefix, as behind Traefik in Compose. A standard Ingress cannot remove a path prefix, so the chart does not use one.

The Gateway controller must support these HTTPRoute features:

• URLRewrite with ReplacePrefixMatch, for /api.
• ResponseHeaderModifier, for Strict-Transport-Security on each rule.
• HTTPRouteRequestTimeout, for timeouts.request on /api.
• HTTPRouteBackendTimeout, for timeouts.backendRequest on /api.

If the controller does not support a feature, the route does not work as described. The chart cannot detect this. Confirm all four features for the controller that you select.

The /api rule sets both timeouts.request and timeouts.backendRequest to 0s. The Hephaestus live stream stays open for much longer than one request. The two timeouts are separate: the request timeout covers the client request, and the backend timeout covers the full backend response. If a rule omits one of them, the controller selects that behavior. A zero value asks the controller to disable the timeout. A controller that cannot disable it can use its longest possible value instead. Thus, the 0s values do not guarantee a stream without a time limit. The other three rules set no timeout and no retry.

Set trustedProxies to a Tomcat regular expression that matches only the Gateway data plane addresses. The server does not start in production with a blank value.

The Gateway API has no portable request body limit or idle timeout. Configure these values on the Gateway:

• A request body limit of 25 MiB on /webhooks and /api.
• An idle timeout of more than 20 seconds for the Hephaestus live stream.

An idle timeout of more than 20 seconds does not replace the request and backend timeouts. No Gateway controller or version is qualified yet. A qualification must include these tests on the selected controller and its idle settings:

• A long Hephaestus live stream.
• A long WebSocket connection through the Gateway.
• A cancellation by the client during a stream.

Network policy

Network policy is always on. No value turns it off. Each pod of the release starts with no ingress and no egress. The chart then opens only these paths:

From	To	Port
networkPolicy.gateway pods	server, webhook, webapp	8080
server, webhook, worker	PostgreSQL	5432, or the external port
server, webhook	NATS	4222, or the external port
server, worker	networkPolicy.externalEgress	443
server, if config.email.host is set	networkPolicy.externalEgress	config.email.port
worker	The Kubernetes API, networkPolicy.kubernetesApi	Its port
worker, if networkPolicy.workerGatewayPorts is set	networkPolicy.gateway pods	networkPolicy.workerGatewayPorts
server, worker, for each item of networkPolicy.modelEndpoints	Its cidr	Its ports
Review sandboxes and conversation hosts	worker	8081, the sandbox gateway
PostgreSQL and the backup Jobs, with backups on	postgresql.backup.s3, or networkPolicy.externalEgress	postgresql.backup.s3.port
The backup Jobs, with backups on	PostgreSQL	8432, the pgBackRest TLS server
All pods	networkPolicy.dns pods	53
networkPolicy.metrics pods	All application roles	9090

Sandboxes are in the sandbox namespace, where every pod also starts with no ingress and no egress. Turn sandboxes get no other path.

A model endpoint that this cluster's own Gateway serves resolves to a public address, but the connection ends at a Gateway Pod. That private address is outside networkPolicy.externalEgress, so the worker cannot reach the endpoint. Set networkPolicy.workerGatewayPorts to the container ports of the Gateway Pods, for example [8443, 8000] for the Traefik of the one-host platform.

networkPolicy.gateway must name the namespace and at least one pod label of the Gateway data plane. A namespace without labels would admit every pod in it, so the chart refuses it. The same rule applies to networkPolicy.metrics when you set its namespace.

A model endpoint on the cluster's own ingress

The worker reaches AI providers through networkPolicy.externalEgress, which leaves out private address ranges. If the host name of a model endpoint resolves to the cluster's own ingress, the connection ends at an ingress Pod with a private address, and the policy refuses it. An example is a model proxy that the same cluster serves. Add a NetworkPolicy that lets the worker Pods reach the ingress Pods. NetworkPolicies add up, so this policy does not change the chart's policies. Use the container ports of the ingress Pods, not the Service ports: the policy applies after address translation. For Traefik in the namespace traefik, with container ports 8000 and 8443:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: worker-to-ingress
  namespace: hephaestus
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: hephaestus
      app.kubernetes.io/component: worker
  policyTypes: ["Egress"]
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: traefik
          podSelector:
            matchLabels:
              app.kubernetes.io/name: traefik
      ports:
        - { port: 8443, protocol: TCP }
        - { port: 8000, protocol: TCP }

A model endpoint on a private network

A model endpoint outside the cluster can have a private address, for example a vLLM server on the LAN. The application and the network policy both refuse private addresses by default, so you must allow the endpoint in both places:

config:
  # The application: host names, IP addresses, or CIDR ranges that a provider base URL can use.
  llmPrivateHosts: [vllm.lab.example]
  # Only if the endpoint serves plain HTTP.
  llmPrivateHostsAllowHttp: true
networkPolicy:
  # The network: the address range and the ports of the endpoint.
  modelEndpoints:
    - cidr: 10.20.0.0/16
      ports: [8000]

If only config.llmPrivateHosts names the endpoint, a connection saves, and then the request times out at the network. The chart cannot compare the two lists, because a host name is not an address range. Connect an AI provider tells what the application allows.

A cidr must be in 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 100.64.0.0/10, or fc00::/7, and ports must name at least one port. Thus, no item can open the link-local range of the cloud metadata service. Use modelEndpoints only for addresses outside the cluster. Do not list a range in config.llmPrivateHosts that covers the Pod or Service range of the cluster. A range that covers the Pod's own address stops the server or the worker at startup, but the server cannot detect the Service range. A NetworkPolicy ipBlock does not reliably match Pod addresses. For an endpoint in this cluster, use networkPolicy.workerGatewayPorts or a policy with a podSelector, as the previous section shows.

The management listener binds to the pod address. The kubelet probes it, and the metrics scraper can read it. No Service and no route expose it.

The external egress rule excludes private, link-local, and shared address ranges by default. An external database or broker in a private range needs postgresql.external.cidr or nats.external.cidr.

A rendered policy does not show enforcement. Each network plugin needs its own positive and negative traffic tests before the chart is supported.

Network policy controls which pods can connect. It does not encrypt traffic. The connections to PostgreSQL use TLS (PostgreSQL TLS Secret). These internal channels send plain text:

• The connections from the server and the webhook role to the bundled NATS.
• The connection from the worker to the server.
• The connections from sandboxes to the worker gateway.

The bundled PostgreSQL still accepts a connection without TLS. The roles never use one, and network policy admits only the roles.

On a cluster with several nodes, these channels cross the node network. Use a network plugin that encrypts traffic between nodes, for example WireGuard in Calico or Cilium, when that network is not private to the cluster.

Storage and retention

Data	Bound	Setting
PostgreSQL	Volume size	postgresql.persistence.size
JetStream	Broker max_file and stream storage budget	nats.jetstream.maxFileBytes
JetStream volume	max_file plus 25 %	Derived from nats.jetstream.maxFileBytes
Webhook streams	Age and size for each stream, and one start that may delete stored messages to apply a smaller bound	webhook.stream.*
Server Git mirrors	emptyDir size limit on the node	server.gitCache.sizeLimit
Worker Git mirrors	emptyDir size limit on the node, per worker	worker.gitCache.sizeLimit
AI spend ledger	Retention	config.llmUsageRetention
Temporary files	emptyDir size limit	Fixed in the templates

One value sets both the broker max_file and the webhook role's stream storage budget. The webhook role refuses to start when the stream bounds add up to more than that budget. Webhook ingestion operations gives the durability guarantee.

The small.yaml profile sets GitHub to 4 GiB and each GitLab, Slack and Outline stream to 512 MiB. All four streams keep deliveries for at most seven days. Their byte bounds total 5.5 GiB. The broker file budget is 8 GiB. A fresh bundled broker claims 10 GiB, including the 25 % storage allowance.

The profile overrides the shorter Slack and Outline ages through webhook.stream.maxAgeByStream. It retains 30-day durable expiry through config.consumerInactiveThreshold.

A byte bound can remove messages before seven days. Measure peak traffic before you select these bounds.

For an existing installation, keep its current nats.jetstream.maxFileBytes in your site values before you apply this profile. Do not shrink an existing PVC or change the StatefulSet claim template to adopt the smaller broker budget. Do not lower the broker budget below its stored data.

Before you shorten stream limits, verify an off-host backup and test its restore. Then use the guarded stream-limit procedure. A higher limit cannot recover deleted messages. The server updates expiry only on durables it uses. Review orphan durables separately before you change or delete them.

The Git mirrors are caches. A restarted server or worker clones them again, so they use node storage and no volume claim. The node needs that much free ephemeral storage, and the container limits include it.

The bundled NATS is one JetStream server, and every stream has one replica. The application has no setting for more replicas. A lost NATS volume loses each provider delivery that the broker accepted and the server had not yet processed. That window is usually seconds, and it grows while the server is stopped or behind. Use a storage class whose volumes survive the loss of a node, and back up the volume as Backup and restore describes for Compose.

Disk on one host

Each upgrade pulls the images of the new release, and the chart removes no image. The kubelet removes unused images when the image filesystem is fuller than imageGCHighThresholdPercent (85 % by default), until it is below imageGCLowThresholdPercent (80 %). On one host, set lower thresholds and a maximum age, so that the images of old releases go before the disk fills. On k3s, put a KubeletConfiguration file whose name ends in .conf into /var/lib/rancher/k3s/agent/etc/kubelet.conf.d/, then run sudo systemctl restart k3s:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
imageGCHighThresholdPercent: 75
imageGCLowThresholdPercent: 65
imageMaximumGCAge: 168h

The thresholds count the whole filesystem, also data that is not an image. If other data keeps the disk above the low threshold, the kubelet logs that image garbage collection failed, and it removes every unused image. The kubelet never removes an image that a Pod uses.

Pod security

Each pod meets the Kubernetes Restricted Pod Security Standard:

• A non-root user, RuntimeDefault seccomp, and no privilege escalation.
• All capabilities dropped and a read-only root filesystem.
• No host path, host network, or container runtime socket.
• No service account token, except on the worker. Its Role reaches only the sandbox namespace.

The webapp runs as user and group 101 and listens on port 8080. Its only writable path is a 16 MiB volume at /tmp/hephaestus-webapp. The image writes its generated files, its nginx PID file, and its temporary directories there. /tmp itself and the assets stay read-only.

The bundled PostgreSQL runs as the image user 999 and owns its volume through fsGroup. The volume is at /var/lib/postgresql, as ADR 0038 decides. The image keeps its versioned data directory below it, and the chart does not set PGDATA. The image's entrypoint starts PostgreSQL with ssl=on and the certificate and key from the TLS Secret, read-only at /etc/postgresql/tls. No real volume driver has shown this configuration yet. First initialization, restart, and restore need a start of the release image on a volume before this path counts as working.

Every container of the release has CPU, memory, and ephemeral storage requests and limits. The schema refuses a zero value. Sandbox containers have CPU, memory, and disk limits (Sandbox sizes). They request the memory that their kind of sandbox was measured to use, below their limit (Sandbox sizes). They request a tenth of their CPU limit, because the CPU limit is a share of CPU time, not a reserved core (Sizing).

Profiles

Four values files in profiles/ set capacity for one use. All four use the same templates. Each release chart contains them in chart-$VERSION/hephaestus/profiles/ after you verify the release chart. Give the profile as the first -f file, before your own values, as Install shows.

Profile	Use	PostgreSQL and NATS
dev.yaml	One workstation cluster	Bundled, small volumes
preview.yaml	One pull request preview	Bundled, small volumes
small.yaml	One small cluster, for example one k3s node	Bundled, bounded streams
small-16gib.yaml, on top of small.yaml	One k3s node of 16 GiB	Bundled, bounded streams

small.yaml fits one node of 4 vCPU, 32 GiB and 150 GiB of disk, with one worker that runs one review and up to 8 conversation turns at a time. A chart test adds up what that node holds during an upgrade, with backups on:

Part	Memory	CPU requests	Disk
Server, webhook role, worker, webapp, PostgreSQL with pgBackRest, NATS	9.8 GiB	1.25	The volumes of PostgreSQL and NATS, and bounded caches
An upgrade: the new worker, webhook and webapp Pods beside the old ones, the migration Job, the stanza Job and one backup	5.8 GiB	1.00	The new Pods' caches
The sandbox quotas during the upgrade: in the reviews tier one review and the isolation probes of both workers, in the conversations tier the conversation host, its two pooled turn sandboxes and 8 running turns of each worker	5.8 GiB	1.00	The disk requests of the sandbox Pods (Sandbox sizes)
Total, against the node less platform reservations and the hard eviction floor	21.4 GiB of 30.25 GiB	3.25 of 3.5	80 GiB of the 110 GiB left beside images and review workspaces

The worker upgrades by a rolling update with a surge of one. The new worker passes its isolation probe and opens its conversation host's command channel before the old worker drains. Hephaestus keeps answering during the upgrade. The old worker finishes its running review within its drain of up to 60 minutes. The new worker's review waits for that review's room in the reviews quota. A turn sandbox requests 25m of CPU, so a second conversation host and turn pool fit beside the old ones.

The table gives the measured resident memory of each role and the chart's sizes. Each JVM role requests all of its memory limit, because the heap is sized from the limit, and the limit is the measurement plus headroom. PostgreSQL and its backup containers also request their full memory limits. CPU remains burstable, so these Pods keep the Burstable quality-of-service class. NATS requests less than its limit, because JetStream's caches grow above the request.

Role	Measured	Request	Limit
Server	1.5 GiB	2.5 GiB	2.5 GiB
Worker	0.8 to 1.4 GiB	2 GiB	2 GiB
Webhook role	1.0 GiB	1.25 GiB	1.25 GiB
PostgreSQL, at rest in the earlier staging capture	0.7 GiB	2 GiB	2 GiB
pgBackRest	Not measured	512 MiB	512 MiB
NATS, with JetStream's memory and file caches	1.3 to 2.7 GiB	1.5 GiB	3 GiB

The server's and the worker's caches have a bound of 10 GiB each, so that the node does not run out of disk and evict a draining worker. On a node of 16 GiB, the Pods of an upgrade with small.yaml alone do not fit beside a running review: they stay pending, the previous release keeps serving, and the upgrade rolls back when its timeout ends.

For one node of 16 GiB, apply small-16gib.yaml after small.yaml:

--values deploy/chart/hephaestus/profiles/small.yaml \
--values deploy/chart/hephaestus/profiles/small-16gib.yaml

The overlay lowers the NATS request to 512 MiB and keeps its limit at 3 GiB. It also changes worker updates to Recreate, with a drain of at most 10 minutes. The worker still runs one review and up to 8 conversation turns at a time, as in small.yaml. A second worker's conversation host and turn pool do not fit in 16 GiB beside the first: the upgrade stops reviews and conversation turns for the drain, the new worker's start and its isolation probe. The webhook role keeps its rolling update: it receives provider deliveries during an upgrade.

A chart test adds up what one node of 16 GiB holds, with backups on:

State	Memory	CPU requests
Steady: every role, one backup, one review, the conversation host, two pooled turn sandboxes and 8 running turns	11.8 GiB of 13.9 GiB	1.75 of 3.5
The same, with the isolation probe	12.5 GiB	—
An upgrade: the old worker gone, the webhook and webapp surge, the migration and stanza Jobs, the new worker's probe, and the old worker's pooled turn sandboxes	13.2 GiB of 13.9 GiB	2.40 of 3.5

The table counts requests. Its node budget uses the recorded Node capacity of 16 372 036 KiB, less the current platform reservations and hard eviction floor. It does not qualify the peak memory of running work. Sandbox Pods may use more than they request, up to their limits. If every sandbox Pod used its limit at the same time, the node would be full.

A chart test derives the conversation turns of both single-node profiles: 8, the worker's automatic number, as long as one more turn sandbox would still fit.

An earlier staging capture recorded these values on one node of 16 GiB with k3s 1.37. Its request figures precede the PostgreSQL memory-request change:

• At rest, the release requests 10.3 GiB of 15.6 GiB and uses about 5.9 GiB on the host: the server 2.0 GiB, the worker 1.3 GiB, the webhook role 0.8 GiB, PostgreSQL 0.7 GiB and NATS 60 MiB.
• An upgrade that recreated the worker and restarted PostgreSQL stopped reviews for about 4 minutes, with the isolation probe. The webhook role kept receiving.
• A point-in-time restore of a database of 1.8 GB took 152 s. The release served again 5 minutes 41 seconds after the restore started.
• The peak during a review and a conversation turn is not measured yet.

For a cluster with several nodes, start from small.yaml, and set the workers, their reviews, their quota and the external services in your own values file.

ADR 0035 owns what a preview can run. The preview profile sets only capacity.

External PostgreSQL and NATS

Set postgresql.bundled=false to use your PostgreSQL server. That server must have the pg_partman extension. The chart connects to postgresql.external.host and postgresql.external.port with sslmode=verify-full. The server certificate must name that host. The Secret in postgresql.tls.existingSecret needs only ca.crt, the CA that signed the server certificate. The chart has no setting for a connection without certificate verification or without TLS.

Set nats.bundled=false to use your NATS JetStream broker. Put the broker on a private network that only this release can reach. The released NATS client verifies the certificate chain of a tls:// broker, but not the host name in the certificate. Thus, TLS alone does not prevent a connection to another broker that has a certificate from the same CA. The bundled broker stays inside the release namespace, and network policy admits only the server and the webhook role. The application has no setting for the stream replica count. A three-server JetStream cluster therefore keeps one replica of each stream.

Install on one host (k3s)

This procedure installs Hephaestus on one Linux host with k3s and the Helm chart. One command, k3s-install.sh, does the install. It checks the host, installs the pinned k3s and the platform, and keeps the release Secret. Then it verifies the release chart and installs it. It uses the small.yaml profile, the bundled PostgreSQL and NATS, and one worker that runs reviews.

A second run of the same release changes nothing that the first run made. A run with the files of a newer release upgrades the host.

Before it publishes a release, the release workflow runs k3s-install.sh on Ubuntu 24.04 on amd64 and on arm64. It runs the command twice, then an upgrade that restarts every role except PostgreSQL and NATS, whose Pods must stay. That run differs from your install in these points:

• A local CA issues the Gateway certificate, because Let's Encrypt cannot reach a CI runner. CI does not test the HTTP-01 challenge of the Let's Encrypt issuer.
• It skips the DNS check, because a runner has no public host name.
• It skips the clock check: a hosted runner takes its time from the hypervisor and can report no NTP sync.
• A runner has 16 GiB of memory, so the installer adds the small-16gib.yaml profile.
• It checks the shutdown settings that the kubelet and logind use. It does not reboot the host.

The platform files

Each release publishes the files of the one-host install as signed assets.

Asset	What it is
k3s-install.sh	The installer. It is a POSIX shell script, and it needs no Node.js and no checkout of the repository.
k3s-platform.yaml	The pins and the platform add-ons. It goes to the k3s manifests directory.
k3s-certificates.yaml	The Let's Encrypt issuer, the Gateway certificate, and the PostgreSQL CA and server certificate. The installer writes your host name and address into it.
k3s-config.yaml	The k3s configuration: the kubelet configuration file, and no bundled Traefik.
k3s-kubelet.yaml	A limit of 4096 processes for each Pod, the shutdown grace period, the memory and CPU reserved for k3s and the host, and the eviction thresholds.
k3s-logind.conf	How long logind lets the kubelet hold a shutdown.
k3s-journald.conf	Host journal limits: 512 MiB on disk, 128 MiB in memory, and seven days.
k3s-values.yaml	The chart values of every one-host install: the Gateway, the Traefik Pods, the Secret names, the sandbox namespace, and reviews with repository checkouts.

k3s-platform.yaml contains these add-ons:

• Traefik as the Gateway controller on the host's ports 80 and 443. Its chart creates the Gateway traefik/traefik-gateway and a redirect from HTTP to HTTPS.
• cert-manager, with Gateway API support.

The pin: k3s line of k3s-platform.yaml sets the k3s release, and its two HelmChart objects set the versions of Traefik and cert-manager. The file pins each chart by version and each image by digest. The Gateway API resource definitions come from the gateway-api-crd chart that k3s includes, so no second owner installs them. k3s deploys each file in /var/lib/rancher/k3s/server/manifests/, and it applies a file again every 15 seconds until the apply succeeds. Thus, cert-manager and the Gateway start some minutes after k3s.

The installer does not run the application itself as a k3s HelmChart. The k3s Helm controller does not wait for the migration Job, and it does not roll back a failed upgrade.

The limit of 4096 processes applies to each Pod on the host, the server and the worker included. On Compose, the server and the worker had a limit of 512 each, so a lower limit for every Pod could stop them. CI's clusters run with the same 4096.

When the host shuts down, the kubelet stops the Pods first. The configured budget is 5 minutes: 4 minutes for the application Pods, then 1 minute for Traefik and CoreDNS. PostgreSQL stops within its termination grace period of 120 seconds, inside the first 4 minutes. Ubuntu sets the logind limit to 30 seconds, so k3s-logind.conf raises it to 5 minutes. At the end of the budget, a worker stops the reviews that it runs. They run again after the start.

Pods cannot request all of the host's memory. k3s-kubelet.yaml reserves 1 GiB and 250m CPU for k3s, which runs the API server, the kubelet and containerd in one process of about 1 GB. It reserves 512 MiB and 250m CPU for the host's own services. The kubelet evicts Pods when less than 256 MiB of memory is available, before the kernel's out-of-memory killer stops a process on the host.

For a Node API capacity of exactly 16 GiB and 4 vCPU, this leaves 14 592 MiB (14.25 GiB) and 3.5 CPUs for Pods. At the recorded capacity of 16 372 036 KiB used in the profile table, Pods get about 13.9 GiB.

The installer restarts k3s when this file changes. After the node becomes ready, it reads the kubelet configuration endpoint and Node allocatable. It stops before the application upgrade if the configured policy is absent or overridden. Allocatable must equal measured capacity less the reservations and the hard memory floor. The soft threshold does not reduce allocatable again. Only Pod allocatable is enforced. The reservations do not impose hard memory caps on the host or k3s system slices. To repeat these read-only cluster checks with your current kubeconfig, run sh k3s-install.sh --verify-kubelet. The command uses Helm locally to parse the policy; it does not change the cluster. After a restart, verification allows 31 checks with two-second pauses for the Node status and configuration endpoint to converge.

Soft eviction is disabled until stateful-service requests and admission protection are qualified. For memory eviction, usage above requests ranks before Pod priority. Thus, a higher priority cannot protect PostgreSQL or NATS above their requests from a review below its request. The existing hard thresholds remain emergency recovery boundaries. The kubelet targets an additional 256 MiB of reclaim and holds a pressure condition for five minutes. Application admission must stop new background work before those boundaries. Container logs rotate at 10 MiB with at most five files per container. Image garbage collection starts at 75 percent disk use and targets 65 percent. It does not reclaim live volumes or database files. The installer bounds the host journal and restarts journald only when its policy changes.

Host swap must be disabled, including at boot. The installer rejects active swap, and the kubelet requires failSwapOn=true and NoSwap. These policies do not qualify a smaller container memory limit or a higher review count. Before enabling either, measure synchronized node peaks, pressure, queue waits, and OOM counters under the reference load. The verifier accepts the declared Ubuntu 24.04/kernel 6.8 GA baseline and rejects unknown OS or kernel families. The optional --allow-unqualified-host flag permits an explicitly authorized rehearsal on another family. It preserves all effective-policy and allocatable checks. A matching host family alone does not prove installation, pressure behavior, or loaded capacity.

The current proof consists of policy/parser fixtures, read-only Node API observations, and rendered chart checks. The release host smoke targets Ubuntu 24.04 on both architectures; this change has not run a full host installation.

Before target-host rollout, the release lane must rehearse the exact assets and pinned k3s release on disposable Ubuntu 26.04/kernel 7 with 16 GiB. Use the qualification override for that rehearsal and record the actual OS, kernel, CPU count, capacity, and allocatable. Run the installer, then repeat it and verify stateful Pod identity and service continuity. The second run must preserve unchanged configuration and avoid unnecessary k3s or journald restarts. Verify disabled swap and its boot activation settings.

Apply pressure with PostgreSQL and NATS above their requests and review/turn Pods below theirs. Require application backoff and recovery before hard eviction, with no database/broker loss or OOM. Observe container and journal rollover, image GC at 75/65, and image pull/start latency. Retain the command logs, effective configuration, pressure/OOM counters, and synchronized footprint samples as release evidence.

Requirements

• A host of 4 vCPU, 32 GiB of memory, and 150 GiB of disk, as Profiles gives for small.yaml. On a host of 16 GiB, the installer adds small-16gib.yaml, and each upgrade stops reviews and conversations for a while.
• Ubuntu 24.04 with its 6.8 GA kernel on amd64 or arm64. Other OS or kernel families require explicit qualification with --allow-unqualified-host.
• A DNS A record for your host name that points to the public address of the host.
• Ports 80 and 443 open from the internet, for Let's Encrypt, OAuth callbacks, and provider webhooks. Keep port 6443, the Kubernetes API, and port 10250, the kubelet, closed to the internet.
• If ufw is active, rules that let in ports 80 and 443 and the k3s networks: sudo ufw allow 80,443/tcp, sudo ufw allow from 10.42.0.0/16, and sudo ufw allow from 10.43.0.0/16.
• Outbound HTTPS to the destinations in Network egress.
• curl, jq, openssl, python3, Cosign, and the Helm CLI, as Verify the release chart describes. Ubuntu includes python3.
• A clock that NTP keeps synchronized, as Clocks describes.
• A GitHub OAuth app for sign-in, as First steps § 1 describes. Its callback URL is https://<host>/api/login/oauth2/code/github.

What the installer checks

The installer stops before it changes the host when one of these checks fails. Each message tells you what to change.

Check	Condition
Input	The host name, the address, the OAuth client ID, and the admin list have a strict format.
Size	The host has 4 CPUs and at least 15 GiB of memory. Below 30 GiB, the installer adds the small-16gib.yaml profile.
Clock	NTP synchronizes the clock.
Swap	No swap device or file is active. The kubelet also refuses swap.
Ports	No other program listens on port 80 or 443. The installer checks this before k3s runs.
Host firewall	If ufw is active, it allows 80/tcp and 443/tcp by a port rule or by the Nginx, Apache, or WWW profile. If firewalld is active, the zone of the default route allows http and https, or ports 80 and 443. Otherwise the installer stops and gives the command to run.
DNS	The host name resolves to this host. Before k3s runs, the installer serves a random token on port 80 for a moment, and it fetches the token through the host name.

The DNS check proves only that the host name leads to this host. The request can stay on the host, so it does not prove that the internet reaches port 80. A cloud firewall or a security group of your provider is yours to open, and the installer cannot check it.

cert-manager needs port 80 from the internet to get the certificate. The installer waits for the certificate for 10 minutes, and then it stops with the command that shows the reason. If your network cannot reach its own public address, the DNS check fails. Then run the installer with --skip-check dns.

1. Get the release files

1. Make a directory for the release files. The installer keeps its state there: site.yaml, the verified chart, and a kubeconfig file.

   VERSION=0.95.0   # the release, without the leading "v"
   mkdir -p ~/hephaestus && cd ~/hephaestus

2. Download each asset and its signature, and verify the signature. Cosign stops with an error if a file does not come from the release workflow:

   release="https://github.com/hephaestus-build/Hephaestus/releases/download/v$VERSION"
   for file in k3s-install.sh k3s-platform.yaml k3s-certificates.yaml k3s-config.yaml k3s-kubelet.yaml k3s-logind.conf k3s-journald.conf k3s-values.yaml; do
     curl -fsSLO "$release/$file"
     curl -fsSLO "$release/$file.sigstore.json"
     cosign verify-blob --bundle "$file.sigstore.json" \
       --certificate-identity https://github.com/hephaestus-build/Hephaestus/.github/workflows/release.yml@refs/heads/main \
       --certificate-oidc-issuer https://token.actions.githubusercontent.com \
       "$file"
   done

2. Run the installer

1. Type the client secret of your GitHub OAuth app. The shell does not show it:

   read -rs GITHUB_OAUTH_CLIENT_SECRET && export GITHUB_OAUTH_CLIENT_SECRET

2. Run the installer. Give your host name, and the address that gets the expiry notices of Let's Encrypt. Also give your OAuth client ID and your numeric GitHub user ID:

   sh k3s-install.sh --version "$VERSION" --host hephaestus.example.com --email ops@example.com \
     --github-client-id <the client ID of your OAuth app> --bootstrap-admins github:<your numeric GitHub user ID>

   For a first try, add --staging: the Let's Encrypt staging service has higher rate limits, and no browser trusts its certificate. Later, run the installer again without --staging, with the explicit certificate choice described below. Delete the Secret traefik/hephaestus-tls to get a trusted certificate. First steps § 2 explains the admin list.

3. Keep the required keys of the release Secret outside the host, as Keep the required keys outside the cluster describes. For kubectl on this host, run export KUBECONFIG=~/hephaestus/kubeconfig first.

4. Open https://<host> and sign in as the bootstrap admin.

5. Continue with First steps § 4.

The installer keeps every key of the release Secret hephaestus-app and generates only the keys that it lacks. It generates each key as Secret custody and rotation gives it. If the PostgreSQL volume exists and the Secret lacks the database login or an encryption key, the installer stops: restore the Secret from your backup.

The installer writes site.yaml only when the file does not exist. It holds the host name, the Kubernetes API endpoint, the OAuth client ID, the admin list, and config.gitCheckoutEnabled: true. Change the file to change these values, and run the installer again.

On a repeat run, select certificates explicitly:

• For standard apex certificates, pass --standard-certificates. Omit --staging to use the production issuer.
• For standard workspace certificates, repeat --workspace-subdomains --dns01-issuer NAME.
• For custom certificates, including workspace hosts, pass the intended manifest with --certificates FILE. Keep its issuer and every required apex and wildcard name.

This check prevents a repeat run from replacing wildcard TLS with an implicit apex-only certificate. --render-certificates previews the selected manifest without host changes and needs no repeat-install choice.

The installed manifest is root-only, mode 0600. Before using it with --certificates, copy it to a file that you own:

sudo install -m 0600 -o "$(id -u)" -g "$(id -g)" \
  /var/lib/rancher/k3s/server/manifests/hephaestus-certificates.yaml ./saved-certificates.yaml

Then pass --certificates ./saved-certificates.yaml. This preserves the existing issuer, including a staging issuer.

A new GITHUB_OAUTH_CLIENT_SECRET replaces the stored one and restarts the server.

Certificate renewal

cert-manager renews the Gateway certificate and the PostgreSQL certificates before they expire. Traefik reads the renewed Gateway certificate without a restart. The PostgreSQL server certificate is valid for one year, and cert-manager renews it 30 days before it expires. PostgreSQL reads it only when it starts, as TLS certificates describes. The CA of the PostgreSQL certificate is valid for ten years.

1. Find the time of the next renewal:

   kubectl --namespace hephaestus get certificate hephaestus-postgresql-tls -o jsonpath='{.status.renewalTime}'

2. After that time, restart PostgreSQL:

   kubectl --namespace hephaestus rollout restart statefulset/hephaestus-postgresql

Upgrade the host

1. Get the files of the new release into the same directory, as Get the release files describes.
2. Run the installer of the new release with the same arguments. It installs a new k3s pin, replaces the platform files, and runs the upgrade that Upgrades describes.

Known limits of the one-host platform

• Traefik has no request body limit on this platform. Its default idle timeout is 180 seconds.
• The k3s default Pod network is 10.42.0.0/16. trustedProxies in k3s-values.yaml matches it. If you change the network of k3s, change trustedProxies too.
• The platform issues a separate backup client certificate, hephaestus-postgresql-backup-tls. Enable backups after you configure the off-host repository (Backup and restore).
• If the model endpoint resolves to this host's own Gateway, set networkPolicy.workerGatewayPorts: [8443, 8000] in site.yaml, as Network policy describes.
• CI's clusters run no cert-manager. Only the release workflow's install tests the cert-manager pin.

Install

This procedure installs the chart on a cluster that you operate. For one host, use Install on one host (k3s).

1. Create the release namespace and the sandbox namespace, and label both with the Restricted Pod Security level. Use the sandbox namespace for this release alone.

   kubectl create namespace hephaestus
   kubectl create namespace hephaestus-sandboxes
   kubectl label namespace hephaestus hephaestus-sandboxes pod-security.kubernetes.io/enforce=restricted

2. Make the OAuth App of your login provider, as First steps § 1 describes. Then create the Secret with its client secret, and the PostgreSQL TLS Secret. The server copies the client secret into the database at its first start.

3. Write your values file, site.yaml, with host, trustedProxies, secret.name, postgresql.tls.existingSecret, route.parentRefs, and networkPolicy.gateway with its pod labels. Add config.github.oauthClientId and config.auth.bootstrapAdmins, as First steps § 1 and § 2 describe. Do not set release or images: the release chart sets them. For a public instance, create the ConfigMap with your legal pages and name it in webapp.legalOverrides, as Legal pages describes. The webapp Pod waits for a ConfigMap that it names.

4. Enable reviews. small.yaml enables the worker, and the chart does not render it without the sandbox namespace and the Kubernetes API endpoint. Find the endpoint that the network plugin sees:

   kubectl get endpointslices --namespace default --selector kubernetes.io/service-name=kubernetes

   On a single k3s node, PORTS is 6443 and ENDPOINTS is the node's internal address, for example 192.168.5.15. Add both values to site.yaml:

   worker:
     sandbox:
       namespace: hephaestus-sandboxes
   networkPolicy:
     kubernetesApi:
       cidr: 192.168.5.15/32
       port: 6443

   Use /128 for an IPv6 address. If ENDPOINTS lists more than one address, one for each control plane node, set a CIDR that holds all of them.

5. Verify the release chart.

6. Install the verified chart with a profile and your values:

   helm upgrade --install hephaestus "chart-$VERSION/hephaestus" -n hephaestus \
     -f "chart-$VERSION/hephaestus/profiles/small.yaml" -f site.yaml \
     --rollback-on-failure --wait=watcher --wait-for-jobs --timeout 20m

7. Follow the migration Job's log until the migrations are complete: kubectl --namespace hephaestus logs job/hephaestus-migration-1.

8. Wait until the worker is ready. A worker is ready only after its isolation probe passed and it watches the sandbox namespace.

   kubectl --namespace hephaestus rollout status deployment/hephaestus-worker

   If the worker does not become ready, read Troubleshooting.

9. Continue with First steps § 3. Sign in.

The schema refuses an unknown key, an image tag, and a value outside its bounds. Each object name is <release>-<component>, with at most 52 characters. A StatefulSet revision label adds up to 11 characters to its name, and a label value has at most 63. If a name is longer, the chart keeps its start and adds 8 characters of a hash of the full name. The same release always gets the same names. The app.kubernetes.io/instance label always holds the full release name.

config.auth.assuranceEnforced controls the passkey and recent sign-in checks for administration. Its default is true, as in the application. The chart never changes it from the environment name. To turn it off, for example on a staging instance, set it to false yourself.

These values set the server's settings of the same meaning (Production setup):

Value	Setting	Default
config.researchOrganization	HEPHAESTUS_RESEARCH_ORGANIZATION	Empty
config.publicActivity.enabled	HEPHAESTUS_PUBLIC_ACTIVITY_ENABLED	false
config.publicActivity.repositoryVisibilityMaxAge	HEPHAESTUS_PUBLIC_ACTIVITY_REPOSITORY_VISIBILITY_MAX_AGE	48h
config.llmDisplayCurrency	HEPHAESTUS_LLM_DISPLAY_CURRENCY: empty or EUR	Empty
config.auth.passkeys.instanceAdminRequired	HEPHAESTUS_AUTH_PASSKEYS_INSTANCE_ADMIN_REQUIRED	true
config.auth.passkeys.workspaceAdminRequired	HEPHAESTUS_AUTH_PASSKEYS_WORKSPACE_ADMIN_REQUIRED	false
config.auth.passkeys.rpId	HEPHAESTUS_AUTH_PASSKEYS_RP_ID	Empty: the host
config.auth.passkeys.allowedOrigins	HEPHAESTUS_AUTH_PASSKEYS_ALLOWED_ORIGINS	Empty: the host
config.email.tls	starttls sets SPRING_MAIL_PROPERTIES_MAIL_SMTP_STARTTLS_ENABLE and _REQUIRED; implicit sets SPRING_MAIL_PROPERTIES_MAIL_SMTP_SSL_ENABLE. SPRING_MAIL_PROPERTIES_MAIL_SMTP_AUTH is on when config.email.username is set.	starttls
config.monitoring.runOnStartup	MONITORING_RUN_ON_STARTUP	true
config.monitoring.timeframeDays	MONITORING_TIMEFRAME	7
config.monitoring.syncCron	MONITORING_SYNC_CRON, a Spring cron expression with six fields	0 0 * * * *
config.monitoring.cooldownMinutes	MONITORING_SYNC_COOLDOWN_IN_MINUTES	60

webapp.securityContact sets the webapp's SECURITY_CONTACT: with a value, /.well-known/security.txt names it as the contact of this deployment (Production setup). Empty serves no file.

Integration values

Each integration takes its settings from site.yaml and its secrets from the Secret. values.schema.json describes each value. Integration settings explains each setting.

Integration	Values	Secret keys
GitHub login	config.github.oauthClientId	GITHUB_OAUTH_CLIENT_SECRET
GitHub App	config.github.appId, config.github.appClientId, config.github.appInstallationUrl	GH_APP_PRIVATE_KEY, GH_APP_CLIENT_SECRET
GitLab	config.gitlab.*	GITLAB_OAUTH_CLIENT_SECRET
Slack	config.slack.*	HEPHAESTUS_INTEGRATION_SLACK_SIGNING_SECRET, HEPHAESTUS_INTEGRATION_SLACK_CLIENT_SECRET
Outline	config.outline.*	OUTLINE_OAUTH_CLIENT_SECRET
Email	config.email.*	SPRING_MAIL_PASSWORD
Provider webhooks	None	WEBHOOK_SECRET, WEBHOOK_ROUTING_SECRET

Worker

The worker role runs every review sandbox, the conversation host, and every turn sandbox. The chart's default disables the worker, and small.yaml enables it. Prepare the sandbox prerequisites. Set worker.sandbox.namespace and networkPolicy.kubernetesApi (Install, step 4). The chart refuses a worker without them.

The chart runs the workers as one Deployment:

• Two replicas by default, spread across nodes where the cluster has several. worker.replicas sets the count.
• A rollout starts all new workers before an old one stops, and a PodDisruptionBudget lets a node drain take one worker at a time. worker.rolloutStrategy=Recreate stops the old workers first, for a node that cannot hold both (Profiles).
• A stopping worker claims no new work and serves its running attempts for worker.drainTimeoutSeconds, 3600 by default. A practice review usually takes 10 to 16 minutes. An attempt that outlasts the drain is tried again.
• A worker on a lost node is replaced after 60 seconds.
• AGENT_ENABLED=true on the workers and the server, so that the server submits practice reviews and the workers run them. Without a worker, the server submits none.
• worker.reviews sets the reviews that one worker runs at once, and worker.conversationTurns the conversation turns (HEPHAESTUS_WORKER_CAPACITY_MENTOR_MAX, auto by default). Each running turn holds a turn sandbox beside the 2 pooled ones, and the conversations quota counts them (Runtime roles § Conversation turn capacity). small.yaml sets 8.
• The settings HEPHAESTUS_SANDBOX_RUNTIME=kubernetes, the sandbox namespace, and the pod name and address from the Downward API (Runtime roles § Sandbox runtime).
• The existing gateway setting SANDBOX_API_PORT, with the value 8081.

The sandbox namespace is the one that you created for this release. The chart creates no namespace. It writes these objects into it:

• One ResourceQuota for each PriorityClass tier: reviews, and conversations for the conversation host and turn sandboxes. The chart derives both from sandbox-pod-requests.json and from worker.replicas, worker.reviews and worker.conversationTurns. They count requests, not limits: sandbox Pods may use more than they request, up to their limits. Thus, Hephaestus's sandboxes never take a review's room, and reviews never take Hephaestus's. Each tier's quota counts Pods, CPU and memory. Kubernetes does not let a quota that selects a PriorityClass count disk. The quota without a scope counts the disk requests of both tiers together. A quota also counts the Pod overhead of a RuntimeClass, such as gVisor's on some clusters. For such a class, set worker.runtime.podOverhead to its overhead.podFixed: cpuMillis and memoryMi. The chart then adds the overhead to each Pod in the quotas. Set worker.sandbox.quota.enabled=false to manage the quotas yourself. priorityClasses.reviews and priorityClasses.conversations must name different classes, because each quota selects its Pods by class. During a rolling update, the chart's quotas hold the workers of the new and the old version. A worker that still drains during a second update can find no room: its sandbox then waits and is tried again. A worker claims a practice review only when every quota that counts review Pods has room for the review's Pod. An unscoped quota counts them, and so does a quota that selects the reviews' PriorityClass. It counts the reviews that it claimed but has not started yet. Otherwise the review stays queued. The worker's Role lets it list the quotas; its permissions are listed below. It reads the Pod overhead of the sandbox RuntimeClass from its own sandbox Pods. A quota with another scope does not hold a review back, and neither do quotas that the worker cannot read. Another worker can still take the room after the claim. The quota then refuses the review's Pod, and the review is tried again.
• A deny-all policy for ingress and egress.
• One more policy that lets review and host sandboxes reach port 8081 of the worker Pods, and nothing else. Those sandboxes carry the label hephaestus.build/sandbox-kind from the runtime. Turn sandboxes get no other policy, so they have no network. The worker policy admits the same sandboxes on the same port.
• The worker's Role and RoleBinding.

Priorities

Under memory pressure, the kubelet evicts Pods of a lower priority first. The scheduler can also preempt Pods of a lower priority to place one of a higher priority. The chart gives every Pod one of three priorities, highest first:

1. The roles, PostgreSQL, NATS, and the migration and backup Jobs. They serve everyone.
2. The conversation host and the turn sandboxes. A person waits for the answer.
3. The review sandboxes and the isolation probe. A review is an attempt that runs again. It never preempts another Pod.

A preempted or evicted review counts as an attempt that the worker lost, and the review runs again.

By default, the chart creates the three PriorityClasses. If your account cannot create cluster-wide objects, ask the cluster's administrator for three classes, with values in the order above. Then set priorityClasses.create=false and name them:

priorityClasses:
  create: false
  roles: platform-high
  conversations: people-waiting
  reviews: batch-low

The chart refuses a name while create is true, and a missing name while it is false.

The workers remove what no attempt and no worker uses in the sandbox namespace. Do not share it with another release or another application.

When the sandbox quota is full

A review waits in the queue while the quota has no room for its sandbox. Waiting behind running reviews is normal backlog, and it never ends a review.

Two cases end queued reviews, with the stop reason NO_SANDBOX_ROOM:

• A quota is smaller than one review sandbox. The worker cancels the queued reviews within 30 seconds, with "The sandbox quota is smaller than one review sandbox." It logs one warning.
• Reviews have waited for an hour while the quota had no room for one and no review ran, so nothing frees room. Usually a quota of your own holds the room. The worker logs a warning when this starts. After the hour, it cancels the queued reviews with "Hephaestus had no room to run this review in time."

A scan that loses a file's job this way ends with the same reason. It does not queue its next file.

A review whose Pod the quota refuses after its claim also waits, and spends none of its retries. Its next attempt gets a new number, as every attempt does. The waits grow from 30 seconds to 5 minutes. The number of refusals never ends a review. Only the two cases above do.

The metric sandbox_start_failures_total{kind="review",reason} counts these refusals and cancellations, with the reasons quota_exceeded and quota_cannot_fit. The gauge sandbox_review_starved_seconds shows how long no review could start while none ran. Alert on it well before an hour: the quota is too small for the sandboxes beside the reviews.

The worker learns of the refusal from the events of the sandbox namespace, and reads its quotas to tell whether a sandbox can fit at all. It needs get and list on events, and list and watch on resourcequotas there. The chart's Role grants them. Without the verbs on events, the worker logs one warning, and a refused review waits 2 minutes and spends a retry. Without list and watch on resourcequotas, pods or jobs, the worker stays not ready (Troubleshooting). The worker watches the Pods, Jobs and quotas of the sandbox namespace. Thus, a claim poll and a readiness probe send no request to the API server. A sandbox's credential, not the network, binds it to the worker that started it.

The worker reaches the Kubernetes API only at networkPolicy.kubernetesApi. Set the address and port that the selected network plugin sees after address translation. The plugins that CI runs, k3s's kube-router and Calico, see the endpoint of the kubernetes Service (Install, step 4). The chart has no default for them, because only the running cluster knows them.

The worker's service account has these permissions in the sandbox namespace, and none elsewhere:

• Pods: get, list, watch, create, delete, and patch. A cancellation or a timeout patches the deadline of the sandbox's Pod.
• pods/log: get.
• pods/exec: get and create. The client opens an exec as a WebSocket GET, and the API server also authorizes it as create.
• Jobs: get, list, watch, create, and delete.
• Events: get and list. A review Pod that the sandbox quota refuses waits for room: the Job controller reports the refusal as an event.
• ResourceQuotas: list and watch. The quota tells how many reviews fit and whether a review can ever fit.

A practice review attempt runs as one Job. The Job ends at the attempt's upload deadline plus two minutes, also when its worker is gone, and Kubernetes removes it 15 minutes after it ends. The review-ledger worker identity stays per process. The chart does not set HEPHAESTUS_WORKER_ID.

Sandbox prerequisites

The sandbox nodes need these items:

1. A network plugin that enforces NetworkPolicy, for egress as well as ingress.
2. The sandbox namespace with the label pod-security.kubernetes.io/enforce=restricted.
3. Encrypted node storage, for example LUKS or the disk encryption of the cloud provider. A sandbox Pod writes to the node's disk: the review workspace with the repository, /tmp, ~/.local and a turn sandbox's workspace. Only /run is in memory. Encryption protects a stolen disk and a backup of it without the key. It does not protect against a compromised running node or a privileged workload on the node. The kubelet deletes a Pod's volumes with the Pod, but deletion does not erase the data from the disk.

These settings are optional:

• worker.runtime.runtimeClassName: a RuntimeClass for sandbox Pods. gVisor (runsc) is recommended for workloads from people that you do not trust.
• worker.runtime.podOverhead: the RuntimeClass's overhead.podFixed, as cpuMillis and memoryMi. The sandbox quotas count it with each Pod.
• worker.runtime.seccompProfile: the bundled sandbox/agent-seccomp-profile.json as a Localhost profile. Put the file below the kubelet's seccomp directory on every sandbox node, and set its relative path. Without it, sandbox Pods use RuntimeDefault.
• worker.runtime.nodeSelector and worker.runtime.tolerations: the nodes of sandbox Pods.
• A kubelet podPidsLimit on the sandbox nodes.

Each worker runs the isolation probe in the sandbox namespace at its start, and then every worker.runtime.probeInterval, 15 minutes by default. The probe checks these items on the running cluster:

• The API server refuses, in a dry run, a Pod that the Baseline standard admits and the Restricted standard refuses.
• A Pod with the labels of a review sandbox reaches the worker's gateway port. It does not reach the worker's management port, the Kubernetes API, or the cluster DNS server.
• A Pod of no sandbox kind does not reach the gateway port. Turn sandboxes have the same policy.
• The probe Pods have no service account token, a read-only root filesystem, and a non-root user.
• A named RuntimeClass exists: without it, no probe Pod runs.

The worker starts no sandbox while one of these items fails. At its start, the worker then logs each failed item and stops, and Kubernetes starts it again. On a later probe, it logs each failed item, reports itself not ready, and starts no new sandbox until a probe passes again. Until a probe passes, at the start and after a failed later probe, the worker is not ready and claims no practice review and no conversation turn. They stay queued for a ready worker. Reviews and turns that already run continue. A later probe whose Pods cannot start, because the sandbox quota is full or the nodes have no room, proves nothing: the worker logs a warning and keeps its last passing proof. A Pod that the sandbox quota refuses delays that verdict by up to 2 minutes.

The probe also records the isolation tier of the cluster, which is not a requirement:

Item	Effect
RuntimeClass	The runtime that isolates sandboxes from the node kernel, or the cluster's default.
User namespaces	When a Pod with hostUsers: false runs in a user namespace of its own, every sandbox Pod runs that way. The root of a sandbox is then no user of the node.
Process limit	The number of processes that a sandbox Pod could start before the node refused one, or no limit.

The worker's log and the metrics sandbox.isolation.passed, sandbox.isolation.user_namespaces and sandbox.isolation.process_limit show the tier. The probe Pods use the agent image and the sandbox configuration. Each probe Pod is the Pod of a Job, and a run takes 768 MiB of the sandbox quota on the cluster's default runtime: 512 MiB for the Pod that tries 1024 processes, and 128 MiB for each of the two others. The worker deletes the Jobs when the probe ends. If the worker stops during a run, the Job's deadline ends a Pod that still waits for a node, and Kubernetes deletes the finished Job with its Pod after 15 minutes.

Some network plugins apply the policy of a new Pod a few seconds after the Pod starts. Each sandbox Pod therefore starts with an init container that waits, at most 60 seconds, until it cannot reach the Kubernetes API.

Sandbox sizes

The worker sizes each sandbox Pod from the measured use of that kind of sandbox. SANDBOX_MEMORY_BYTES applies to Docker only.

Sandbox Pod	CPU request	Memory request	Memory limit	Disk request	Size limit of each disk volume	Disk limit
Review	100m	640 MiB	1,280 MiB	1 GiB	1 GiB, 7 volumes	7 GiB and 64 MiB
conversation host	50m	256 MiB	1 GiB	256 MiB	1 GiB, 7 volumes	7 GiB and 64 MiB
Turn sandbox	20m, and 5m for its stager	128 MiB, and 32 MiB for its stager	512 MiB, and 32 MiB	256 MiB, and 16 MiB	512 MiB, 3 volumes	1.5 GiB and 64 MiB, and 16 MiB

A review and the conversation host have seven volumes on the node's disk: the workspace, work, out, .pi, .sessions, /tmp and ~/.local. A turn sandbox has three: its workspace, /tmp and ~/.local. The disk limit of the main container holds all the volumes full and 64 MiB for its logs. Each other container, such as the isolation init container and the stager, has a disk limit of 16 MiB for its logs.

The stager and each probe Pod request a tenth of their CPU limit. The isolation init container requests 10m. The chart's sandbox quotas count the requests of this table, not the limits. Four large reviews at once peaked at 632 to 657 MiB each. One conversation used 136 MiB in the host, and a turn sandbox used 3 MiB. The host's Node heap stops at 768 MiB.

/run is in memory, with 16 MiB. The kubelet evicts a Pod whose volume passes its size limit, or whose disk use passes its disk limit. Thus, no sandbox can fill the node's disk and cause the eviction of other Pods. The worker counts the eviction as a failure of the infrastructure, and the review spends a retry: a review whose sandbox keeps filling its disk ends at AGENT_MAX_RETRIES. The chart's sandbox quota without a scope counts the disk requests of all sandbox Pods.

Kubernetes has no field for two Docker sandbox settings. /tmp, /run and a turn sandbox's /workspace are not noexec, and a sandbox Pod keeps /dev/shm in its private IPC namespace. ADR 0041 records why.

Every container command and every exec starts through prlimit from the agent image. prlimit sets the nofile limit to 1024 and the core limit to 0, both soft and hard. A sandbox process cannot raise them.

Backup and restore

With postgresql.backup.enabled=true, the bundled PostgreSQL archives its WAL to one S3-compatible bucket with pgBackRest, as the Compose backup overlay does. The repository is encrypted with BACKUP_CIPHER_PASS. A CronJob takes a full backup each Sunday at 02:15 UTC, and another takes a differential backup on the other days at the same time. postgresql.backup.retentionFull full backup chains stay, 2 by default. A Job of each release creates the repository's stanza, so that the WAL archive works from the install.

The backup Jobs do not mount the PostgreSQL volume. They read the data directory through the pgBackRest TLS server in the PostgreSQL Pod. That server admits only a client certificate for the common name hephaestus-backup that ca.crt signed.

To prepare backups:

1. Create the bucket for this instance alone, and an identity with read, write and delete rights in it.
2. Add BACKUP_S3_KEY, BACKUP_S3_KEY_SECRET and BACKUP_CIPHER_PASS to the release Secret. Generate the passphrase and keep a copy as Secret custody describes.
3. Set postgresql.backup.clientTlsSecret to a TLS Secret for the common name hephaestus-backup. It needs ca.crt, tls.crt and tls.key, signed by the PostgreSQL CA. The one-host platform issues hephaestus-postgresql-backup-tls and sets this value.
4. Set postgresql.backup.enabled=true and postgresql.backup.s3 with the bucket, the endpoint host name and the region. Set cidr and port when the endpoint is in a private address range. When a private CA signs the endpoint's certificate, add it to the PostgreSQL TLS Secret as s3-ca.crt, and set postgresql.backup.s3.privateCa=true.
5. Upgrade the release, and check that the stanza Job completed.

Backup bucket credentials and Backup cipher passphrase give the rotation of these values.

A failed WAL archive keeps the WAL on the PostgreSQL volume until the volume is full. The alert rules watch the WAL archive, the age of the latest backup and the free space of the volume.

Restore

A restore replaces the bundled database with a backup. It loses every change after the backup or the time that you select. The restored database is readable only with the keys that sealed its values. Keep them as Secret custody describes. Restore the NATS JetStream data separately, as Backup and restore describes.

Isolated restore drill

Run the following procedure only on a disposable, isolated clone. It changes control settings with clone lockdown SQL. Use Recover the live installation for a live recovery.

1. Upgrade the release with postgresql.backup.restore.enabled=true. Set postgresql.backup.restore.time to restore the database as it was at that time, for example 2026-10-24 18:00:00+00. Leave it empty to restore the latest backup and every archived WAL segment. While the setting is on, the server, the webhook role, the workers and PostgreSQL stop, and the backups pause.

2. Wait for the Job <release>-postgresql-backup-restore-<revision> to complete:

   kubectl --namespace hephaestus wait --for=condition=complete \
     job/hephaestus-postgresql-backup-restore-<revision> --timeout=30m

   The Job retries while PostgreSQL still shuts down.

3. Keep the clone isolated before any application code starts. Deny provider and model egress independently of silent mode.

4. Upgrade with postgresql.backup.restore.enabled=false, migration.enabled=false, and server, webhook and worker replica counts of zero. PostgreSQL alone replays the WAL up to the target. Readiness does not prove that replay finished. Wait until this command prints f:

   kubectl -n hephaestus exec hephaestus-postgresql-0 -c postgresql -- \
     sh -c 'exec psql -Atc "SELECT pg_is_in_recovery()" -U "$POSTGRES_USER" -d "$POSTGRES_DB"'

5. Apply the lockdown SQL from the verified chart, before starting migrations or application roles:

   kubectl -n hephaestus exec -i hephaestus-postgresql-0 -c postgresql -- \
     sh -c 'exec psql -v ON_ERROR_STOP=1 -U "$POSTGRES_USER" -d "$POSTGRES_DB"' \
     < "chart-$VERSION/hephaestus/files/restore-clone-lockdown.sql"

6. Read back silent mode, paused feedback and disabled conversation bindings. Keep external egress denied.

7. Restore JetStream with a recovery point consistent with PostgreSQL. Resolve provider write receipts before consumers start.

8. Set migration.enabled=true and start the server alone. Verify migrations, readiness and authenticated reads.

9. Restore the other replica counts only after the isolated checks pass. Follow Prove silence before opening any external destination.

The restored database has the database password that it had at the backup. If DATABASE_PASSWORD changed since then, put the old value back in the Secret before step 3. Rehearse a restore on a copy of the instance before you need one.

Recover the live installation

1. Close traffic and provider reception. Stop application roles and sandboxes, and pause reconciliation.
2. Keep a copy of the failed data and record the chosen recovery point and expected lost writes.
3. Check whether DATABASE_PASSWORD changed after the recovery point. If it changed, restore the matching value in the Secret before startup. Complete steps 1, 2 and 4 of the isolated drill with application replicas at zero. Keep provider and model egress denied. Wait for pg_is_in_recovery() to return f.
4. Do not apply restore-clone-lockdown.sql to the live recovery database. Verify silent mode, delivery status, review triggers and conversation bindings against the intended live settings.
5. Complete Choose the broker recovery strategy before consumers start. The default is to quarantine and recreate the instance's broker state, with recorded event loss. Snapshot replay needs a separate drill. A matching timestamp or recovery fence alone does not make it safe.
6. Enable migrations and start the server alone. Verify readiness, authenticated reads and encrypted values.
7. Before access or delivery resumes, reapply post-backup security and privacy decisions from the restricted incident record. Before restarting services gives the required suspension, session, consent and deletion checks.
8. Start the worker and require its isolation probe. Permit only the approved model endpoint for the controlled test while provider writes remain blocked. Test an approved practice review and conversation with delivery paused.
9. Complete Prove silence, Inspect and decide, and Turn off silent mode deliberately in the silent checklist. Restore the intended live control settings, provider reception and egress. Start the remaining roles and reopen traffic after the checks pass.
10. Verify every connected provider and feedback destination. Resume reconciliation only after its release and apply mode match the recovered deployment.

Upgrades

An upgrade to a new release is one helm upgrade, with the new release's chart, its profile, and your own values. First, verify the release chart of the new release. Then upgrade:

helm upgrade --install hephaestus "chart-$VERSION/hephaestus" -n hephaestus \
  -f "chart-$VERSION/hephaestus/profiles/small.yaml" -f site.yaml \
  --rollback-on-failure --wait=watcher --wait-for-jobs --timeout 20m

The migration Job of the new revision and the new Pods start together. The new Pods wait for the migrated schema, and the previous release serves until they are ready. --wait-for-jobs fails the upgrade when the migration fails, and --rollback-on-failure then returns the Deployments to the previous revision. The timeout is longer than the 10 minutes that a new Pod waits for the schema. The server stops before its new version starts (Recreate). The webhook role and the workers roll: new workers prove the cluster with the isolation probe before an old worker drains. An old worker drains for up to worker.drainTimeoutSeconds, 60 minutes by default, and Kubernetes no longer counts it while it terminates. Start the next upgrade only after the old worker is gone: kubectl --namespace hephaestus get pods -l app.kubernetes.io/component=worker shows no Terminating Pod. A review runs for up to 3 hours. When the drain ends first, the review stops: an attempt that was not yet admitted runs again on another worker, and an admitted one keeps its results. A drained attempt spends none of the review's retries. After 3 drains, the next drain cancels the review with the message "Workers stopped this review for updates too often. It is not tried again." The metric agent_job_drained_total{outcome} counts drained reviews that run again (requeued) and that are cancelled (cancelled). Read the release notes before the upgrade. If the release notes say Stop every role, use Stopped schema upgrades. For other database maintenance, use Database maintenance.

Roll forward to fix a failed upgrade. helm rollback and an older image do not undo a migration: restore the database from the backup taken before the upgrade (Restore).

Stopped schema upgrades

Use this procedure when the migration guide says Stop every role. It takes precedence over the normal upgrade and database-maintenance procedures. Do not use --rollback-on-failure: a rollback can restart old roles against a changed schema. Pause deployment automation that can change the release until this procedure is complete.

Before the first phase, record the current chart, profile, site values and all three replica counts. Back up PostgreSQL and JetStream, verify the restore, and keep the matching signed release lock. Verify the new release's chart before you use it. Each phase is a separate helm upgrade with that phase's chart, profile and site values:

Phase	Chart	migration.enabled	server.replicas	worker.replicas	webhook.replicas	Before the next phase
1. Stop every role	Current release	false	0	0	0	Every old role Pod, including terminating Pods, and its sandbox Pods and Jobs are gone.
2. Migrate	New release	true	0	0	0	The migration Job of this revision completed successfully.
3. Start the new roles	New release	true	Recorded count	Recorded count	Recorded count	Readiness, authenticated reads, a practice review and a conversation pass.

Keep PostgreSQL and NATS running throughout these phases. Use worker.replicas=0, not worker.enabled=false, to stop the worker. Apply each row's values after your profile and site values, so that they take precedence. Use --wait=watcher --wait-for-jobs --timeout 20m, without automatic rollback. A worker can drain for up to its configured timeout. If the first phase times out, keep all counts at zero and wait for every old Pod and sandbox to stop before phase 2.

In phase 2, identify the migration Job of the new Helm revision. Its default name is hephaestus-migration-<revision> for release hephaestus. Save its log with kubectl --namespace hephaestus logs job/<migration-job> -c migration > migration.log. Require its Complete condition: kubectl --namespace hephaestus wait --for=condition=complete job/<migration-job> --timeout=20m. A failed or timed-out Job does not permit phase 3. Keep all roles stopped, preserve the log and failed change set before the Job's 24-hour expiry, and follow failed-migration recovery.

Phase 3 creates another migration Job for the same release; it has no new schema changes to apply. Resume deployment automation only after its release and replica counts match the verified installation.

Database maintenance

Use this procedure when the release notes require it, before you change PostgreSQL itself, or before you change the CA in its TLS Secret. The server and the workers are the database roles. The webhook role has no database, so it keeps receiving deliveries into NATS JetStream through every phase. Each phase is one helm upgrade with the replica counts of that phase:

Phase	server.replicas	worker.replicas	Before the next phase
1. Stop the writers	0	0	The old pods of the two roles are gone. The worker's sandbox pods are gone.
2. Change the database	0	0	PostgreSQL is ready and accepts connections.
3. Migrate	1	0	The migration Job of the revision completed. The server is ready.
4. Restore	1	The prior count	The pods of the two roles are ready. The server consumes the deliveries that the receiver published meanwhile.

Phase 1 uses the verified chart and the values of the current release, with only the two counts changed. Phase 2 applies the change. The change can be the new release values, a new PostgreSQL configuration, or a new CA in the TLS Secret and a restart of PostgreSQL. Phase 3 runs the new release's migration Job and starts its server alone.

Before phase 1, record the current worker.replicas, and back up the database and the NATS JetStream data. A count of 0 is a desired state: check on the cluster that the pods stopped, that the sandbox Pods and Jobs are gone, and that the migrations completed before the next phase. Set worker.replicas=0 to stop the workers; worker.enabled=false removes the worker objects. A change of NATS itself stops reception: while NATS is down, the receiver answers deliveries with 503, and each provider decides whether it retries.

Uninstall

1. Delete the sandbox namespace first. This stops every sandbox at once. If you remove the release first, its network policies go, and the conversation hosts and turn sandboxes that are left run without them.

   kubectl delete namespace hephaestus-sandboxes

2. Remove the release. Running reviews stop.

   helm uninstall hephaestus --namespace hephaestus

3. These items stay:

   • The PostgreSQL and NATS volumes.
   • The Secrets that you created.
   • The backups in the S3 bucket.

4. To remove the data too, delete the release namespace: kubectl delete namespace hephaestus. A PersistentVolume with the Retain reclaim policy stays until you delete it.

Troubleshooting

Symptom	Cause and check
A worker stays unready, and its log says that a sandbox namespace informer failed and retries	The worker cannot watch the Pods, Jobs or ResourceQuotas of the sandbox namespace. The log gives the HTTP status. A 403 means that the worker's Role lacks list or watch on one of them (Sandbox prerequisites). The worker retries and becomes ready by itself once it can watch.
A worker stays unready, and its log says that the cluster does not hold the sandbox boundary	The isolation probe failed. The log names each failed check. A failed egress check usually means that the network plugin does not enforce NetworkPolicy, or that networkPolicy.kubernetesApi names an address that the plugin does not see.
A worker log says that the network policy did not filter a sandbox Pod in time	The network plugin applied the policy of a new Pod more than 60 seconds after the Pod started. The attempt is tried again.
A review fails with "The sandbox quota is smaller than one review sandbox"	A quota of your own holds less than one review sandbox requests. Raise it. The chart's quotas always hold one.
A sandbox Pod stays Pending with a quota error	Its tier's quota is full. The chart sizes each tier for its workers' capacity, so check a quota of your own, or worker.reviews and worker.conversationTurns.
Reviews stay queued, and the worker log says that reviews wait for room in the sandbox quota	The reviews tier holds the running reviews and the isolation probe. The reviews start when a sandbox ends. To run more at once, raise worker.reviews.
A new Pod restarts during an upgrade, and its log names a pending change set	It waited 10 minutes for the schema. Check the migration Job's log. A longer migration needs the database maintenance procedure.
A backup Job fails with unable to find primary cluster	The Job cannot reach pgBackRest's TLS server on port 8432 of the PostgreSQL Service, or its client certificate is not for hephaestus-backup from ca.crt.
PostgreSQL's volume fills	A failed WAL archive keeps the WAL on the volume. Check the latest backup Job's log and the bucket's credentials.
A practice review fails with "The sandbox could not prepare its workspace."	The sandbox's initializer failed before any model call. The job runs again within its retry limit, so a failure that repeats on every attempt uses all of them. The worker log names the initializer's exit code. After the last attempt, read the exit code and the last lines of the initializer's log on the job: kubectl --namespace hephaestus exec hephaestus-postgresql-0 -c postgresql -- psql -U "$DATABASE_USERNAME" -d hephaestus -c "SELECT capture->'initializationFailure' FROM review_job WHERE id='<job id>'". Known credential formats in those lines are masked.

kubectl --namespace <release namespace> logs deployment/<release>-worker shows a worker's probe result and its isolation tier. The worker's /actuator/prometheus on its management port shows the same tier in the sandbox_isolation_* metrics.

Known limits

• NATS JetStream runs as one server with one replica of each stream. A lost NATS volume loses the provider deliveries that the server had not processed (Storage and retention).
• PostgreSQL runs as one instance. A lost node stops the release until PostgreSQL runs again; a lost volume needs a restore.
• The server runs as one replica until the server processes provider events in order across replicas.
• The worker records the isolation tier in its log and metrics, not with each review. A record for each review needs a schema change.
• Turn sandboxes take their commands through pods/exec until they pull them through the gateway, as the conversation host does.
• The kubelet enforces sandbox disk limits by eviction after it measures the use, not when a process writes. A sandbox can pass its limit until the next measurement, for some seconds. Only file system project quotas (LocalStorageCapacityIsolationFSQuotaMonitoring) refuse the write itself.
• Move a Compose install at the same release version. Back up PostgreSQL, JetStream and the encryption keys before the move.

What CI does not show

A successful install in CI does not show that your cluster isolates sandboxes. The worker's isolation probe checks that on your cluster, at each start and on its schedule. CI does not qualify your Gateway controller: check its four HTTPRoute features, its body limit and its idle timeout yourself, and rehearse a restore on your own cluster.