# Install on a Kubernetes cluster

This procedure installs the signed Helm chart of a release on a Kubernetes cluster on which you are a cluster-admin. For one Linux host, [Install on one host](/next/admin/k3s.md) does all of these steps with one command. For a cluster without cluster-admin rights, or a cluster of several nodes, use [Install on a managed cluster without cluster-admin](/next/admin/managed-cluster.md).

The commands use the release `hephaestus` in the namespace `hephaestus`, and the sandbox namespace `hephaestus-sandboxes`.

## Before you start

* Helm 4. The install command uses `--wait=watcher`, which Helm 3 does not have.
* [Cosign](https://docs.sigstore.dev/cosign/system_config/installation/) 3.1 or later.
* `kubectl`, `curl`, `jq` and `openssl`.
* A cluster that meets the [supported cluster](/next/admin/helm-chart.md#supported-clusters) requirements.
* A host name, with a DNS record that points to your Gateway. Serve Hephaestus on HTTPS port 443 of that host name. The webapp calls the API at `https://<host>/api`, without a port, and the chart has no setting for another port.
* A GitHub OAuth app, its client ID and its client secret, as [First steps § 1](/next/admin/first-steps.md#1-configure-a-login-provider) describes.
* The version of the newest release, from the [releases page](https://github.com/hephaestus-build/Hephaestus/releases), without the leading `v`.

Choose a [profile](/next/admin/helm-chart.md#profiles):

* `small.yaml` runs practice reviews on one node of 4 vCPU, 16 GiB of memory, and 150 GiB of disk. The sandbox nodes must meet the [sandbox prerequisites](/next/admin/sandboxes.md#sandbox-prerequisites). After the install, you run the [sandbox isolation gate](/next/admin/sandboxes.md#sandbox-isolation-gate), which has its own prerequisites.
* `dev.yaml` is for a trial on a workstation cluster, such as k3d or kind. It runs no practice reviews. Skip each step below that is for `small.yaml` only.

## 1. Verify the release chart

Run these commands in the directory of your values file. If one command fails, stop: do not install the chart.

```
VERSION=X.Y.Z

CHART=ghcr.io/hephaestus-build/charts/hephaestus

# 1. The chart digest, from the joint lock of the release.

DIGEST=$(curl -fsSL "https://github.com/hephaestus-build/Hephaestus/releases/download/v$VERSION/release-v$VERSION.chart.json" |

  jq -er .chart.digest)

# 2. The release workflow signed that digest.

cosign verify "$CHART@$DIGEST" \

  --certificate-identity 'https://github.com/hephaestus-build/Hephaestus/.github/workflows/release.yml@refs/heads/main' \

  --certificate-oidc-issuer https://token.actions.githubusercontent.com

# 3. The one pull: Helm gets the chart by that digest, never by a tag.

helm pull "oci://$CHART@$DIGEST" --untar --untardir "chart-$VERSION"

# 4. The signed chart is the chart of this release.

helm show chart "chart-$VERSION/hephaestus" | grep -x "version: $VERSION" >/dev/null
```

Step 1 only looks up the digest, so the joint lock needs no check of its own. Step 2 refuses every digest that the release workflow did not sign. Step 4 binds the signed chart to the release that you asked for. A version tag can move, so Helm never pulls the chart by a tag. Step 3 stops if `chart-$VERSION/hephaestus` exists.

`chart-$VERSION/hephaestus` is the verified chart, with its `profiles/`. The chart names the images of the release by digest, so you give Helm only your own values. [Release assets](/next/admin/release-image-lock.md#release-assets) gives the purpose of each signed asset of a release.

Then get the k3s bundle of the same release. Download, verify and unpack it, as steps 2 and 3 of [Get the release files](/next/admin/k3s.md#1-get-the-release-files) describe. You need two files from it: `hephaestus-keys.sh` and, for `small.yaml`, `k3s-agent-sandbox.yaml`.

## 2. Create the namespaces

1. Create the release namespace and the sandbox namespace. Label both with the Restricted Pod Security level. Use the sandbox namespace for this release only.

   ```
   kubectl create namespace hephaestus

   kubectl create namespace hephaestus-sandboxes

   kubectl label namespace hephaestus hephaestus-sandboxes pod-security.kubernetes.io/enforce=restricted
   ```

2. For `small.yaml`, install the agent-sandbox controller of the release ([Sandbox templates](/next/admin/sandboxes.md#sandbox-templates)). Apply it again for each release.

   ```
   kubectl apply --server-side -f k3s-agent-sandbox.yaml

   kubectl -n agent-sandbox-system rollout status deployment/agent-sandbox-controller
   ```

## 3. Create the Secret

The chart reads secret values from one Secret in the release namespace, with fixed key names. It never writes a secret value. A minimal install needs these keys:

* `HEPHAESTUS_SECURITY_ENCRYPTION_KEY` and `HEPHAESTUS_SECURITY_CREDENTIAL_ENCRYPTION_KEY`.
* `HEPHAESTUS_AUTH_STATE_COOKIE_KEY`.
* `WEBHOOK_SECRET` and `WEBHOOK_ROUTING_SECRET`.
* `DATABASE_OWNER_PASSWORD` and `DATABASE_APP_PASSWORD`, the passwords of the [database roles](/next/admin/helm-chart.md#database-roles).
* With the bundled PostgreSQL, also `DATABASE_ADMIN_USERNAME` and `DATABASE_ADMIN_PASSWORD`, its administrator.
* `GITHUB_OAUTH_CLIENT_SECRET`, for sign-in with GitHub.

If a key is missing, the Pod that reads it stays in `CreateContainerConfigError`. [Secret custody and rotation](/next/admin/secret-custody.md#inventory) lists each key, the keys of features that you turn on later, and how to replace each key.

The release script `hephaestus-keys.sh` creates the Secret. It generates each key that you do not supply, and it never changes a key that exists. It prints key names, never a value.

1. Type the client secret of your GitHub OAuth app. The shell does not show it:

   ```
   read -rs GITHUB_OAUTH_CLIENT_SECRET && export GITHUB_OAUTH_CLIENT_SECRET
   ```

2. Run the script:

   ```
   sh hephaestus-keys.sh --namespace hephaestus --secret hephaestus-app
   ```

   Add these options when they apply:

   * `--external-database`: you use your own PostgreSQL. The script takes `DATABASE_OWNER_PASSWORD` and `DATABASE_APP_PASSWORD` from the environment. Use the passwords that you gave the [database roles](/next/admin/external-database.md#set-up-the-database-roles).
   * `--new-keys`: with `--external-database`, for a database without Hephaestus data only. The script cannot see whether the database holds data. For a database with data, restore both encryption keys from your backup first.
   * `--backups`: you turn on backups with a new, empty backup repository. The script generates `BACKUP_CIPHER_PASS`. For an existing repository, restore its passphrase into the Secret instead.
   * `--metrics`: you turn on the [database metrics](/next/admin/monitoring.md#turn-on-database-metrics). The script generates `DATABASE_MONITOR_PASSWORD`.

3. Copy the keys off the cluster, as [Copy the keys from the Secret](/next/admin/secret-custody.md#copy-the-keys-from-the-secret) describes. Without the encryption keys, a restore cannot read the data that they protect.

Run the script again at any time, for example with `--backups` when you turn on backups. It adds only the keys that the Secret lacks. The script stops if the bundled PostgreSQL volume exists but the Secret lacks the administrator login or an encryption key. Then restore the Secret from your backup.

If Git holds your release, for example with Argo CD or Flux, run the script once from your workstation before the first sync. Keep the Secret out of Git. To write the Secret yourself instead, generate each value as [Generation and rotation](/next/admin/secret-custody.md#generation-and-rotation) gives it.

## 4. Create the PostgreSQL TLS Secret

The server and the migration Job connect to PostgreSQL with `sslmode=verify-full`. They check the server certificate against a CA, and they check that the certificate names the host. The chart creates no certificate, no key, and no CA, and it refuses to render without this Secret.

| Key                  | Bundled PostgreSQL | External PostgreSQL |
| -------------------- | ------------------ | ------------------- |
| `ca.crt`             | Yes                | Yes                 |
| `tls.crt`, `tls.key` | Yes                | No                  |

For the bundled PostgreSQL, `tls.crt` must name `hephaestus-postgresql.hephaestus.svc`, without a cluster domain. If you have no CA, make one with OpenSSL. This example makes a CA that is valid for ten years, and a server certificate that is valid for one year:

```
RELEASE=hephaestus NAMESPACE=hephaestus

HOST="$RELEASE-postgresql.$NAMESPACE.svc"

TLS_DIR=$(mktemp -d)

openssl req -x509 -newkey rsa:3072 -nodes -sha256 -days 3650 -subj '/CN=Hephaestus PostgreSQL CA' \

  -addext 'basicConstraints=critical,CA:TRUE' -addext 'keyUsage=critical,keyCertSign,cRLSign' \

  -keyout "$TLS_DIR/ca.key" -out "$TLS_DIR/ca.crt"

openssl req -newkey rsa:3072 -nodes -sha256 -subj "/CN=$HOST" \

  -keyout "$TLS_DIR/tls.key" -out "$TLS_DIR/server.csr"

openssl x509 -req -sha256 -days 365 -in "$TLS_DIR/server.csr" \

  -CA "$TLS_DIR/ca.crt" -CAkey "$TLS_DIR/ca.key" -CAcreateserial \

  -extfile <(printf 'subjectAltName=DNS:%s\nextendedKeyUsage=serverAuth\nkeyUsage=digitalSignature,keyEncipherment\nbasicConstraints=CA:FALSE\n' "$HOST") \

  -out "$TLS_DIR/tls.crt"

openssl verify -CAfile "$TLS_DIR/ca.crt" -verify_hostname "$HOST" -purpose sslserver "$TLS_DIR/tls.crt"
```

The last command prints `OK` when the certificate names the host. Then create the Secret:

```
kubectl --namespace "$NAMESPACE" create secret generic hephaestus-postgresql-tls \

  --from-file=ca.crt="$TLS_DIR/ca.crt" \

  --from-file=tls.crt="$TLS_DIR/tls.crt" \

  --from-file=tls.key="$TLS_DIR/tls.key"
```

Keep `ca.key` outside the cluster: you need it to renew the server certificate. For an external PostgreSQL, the Secret holds only `ca.crt`. [TLS certificates](/next/admin/secret-custody.md#tls-certificates) gives the renewal of each certificate.

## 5. Prepare the Gateway

The chart attaches one `HTTPRoute` to the Gateways in `route.parentRefs`. It installs no Gateway, no Gateway controller, and no certificate. Before the install, the cluster needs these items:

* The Gateway API resource definitions, and a Gateway controller with the two `HTTPRoute` features in [Gateway routes](/next/admin/helm-chart.md#gateway-routes).
* A Gateway that serves HTTPS on port 443 of your host name, with a TLS certificate for it.
* A redirect from HTTP to HTTPS.
* A listener that accepts routes from the release namespace. The Gateway API calls this setting `allowedRoutes.namespaces`. The Traefik chart calls it `namespacePolicy`. Without it, the Gateway rejects the route, and every request returns `404`.
* An idle timeout of at least 60 seconds for the Hephaestus live stream.

Also set a request body limit of 25 MiB at the Gateway, if your controller has one. The server refuses a webhook body over 25 MiB before it reads the body, but it has no such limit on `/api`. The route of the chart adds no filter of a specific controller, so set the limit on the Gateway or its controller.

For example, on a k3s cluster that you set up yourself, the bundled Traefik serves the Gateway API when you turn on its Gateway provider. Put this `HelmChartConfig` in `/var/lib/rancher/k3s/server/manifests/`, or apply it with `kubectl apply -f`:

```
apiVersion: helm.cattle.io/v1

kind: HelmChartConfig

metadata:

  name: traefik

  namespace: kube-system

spec:

  valuesContent: |-

    providers:

      kubernetesGateway:

        enabled: true

    gateway:

      enabled: true

      listeners:

        web:

          port: 8000

          protocol: HTTP

          # Only the redirect below, which is in this namespace.

          namespacePolicy:

            from: Same

        websecure:

          port: 8443

          protocol: HTTPS

          certificateRefs:

            - name: hephaestus-tls

          namespacePolicy:

            from: Selector

            selector:

              matchLabels:

                kubernetes.io/metadata.name: hephaestus

    extraObjects:

      - apiVersion: gateway.networking.k8s.io/v1

        kind: HTTPRoute

        metadata:

          name: https-redirect

          namespace: kube-system

        spec:

          parentRefs:

            - name: traefik-gateway

              sectionName: web

          rules:

            - filters:

                - type: RequestRedirect

                  requestRedirect:

                    scheme: https

                    statusCode: 301
```

This Traefik creates the Gateway `kube-system/traefik-gateway`. It creates the HTTPS listener only when the TLS Secret `hephaestus-tls` exists in `kube-system`. Its default idle timeout of 180 seconds meets the requirement above. It sets no request body limit.

Optional: if you cannot get a certificate from a public CA, make a test certificate. Sign it with your own CA instead if you have one. A browser shows a warning for a self-signed certificate, and GitHub does not deliver webhooks to it:

```
HOST=hephaestus.example.com

openssl req -x509 -newkey rsa:3072 -nodes -sha256 -days 365 -subj "/CN=$HOST" \

  -addext "subjectAltName=DNS:$HOST" -keyout gateway.key -out gateway.crt

kubectl --namespace kube-system create secret tls hephaestus-tls --cert=gateway.crt --key=gateway.key
```

The server trusts forwarded client addresses only from the Gateway Pods. The `IP` column of this command shows their addresses:

```
kubectl get pods --namespace kube-system --selector app.kubernetes.io/name=traefik --output wide
```

Change the namespace and the label for another Gateway controller. A Pod gets a new address when it restarts, so `trustedProxies` matches the Pod network, not one address.

## 6. Write the values file

Write your values file, `site.yaml`. This example is for `small.yaml` and the k3s Traefik above:

site.yaml

```
host: hephaestus.example.com

# A Tomcat regular expression for the Pod network of the Gateway Pods: k3s uses 10.42.0.0/16.

trustedProxies: '10\.42\.\d{1,3}\.\d{1,3}'

secret:

  name: hephaestus-app

postgresql:

  tls:

    existingSecret: hephaestus-postgresql-tls

route:

  parentRefs:

    - name: traefik-gateway

      namespace: kube-system

      sectionName: websecure

networkPolicy:

  # The Gateway Pods that may send requests to the release.

  gateway:

    namespace: kube-system

    podLabels:

      app.kubernetes.io/name: traefik

      app.kubernetes.io/instance: traefik-kube-system

  # The Kubernetes API endpoint, for small.yaml only.

  kubernetesApi:

    cidr: 192.168.5.15/32

    port: 6443

# For small.yaml only.

sandbox:

  namespace: hephaestus-sandboxes

config:

  github:

    oauthClientId: Ov23liExampleClientId

  auth:

    bootstrapAdmins: ["github:1234567"]
```

1. Set `host` to your host name.

2. Set `trustedProxies` to a regular expression for the Pod network of your Gateway Pods. Use single quotes, or no quotes. In double quotes, YAML reads each backslash as the start of an escape sequence.

3. Set `route.parentRefs` to your Gateway and its HTTPS listener.

4. Set `networkPolicy.gateway` to the namespace and the Pod labels of your Gateway Pods. The chart refuses a namespace without labels, because it would admit every Pod in that namespace.

5. For `small.yaml`, find the Kubernetes API endpoint that the network plugin sees:

   ```
   kubectl get endpointslices --namespace default --selector kubernetes.io/service-name=kubernetes
   ```

   Set `networkPolicy.kubernetesApi.cidr` to the address in `ENDPOINTS`, with `/32`, or `/128` for IPv6. Set `networkPolicy.kubernetesApi.port` to the value in `PORTS`. If `ENDPOINTS` lists one address for each control plane node, set a CIDR that holds all of them.

6. Set `config.github.oauthClientId` and `config.auth.bootstrapAdmins`, as [First steps § 1 and § 2](/next/admin/first-steps.md) describe.

7. For a public instance, name your legal pages ConfigMap in `webapp.legalOverrides`, as [Legal pages](/next/admin/legal-pages.md#deployment-specific-overrides) describes. The webapp Pod waits until the ConfigMap that it names exists.

Do not set `release` or `images`: the release chart sets them. For `dev.yaml`, leave out `networkPolicy.kubernetesApi` and `sandbox`.

## 7. Install the chart

1. Render the chart with your profile and your values. This step needs no cluster:

   ```
   helm template hephaestus "chart-$VERSION/hephaestus" -n hephaestus \

     -f "chart-$VERSION/hephaestus/profiles/small.yaml" -f site.yaml >/dev/null
   ```

   If Helm prints an error, correct the value that it names, and render again.

2. Install the chart with the same files:

   ```
   helm upgrade --install hephaestus "chart-$VERSION/hephaestus" -n hephaestus \

     -f "chart-$VERSION/hephaestus/profiles/small.yaml" -f site.yaml \

     --wait=watcher --wait-for-jobs --timeout 45m
   ```

The first install has no automatic rollback, so a failed install keeps its Jobs and Pods for diagnosis. If it fails, use [A first install fails](#a-first-install-fails) before you remove anything.

## Check that it worked

1. Find the revision of the install with `helm history hephaestus --namespace hephaestus`.

2. Read the log of its migration Job: `kubectl --namespace hephaestus logs job/hephaestus-migration-<revision>`. The Job logs `Database migration completed`. The JVM then prints a long native memory summary. It is not an error.

3. Wait until the server is ready:

   ```
   kubectl --namespace hephaestus rollout status deployment/hephaestus-server
   ```

   On a first install, the server can restart while PostgreSQL starts. This is not a failure.

4. Check that the Gateway accepted the route:

   ```
   kubectl --namespace hephaestus get httproute -o jsonpath='{range .items[*].status.parents[*].conditions[*]}{.type}={.status} {.reason}{"\n"}{end}'
   ```

   The output shows `Accepted=True` and `ResolvedRefs=True`. `Accepted=False NotAllowedByListeners` means that the listener does not accept routes from the release namespace.

5. Open `https://<host>`. The login page shows **Continue with GitHub**.

6. For `small.yaml`, run the [sandbox isolation gate](/next/admin/sandboxes.md#sandbox-isolation-gate) for each sandbox template. Connect no repository until the gate passes for each template.

Then continue with [First steps § 3. Sign in](/next/admin/first-steps.md#3-sign-in).

## Undo

1. Delete the sandbox namespace first. This stops every sandbox at once. If you remove the release first, its network policies go, and the remaining sandbox Pods lose their policies.

   ```
   kubectl delete namespace hephaestus-sandboxes
   ```

2. Remove the release. Running reviews stop.

   ```
   helm uninstall hephaestus --namespace hephaestus
   ```

3. These items stay:

   * The PostgreSQL volumes.
   * The Secrets that you created.
   * The backups in the S3 bucket.

4. To remove the data too, delete the release namespace: `kubectl delete namespace hephaestus`. A PersistentVolume with the `Retain` reclaim policy stays until you delete it.

## Troubleshooting

### A first install fails

A failed first install keeps its Helm revision, Jobs and Pods when it runs without `--rollback-on-failure`. On one host, first select the kubeconfig of the installer, as [A first install fails on one host](/next/admin/k3s.md#a-first-install-fails) describes.

1. Preserve the terminal output of the install before you try again.
2. Check the Helm record with `helm list --namespace hephaestus`.
3. If the release remains, read its history with `helm history hephaestus --namespace hephaestus`.
4. List remaining Jobs and Pods with `kubectl --namespace hephaestus get jobs,pods -o wide`.
5. Read recent events with `kubectl --namespace hephaestus get events --sort-by=.metadata.creationTimestamp`.
6. If the migration Job remains, preserve its log with `kubectl --namespace hephaestus logs job/<migration-job>`. Replace `<migration-job>` with the Job name for the failed Helm revision from steps 3 and 4.
7. If a server Pod remains, read its log before the next install replaces it.

Events expire, and the log of a removed Pod can be gone. A failed install can still have changed the database. Use [Failed-migration recovery](/next/admin/upgrade.md#if-migration-fails) before you start an older release against it. For durable logs, configure a [log collector](/next/admin/monitoring.md#ship-logs-to-a-log-store) before the install.

### Other symptoms

| Symptom                                                                                            | Cause and check                                                                                                                                                                                                                                                                                                                                                                                 |
| -------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| A server log says that the network policy did not filter a sandbox Pod in time                     | The network plugin applied the policy of a new Pod more than 60 seconds after the Pod started. The attempt is tried again.                                                                                                                                                                                                                                                                      |
| A sandbox Pod stays `Pending` with a quota error                                                   | Its tier's quota is full. The chart sizes each tier for its servers' capacity, so check a quota of your own, or `sandbox.reviewMemory` and `sandbox.conversationTurns`.                                                                                                                                                                                                                         |
| Every path returns `404`, and `kubectl get httproute` shows `Accepted=False NotAllowedByListeners` | The listener of the Gateway does not accept routes from the release namespace. Set `allowedRoutes.namespaces` on the listener, as [Prepare the Gateway](/next/admin/kubernetes.md#prepare-the-gateway) describes.                                                                                                                                                                               |
| A new Pod restarts during an upgrade, and its log names a pending change set                       | It waited 10 minutes for the schema. Check the migration Job's log. A longer migration needs the [database maintenance](/next/admin/maintenance.md#database-maintenance) procedure.                                                                                                                                                                                                             |
| A backup Job fails with `unable to find primary cluster`                                           | The Job cannot reach pgBackRest's TLS server on port 8432 of the PostgreSQL Service, or its client certificate is not for `hephaestus-backup` from `ca.crt`.                                                                                                                                                                                                                                    |
| PostgreSQL's volume fills                                                                          | A failed WAL archive keeps the WAL on the volume. Check the latest backup Job's log and the bucket's credentials.                                                                                                                                                                                                                                                                               |
| A practice review fails with "The sandbox could not prepare its workspace."                        | The sandbox's initializer failed before any model call. The job runs again within its retry limit, so a failure that repeats on every attempt uses all of them. The server log names the initializer's exit code. After the last attempt, read the exit code and the last lines of the initializer's log with the command below this table. Known credential formats in those lines are masked. |

Replace `<job id>` with the ID of the review job:

```
kubectl --namespace hephaestus exec -i hephaestus-postgresql-0 -c postgresql -- \

  sh -c 'exec psql -U "$POSTGRES_USER" -d "$POSTGRES_DB"' <<'SQL'

SELECT capture->'initializationFailure' FROM review_job WHERE id = '<job id>';

SQL
```