Skip to main content

Install on a Kubernetes cluster

This procedure installs the signed Helm chart of a release on a Kubernetes cluster on which you are a cluster-admin. For one Linux host, Install on one host does all of these steps with one command. For a cluster without cluster-admin rights, or a cluster of several nodes, use Install on a managed cluster without cluster-admin.

The commands use the release hephaestus in the namespace hephaestus, and the sandbox namespace hephaestus-sandboxes.

Before you start​

  • Helm 4. The install command uses --wait=watcher, which Helm 3 does not have.
  • Cosign 3.1 or later.
  • kubectl, curl, jq and openssl.
  • A cluster that meets the supported cluster requirements.
  • A host name, with a DNS record that points to your Gateway. Serve Hephaestus on HTTPS port 443 of that host name. The webapp calls the API at https://<host>/api, without a port, and the chart has no setting for another port.
  • A GitHub OAuth app, its client ID and its client secret, as First steps § 1 describes.
  • The version of the newest release, from the releases page, without the leading v.

Choose a profile:

  • small.yaml runs practice reviews on one node of 4 vCPU, 16 GiB of memory, and 150 GiB of disk. The sandbox nodes must meet the sandbox prerequisites. After the install, you run the sandbox isolation gate, which has its own prerequisites.
  • dev.yaml is for a trial on a workstation cluster, such as k3d or kind. It runs no practice reviews. Skip each step below that is for small.yaml only.

1. Verify the release chart​

Run these commands in the directory of your values file. If one command fails, stop: do not install the chart.

VERSION=X.Y.Z
CHART=ghcr.io/hephaestus-build/charts/hephaestus
# 1. The chart digest, from the joint lock of the release.
DIGEST=$(curl -fsSL "https://github.com/hephaestus-build/Hephaestus/releases/download/v$VERSION/release-v$VERSION.chart.json" |
  jq -er .chart.digest)
# 2. The release workflow signed that digest.
cosign verify "$CHART@$DIGEST" \
  --certificate-identity 'https://github.com/hephaestus-build/Hephaestus/.github/workflows/release.yml@refs/heads/main' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com
# 3. The one pull: Helm gets the chart by that digest, never by a tag.
helm pull "oci://$CHART@$DIGEST" --untar --untardir "chart-$VERSION"
# 4. The signed chart is the chart of this release.
helm show chart "chart-$VERSION/hephaestus" | grep -x "version: $VERSION" >/dev/null

Step 1 only looks up the digest, so the joint lock needs no check of its own. Step 2 refuses every digest that the release workflow did not sign. Step 4 binds the signed chart to the release that you asked for. A version tag can move, so Helm never pulls the chart by a tag. Step 3 stops if chart-$VERSION/hephaestus exists.

chart-$VERSION/hephaestus is the verified chart, with its profiles/. The chart names the images of the release by digest, so you give Helm only your own values. Release assets gives the purpose of each signed asset of a release.

Then get the k3s bundle of the same release. Download, verify and unpack it, as steps 2 and 3 of Get the release files describe. You need two files from it: hephaestus-keys.sh and, for small.yaml, k3s-agent-sandbox.yaml.

2. Create the namespaces​

  1. Create the release namespace and the sandbox namespace. Label both with the Restricted Pod Security level. Use the sandbox namespace for this release only.

    kubectl create namespace hephaestus
    kubectl create namespace hephaestus-sandboxes
    kubectl label namespace hephaestus hephaestus-sandboxes pod-security.kubernetes.io/enforce=restricted

  2. For small.yaml, install the agent-sandbox controller of the release (Sandbox templates). Apply it again for each release.

    kubectl apply --server-side -f k3s-agent-sandbox.yaml
    kubectl -n agent-sandbox-system rollout status deployment/agent-sandbox-controller

3. Create the Secret​

The chart reads secret values from one Secret in the release namespace, with fixed key names. It never writes a secret value. A minimal install needs these keys:

  • HEPHAESTUS_SECURITY_ENCRYPTION_KEY and HEPHAESTUS_SECURITY_CREDENTIAL_ENCRYPTION_KEY.
  • HEPHAESTUS_AUTH_STATE_COOKIE_KEY.
  • WEBHOOK_SECRET and WEBHOOK_ROUTING_SECRET.
  • DATABASE_OWNER_PASSWORD and DATABASE_APP_PASSWORD, the passwords of the database roles.
  • With the bundled PostgreSQL, also DATABASE_ADMIN_USERNAME and DATABASE_ADMIN_PASSWORD, its administrator.
  • GITHUB_OAUTH_CLIENT_SECRET, for sign-in with GitHub.

If a key is missing, the Pod that reads it stays in CreateContainerConfigError. Secret custody and rotation lists each key, the keys of features that you turn on later, and how to replace each key.

The release script hephaestus-keys.sh creates the Secret. It generates each key that you do not supply, and it never changes a key that exists. It prints key names, never a value.

  1. Type the client secret of your GitHub OAuth app. The shell does not show it:

    read -rs GITHUB_OAUTH_CLIENT_SECRET && export GITHUB_OAUTH_CLIENT_SECRET

  2. Run the script:

    sh hephaestus-keys.sh --namespace hephaestus --secret hephaestus-app

    Add these options when they apply:

    • --external-database: you use your own PostgreSQL. The script takes DATABASE_OWNER_PASSWORD and DATABASE_APP_PASSWORD from the environment. Use the passwords that you gave the database roles.
    • --new-keys: with --external-database, for a database without Hephaestus data only. The script cannot see whether the database holds data. For a database with data, restore both encryption keys from your backup first.
    • --backups: you turn on backups with a new, empty backup repository. The script generates BACKUP_CIPHER_PASS. For an existing repository, restore its passphrase into the Secret instead.
    • --metrics: you turn on the database metrics. The script generates DATABASE_MONITOR_PASSWORD.

  3. Copy the keys off the cluster, as Copy the keys from the Secret describes. Without the encryption keys, a restore cannot read the data that they protect.

Run the script again at any time, for example with --backups when you turn on backups. It adds only the keys that the Secret lacks. The script stops if the bundled PostgreSQL volume exists but the Secret lacks the administrator login or an encryption key. Then restore the Secret from your backup.

If Git holds your release, for example with Argo CD or Flux, run the script once from your workstation before the first sync. Keep the Secret out of Git. To write the Secret yourself instead, generate each value as Generation and rotation gives it.

4. Create the PostgreSQL TLS Secret​

The server and the migration Job connect to PostgreSQL with sslmode=verify-full. They check the server certificate against a CA, and they check that the certificate names the host. The chart creates no certificate, no key, and no CA, and it refuses to render without this Secret.

KeyBundled PostgreSQLExternal PostgreSQL
ca.crtYesYes
tls.crt, tls.keyYesNo

For the bundled PostgreSQL, tls.crt must name hephaestus-postgresql.hephaestus.svc, without a cluster domain. If you have no CA, make one with OpenSSL. This example makes a CA that is valid for ten years, and a server certificate that is valid for one year:

RELEASE=hephaestus NAMESPACE=hephaestus
HOST="$RELEASE-postgresql.$NAMESPACE.svc"
TLS_DIR=$(mktemp -d)
openssl req -x509 -newkey rsa:3072 -nodes -sha256 -days 3650 -subj '/CN=Hephaestus PostgreSQL CA' \
  -addext 'basicConstraints=critical,CA:TRUE' -addext 'keyUsage=critical,keyCertSign,cRLSign' \
  -keyout "$TLS_DIR/ca.key" -out "$TLS_DIR/ca.crt"
openssl req -newkey rsa:3072 -nodes -sha256 -subj "/CN=$HOST" \
  -keyout "$TLS_DIR/tls.key" -out "$TLS_DIR/server.csr"
openssl x509 -req -sha256 -days 365 -in "$TLS_DIR/server.csr" \
  -CA "$TLS_DIR/ca.crt" -CAkey "$TLS_DIR/ca.key" -CAcreateserial \
  -extfile <(printf 'subjectAltName=DNS:%s\nextendedKeyUsage=serverAuth\nkeyUsage=digitalSignature,keyEncipherment\nbasicConstraints=CA:FALSE\n' "$HOST") \
  -out "$TLS_DIR/tls.crt"
openssl verify -CAfile "$TLS_DIR/ca.crt" -verify_hostname "$HOST" -purpose sslserver "$TLS_DIR/tls.crt"

The last command prints OK when the certificate names the host. Then create the Secret:

kubectl --namespace "$NAMESPACE" create secret generic hephaestus-postgresql-tls \
  --from-file=ca.crt="$TLS_DIR/ca.crt" \
  --from-file=tls.crt="$TLS_DIR/tls.crt" \
  --from-file=tls.key="$TLS_DIR/tls.key"

Keep ca.key outside the cluster: you need it to renew the server certificate. For an external PostgreSQL, the Secret holds only ca.crt. TLS certificates gives the renewal of each certificate.

5. Prepare the Gateway​

The chart attaches one HTTPRoute to the Gateways in route.parentRefs. It installs no Gateway, no Gateway controller, and no certificate. Before the install, the cluster needs these items:

  • The Gateway API resource definitions, and a Gateway controller with the two HTTPRoute features in Gateway routes.
  • A Gateway that serves HTTPS on port 443 of your host name, with a TLS certificate for it.
  • A redirect from HTTP to HTTPS.
  • A listener that accepts routes from the release namespace. The Gateway API calls this setting allowedRoutes.namespaces. The Traefik chart calls it namespacePolicy. Without it, the Gateway rejects the route, and every request returns 404.
  • An idle timeout of at least 60 seconds for the Hephaestus live stream.

Also set a request body limit of 25 MiB at the Gateway, if your controller has one. The server refuses a webhook body over 25 MiB before it reads the body, but it has no such limit on /api. The route of the chart adds no filter of a specific controller, so set the limit on the Gateway or its controller.

For example, on a k3s cluster that you set up yourself, the bundled Traefik serves the Gateway API when you turn on its Gateway provider. Put this HelmChartConfig in /var/lib/rancher/k3s/server/manifests/, or apply it with kubectl apply -f:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
    providers:
      kubernetesGateway:
        enabled: true
    gateway:
      enabled: true
      listeners:
        web:
          port: 8000
          protocol: HTTP
          # Only the redirect below, which is in this namespace.
          namespacePolicy:
            from: Same
        websecure:
          port: 8443
          protocol: HTTPS
          certificateRefs:
            - name: hephaestus-tls
          namespacePolicy:
            from: Selector
            selector:
              matchLabels:
                kubernetes.io/metadata.name: hephaestus
    extraObjects:
      - apiVersion: gateway.networking.k8s.io/v1
        kind: HTTPRoute
        metadata:
          name: https-redirect
          namespace: kube-system
        spec:
          parentRefs:
            - name: traefik-gateway
              sectionName: web
          rules:
            - filters:
                - type: RequestRedirect
                  requestRedirect:
                    scheme: https
                    statusCode: 301

This Traefik creates the Gateway kube-system/traefik-gateway. It creates the HTTPS listener only when the TLS Secret hephaestus-tls exists in kube-system. Its default idle timeout of 180 seconds meets the requirement above. It sets no request body limit.

Optional: if you cannot get a certificate from a public CA, make a test certificate. Sign it with your own CA instead if you have one. A browser shows a warning for a self-signed certificate, and GitHub does not deliver webhooks to it:

HOST=hephaestus.example.com
openssl req -x509 -newkey rsa:3072 -nodes -sha256 -days 365 -subj "/CN=$HOST" \
  -addext "subjectAltName=DNS:$HOST" -keyout gateway.key -out gateway.crt
kubectl --namespace kube-system create secret tls hephaestus-tls --cert=gateway.crt --key=gateway.key

The server trusts forwarded client addresses only from the Gateway Pods. The IP column of this command shows their addresses:

kubectl get pods --namespace kube-system --selector app.kubernetes.io/name=traefik --output wide

Change the namespace and the label for another Gateway controller. A Pod gets a new address when it restarts, so trustedProxies matches the Pod network, not one address.

6. Write the values file​

Write your values file, site.yaml. This example is for small.yaml and the k3s Traefik above:

site.yaml
host: hephaestus.example.com
# A Tomcat regular expression for the Pod network of the Gateway Pods: k3s uses 10.42.0.0/16.
trustedProxies: '10\.42\.\d{1,3}\.\d{1,3}'
secret:
  name: hephaestus-app
postgresql:
  tls:
    existingSecret: hephaestus-postgresql-tls
route:
  parentRefs:
    - name: traefik-gateway
      namespace: kube-system
      sectionName: websecure
networkPolicy:
  # The Gateway Pods that may send requests to the release.
  gateway:
    namespace: kube-system
    podLabels:
      app.kubernetes.io/name: traefik
      app.kubernetes.io/instance: traefik-kube-system
  # The Kubernetes API endpoint, for small.yaml only.
  kubernetesApi:
    cidr: 192.168.5.15/32
    port: 6443
# For small.yaml only.
sandbox:
  namespace: hephaestus-sandboxes
config:
  github:
    oauthClientId: Ov23liExampleClientId
  auth:
    bootstrapAdmins: ["github:1234567"]

  1. Set host to your host name.

  2. Set trustedProxies to a regular expression for the Pod network of your Gateway Pods. Use single quotes, or no quotes. In double quotes, YAML reads each backslash as the start of an escape sequence.

  3. Set route.parentRefs to your Gateway and its HTTPS listener.

  4. Set networkPolicy.gateway to the namespace and the Pod labels of your Gateway Pods. The chart refuses a namespace without labels, because it would admit every Pod in that namespace.

  5. For small.yaml, find the Kubernetes API endpoint that the network plugin sees:

    kubectl get endpointslices --namespace default --selector kubernetes.io/service-name=kubernetes

    Set networkPolicy.kubernetesApi.cidr to the address in ENDPOINTS, with /32, or /128 for IPv6. Set networkPolicy.kubernetesApi.port to the value in PORTS. If ENDPOINTS lists one address for each control plane node, set a CIDR that holds all of them.

  6. Set config.github.oauthClientId and config.auth.bootstrapAdmins, as First steps § 1 and § 2 describe.

  7. For a public instance, name your legal pages ConfigMap in webapp.legalOverrides, as Legal pages describes. The webapp Pod waits until the ConfigMap that it names exists.

Do not set release or images: the release chart sets them. For dev.yaml, leave out networkPolicy.kubernetesApi and sandbox.

7. Install the chart​

  1. Render the chart with your profile and your values. This step needs no cluster:

    helm template hephaestus "chart-$VERSION/hephaestus" -n hephaestus \
      -f "chart-$VERSION/hephaestus/profiles/small.yaml" -f site.yaml >/dev/null

    If Helm prints an error, correct the value that it names, and render again.

  2. Install the chart with the same files:

    helm upgrade --install hephaestus "chart-$VERSION/hephaestus" -n hephaestus \
      -f "chart-$VERSION/hephaestus/profiles/small.yaml" -f site.yaml \
      --wait=watcher --wait-for-jobs --timeout 45m

The first install has no automatic rollback, so a failed install keeps its Jobs and Pods for diagnosis. If it fails, use A first install fails before you remove anything.

Check that it worked​

  1. Find the revision of the install with helm history hephaestus --namespace hephaestus.

  2. Read the log of its migration Job: kubectl --namespace hephaestus logs job/hephaestus-migration-<revision>. The Job logs Database migration completed. The JVM then prints a long native memory summary. It is not an error.

  3. Wait until the server is ready:

    kubectl --namespace hephaestus rollout status deployment/hephaestus-server

    On a first install, the server can restart while PostgreSQL starts. This is not a failure.

  4. Check that the Gateway accepted the route:

    kubectl --namespace hephaestus get httproute -o jsonpath='{range .items[*].status.parents[*].conditions[*]}{.type}={.status} {.reason}{"\n"}{end}'

    The output shows Accepted=True and ResolvedRefs=True. Accepted=False NotAllowedByListeners means that the listener does not accept routes from the release namespace.

  5. Open https://<host>. The login page shows Continue with GitHub.

  6. For small.yaml, run the sandbox isolation gate for each sandbox template. Connect no repository until the gate passes for each template.

Then continue with First steps § 3. Sign in.

Undo​

  1. Delete the sandbox namespace first. This stops every sandbox at once. If you remove the release first, its network policies go, and the remaining sandbox Pods lose their policies.

    kubectl delete namespace hephaestus-sandboxes

  2. Remove the release. Running reviews stop.

    helm uninstall hephaestus --namespace hephaestus

  3. These items stay:

    • The PostgreSQL volumes.
    • The Secrets that you created.
    • The backups in the S3 bucket.

  4. To remove the data too, delete the release namespace: kubectl delete namespace hephaestus. A PersistentVolume with the Retain reclaim policy stays until you delete it.

Troubleshooting​

A first install fails​

A failed first install keeps its Helm revision, Jobs and Pods when it runs without --rollback-on-failure. On one host, first select the kubeconfig of the installer, as A first install fails on one host describes.

  1. Preserve the terminal output of the install before you try again.
  2. Check the Helm record with helm list --namespace hephaestus.
  3. If the release remains, read its history with helm history hephaestus --namespace hephaestus.
  4. List remaining Jobs and Pods with kubectl --namespace hephaestus get jobs,pods -o wide.
  5. Read recent events with kubectl --namespace hephaestus get events --sort-by=.metadata.creationTimestamp.
  6. If the migration Job remains, preserve its log with kubectl --namespace hephaestus logs job/<migration-job>. Replace <migration-job> with the Job name for the failed Helm revision from steps 3 and 4.
  7. If a server Pod remains, read its log before the next install replaces it.

Events expire, and the log of a removed Pod can be gone. A failed install can still have changed the database. Use Failed-migration recovery before you start an older release against it. For durable logs, configure a log collector before the install.

Other symptoms​

SymptomCause and check
A server log says that the network policy did not filter a sandbox Pod in timeThe network plugin applied the policy of a new Pod more than 60 seconds after the Pod started. The attempt is tried again.
A sandbox Pod stays Pending with a quota errorIts tier's quota is full. The chart sizes each tier for its servers' capacity, so check a quota of your own, or sandbox.reviewMemory and sandbox.conversationTurns.
Every path returns 404, and kubectl get httproute shows Accepted=False NotAllowedByListenersThe listener of the Gateway does not accept routes from the release namespace. Set allowedRoutes.namespaces on the listener, as Prepare the Gateway describes.
A new Pod restarts during an upgrade, and its log names a pending change setIt waited 10 minutes for the schema. Check the migration Job's log. A longer migration needs the database maintenance procedure.
A backup Job fails with unable to find primary clusterThe Job cannot reach pgBackRest's TLS server on port 8432 of the PostgreSQL Service, or its client certificate is not for hephaestus-backup from ca.crt.
PostgreSQL's volume fillsA failed WAL archive keeps the WAL on the volume. Check the latest backup Job's log and the bucket's credentials.
A practice review fails with "The sandbox could not prepare its workspace."The sandbox's initializer failed before any model call. The job runs again within its retry limit, so a failure that repeats on every attempt uses all of them. The server log names the initializer's exit code. After the last attempt, read the exit code and the last lines of the initializer's log with the command below this table. Known credential formats in those lines are masked.

Replace <job id> with the ID of the review job:

kubectl --namespace hephaestus exec -i hephaestus-postgresql-0 -c postgresql -- \
  sh -c 'exec psql -U "$POSTGRES_USER" -d "$POSTGRES_DB"' <<'SQL'
SELECT capture->'initializationFailure' FROM review_job WHERE id = '<job id>';
SQL