Install on a managed cluster without cluster-admin

This procedure installs Hephaestus on a Kubernetes cluster that other people administer, for example a Rancher-managed RKE2 cluster. You have kubectl access to two namespaces, but you are not a cluster-admin. The cluster's admins apply the cluster-scoped objects of the release, the admin bundle. You install everything else with the Helm chart and the cluster.yaml profile.

The release runs the agent-sandbox controller in its own namespace, and every write right of the controller is in a Role of the chart. The pinned controller cannot limit what it reads to one namespace. Thus, the admin bundle grants it one ClusterRole that reads Pods, Services, NetworkPolicies and the four agent-sandbox kinds in all namespaces. The controller can read the specs of the Pods and Services of other namespaces, including their environment values. Each user with the built-in admin role in the release namespace can impersonate the controller and read them too. The admins must accept that before they apply the bundle.

Before you start

The cluster needs these items. Send this list to the admins. The preflight in Install checks each item that your access can read.

A cluster that the chart supports, Kubernetes 1.35 or later. Supported clusters gives the full requirements.
A network plugin that enforces NetworkPolicy for egress as well as ingress. RKE2's default, Canal, does.
The Gateway API with a controller, and a Gateway listener that accepts HTTPRoutes from the release namespace. It serves the public host name of the release with TLS.
Sandbox nodes that meet the sandbox prerequisites. They need user namespace support (Linux 6.3 or later, containerd 2.0 or later) and encrypted node storage.
A StorageClass for ReadWriteOnce volumes, as the default class or by name, for the PostgreSQL volume: 20 GiB by default (postgresql.persistence.size). An external PostgreSQL replaces the PostgreSQL volume.
Egress on port 443 from the release namespace to the Git providers and the model APIs that the workspaces use.
Image pulls from ghcr.io, docker.io and registry.k8s.io on every node.
No other agent-sandbox controller on the cluster. A second controller would also act on the release's sandboxes.

Get the release files of the version that you install:

The chart, which you verify.
The k3s bundle hephaestus-k3s-X.Y.Z.tar.gz with its signature, which you verify as Get the release files shows. It holds the admin bundle, hephaestus-cluster-admin.yaml.
A checkout of the release's Git tag, from which you run the preflight and the sandbox isolation gate.

On your workstation, you need these tools:

kubectl with a context for the cluster.
Helm 4, on the PATH. The install uses --wait=watcher, and the preflight renders the chart with helm.
Node.js at the version that package.json of the checkout names. Node.js includes Corepack, which installs the pinned pnpm.

In the directory of your values file, get the checkout and the packages of the preflight:

git clone --depth 1 --branch "v$VERSION" https://github.com/hephaestus-build/Hephaestus.git hephaestus-src
(cd hephaestus-src && corepack pnpm install --frozen-lockfile --ignore-scripts)

--ignore-scripts runs no install script of a package. The preflight does not need one.

Admin handoff

Give the admins hephaestus-cluster-admin.yaml from the verified bundle, and these steps. The admins repeat step 3 for each release.

Create the release namespace and the sandbox namespace with the Restricted Pod Security level:

kubectl create namespace hephaestus
kubectl create namespace hephaestus-sandboxes
kubectl label namespace hephaestus hephaestus-sandboxes pod-security.kubernetes.io/enforce=restricted

Grant the operator the built-in admin ClusterRole in both namespaces with a RoleBinding. Also grant the two Roles below with RoleBindings to the operator. admin does not cover HTTPRoutes, ResourceQuotas or the agent-sandbox kinds. The chart creates Roles that use these rights. Kubernetes lets a user create a Role only with the rights that the user holds. The quota rule changes only the three quotas of the release, so the admins keep their own quotas.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { name: hephaestus-operator, namespace: hephaestus }
rules:
  - apiGroups: [gateway.networking.k8s.io]
    resources: [httproutes]
    verbs: [get, list, watch, create, update, patch, delete]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { name: hephaestus-operator, namespace: hephaestus-sandboxes }
rules:
  - apiGroups: [""]
    resources: [resourcequotas]
    verbs: [get, list, watch, create]
  - apiGroups: [""]
    resources: [resourcequotas]
    resourceNames:
      - hephaestus-sandbox-quota
      - hephaestus-sandbox-quota-reviews
      - hephaestus-sandbox-quota-conversations
    verbs: [update, patch, delete]
  - apiGroups: [agents.x-k8s.io]
    resources: [sandboxes, sandboxes/status, sandboxes/finalizers]
    verbs: [get, list, watch, create, update, patch, delete]
  - apiGroups: [extensions.agents.x-k8s.io]
    resources:
      - sandboxclaims
      - sandboxclaims/status
      - sandboxclaims/finalizers
      - sandboxtemplates
      - sandboxtemplates/finalizers
      - sandboxwarmpools
      - sandboxwarmpools/status
      - sandboxwarmpools/finalizers
    verbs: [get, list, watch, create, update, patch, delete]

With observability.stack.enabled or alert rules, the operator also needs the Prometheus Operator kinds that they use. The preflight names them.

The sandbox isolation gate also reads the kubernetes Service and its EndpointSlices in default, and the cluster DNS Service. Grant these two Roles to the operator with RoleBindings too. Change the name and the namespace of the DNS Service to those of the cluster:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { name: hephaestus-isolation-gate, namespace: default }
rules:
  - apiGroups: [""]
    resources: [services]
    resourceNames: [kubernetes]
    verbs: [get]
  - apiGroups: [discovery.k8s.io]
    resources: [endpointslices]
    verbs: [list]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { name: hephaestus-isolation-gate, namespace: kube-system }
rules:
  - apiGroups: [""]
    resources: [services]
    resourceNames: [rke2-coredns-rke2-coredns]
    verbs: [get]

Apply the admin bundle:
```
kubectl apply --server-side -f hephaestus-cluster-admin.yaml
```
It holds the four agent-sandbox CRDs, the three PriorityClasses of the release, and the read-only ClusterRole hephaestus-agent-sandbox-read with its ClusterRoleBinding. It holds nothing else.

The ClusterRoleBinding names the controller's ServiceAccount in the namespace hephaestus. For another release namespace, do not edit the signed bundle. Bind the ClusterRole to the ServiceAccount hephaestus-agent-sandbox-controller of that namespace instead. At each apply, run these commands instead of the command above:
```
kubectl delete clusterrolebinding hephaestus-agent-sandbox-read --ignore-not-found
kubectl apply --server-side -f hephaestus-cluster-admin.yaml
kubectl delete clusterrolebinding hephaestus-agent-sandbox-read
kubectl create clusterrolebinding hephaestus-agent-sandbox-read \
  --clusterrole hephaestus-agent-sandbox-read --serviceaccount <namespace>:hephaestus-agent-sandbox-controller
```
The first command removes your earlier binding, so that the apply does not conflict with it.
Tell the operator these facts:
- The name, namespace and listener of the Gateway, and the Pod labels of its data plane.
- The address and port of the Kubernetes API as the network plugin sees it, from kubectl get endpointslices --namespace default --selector kubernetes.io/service-name=kubernetes.
- The name of the cluster DNS Service. On RKE2 it is kube-system/rke2-coredns-rke2-coredns.
- The StorageClass, if it is not the default class.
- If sandboxes must run on dedicated nodes: the node selector and the tolerations of those nodes.

Install

Create the release Secret and the PostgreSQL TLS Secret in the release namespace. Follow Create the Secret and Create the PostgreSQL TLS Secret. You have hephaestus-keys.sh from the k3s bundle.
Write your values file, site.yaml, as Write the values file describes for host, trustedProxies, the Secrets and the login provider. Add the facts from the admins:
site.yaml
```
route:
  parentRefs:
    - name: cluster-gateway
      namespace: gateway-system
      sectionName: https
networkPolicy:
  gateway:
    namespace: gateway-system
    podLabels:
      app.kubernetes.io/name: gateway
  kubernetesApi:
    cidr: 192.168.10.11/32
    port: 6443
sandbox:
  namespace: hephaestus-sandboxes
```
kubernetesApi is an address from the EndpointSlice, not the address of the kubernetes Service (Network policy). A cluster with three server nodes has three addresses: set a CIDR that holds all of them. Use /128 for one IPv6 address.

If the admins gave a node selector and tolerations for the sandbox nodes, set them as sandbox.runtime.nodeSelector and sandbox.runtime.tolerations.
Run the preflight in the directory of your values file:
```
node hephaestus-src/scripts/cluster-preflight.ts --namespace hephaestus \
  --values "chart-$VERSION/hephaestus/values.yaml" \
  --values "chart-$VERSION/hephaestus/profiles/cluster.yaml" \
  --values site.yaml
```
The chart of the checkout names no release images, so the first file gives them from the verified chart. The second file applies the cluster profile again after it. The preflight renders the chart with these files, and checks the cluster with your own access. It does not change the cluster.

Each missing: line names one item that the install needs. Forward it to the admins as it is. Each unverified: line names a fact that your access cannot read. Ask the admins to confirm it. The preflight fails while one missing: line remains.

The preflight checks the controller's read-only ClusterRole as the controller's ServiceAccount, with kubectl auth can-i --as. Without the right to impersonate that account, it compares the ClusterRole and its binding with the admin bundle instead.
Install the verified chart with the cluster profile and your values:
```
helm upgrade --install hephaestus "chart-$VERSION/hephaestus" -n hephaestus \
  -f "chart-$VERSION/hephaestus/profiles/cluster.yaml" -f site.yaml \
  --wait=watcher --wait-for-jobs --timeout 45m
```
The profile runs two servers, twelve reviews at once across the sandbox nodes, and the agent-sandbox controller in the release namespace. It names the PriorityClasses of the admin bundle, so the chart creates no cluster-scoped object. Sizing and scaling gives its requests.

Check that it worked

Run the preflight again after the install, with --installed:
```
node hephaestus-src/scripts/cluster-preflight.ts --namespace hephaestus \
  --values "chart-$VERSION/hephaestus/values.yaml" \
  --values "chart-$VERSION/hephaestus/profiles/cluster.yaml" \
  --values site.yaml --installed
```
It also waits up to 10 minutes for the server rollout. The last line starts with The preflight of hephaestus passed. If the server is not ready, a missing: line gives the command that shows its log.
Run the sandbox isolation gate for each template. Add the cluster's DNS Service, for example --dns kube-system/rke2-coredns-rke2-coredns on RKE2. The rights of step 2 of the admin handoff cover the gate. Each run must end with The sandbox isolation gate passed.
Open the public host name, sign in, and request a review of one pull request.

Undo

Delete the sandbox claims while the agent-sandbox controller still runs:
```
kubectl -n hephaestus-sandboxes delete sandboxclaims --all
```
Uninstall the release:
```
helm uninstall hephaestus -n hephaestus
```
Optional: delete the PostgreSQL volume claim, which the uninstall keeps.

CAUTION: Data loss
This deletes the database. Keep a backup and the required keys if you can restore later.
```
kubectl -n hephaestus delete pvc data-hephaestus-postgresql-0
```
Ask the admins to delete the two namespaces, which also deletes the release Secret and your RoleBindings:
```
kubectl delete namespace hephaestus hephaestus-sandboxes
```
Also ask them to delete the Roles hephaestus-isolation-gate and their RoleBindings in default and in the namespace of the DNS Service.
Ask the admins to remove the admin bundle of the release:
```
kubectl delete -f hephaestus-cluster-admin.yaml
```
This deletes the agent-sandbox CRDs, the three PriorityClasses and the read-only ClusterRole with its binding.

Before you start​

Admin handoff​

Install​

Check that it worked​

Undo​

Before you start

Admin handoff

Install

Check that it worked

Undo