Sizing and scaling
This page gives the capacity of each chart profile, the values that change it, and the database connection budget.
The numbers come from the profiles and values.yaml of the chart.
They are scheduling requests, not measured peaks. Keep room for the operating system, the cluster and a rolling update.
Profiles at a glance
| Profile | Nodes | Servers | Requests of each server | PostgreSQL requests and volume | Reviews at once | Conversation turns at once |
|---|---|---|---|---|---|---|
dev.yaml, preview.yaml | One workstation node | 2 | 250m CPU, 3 GiB memory, 5 GiB Git cache | 100m CPU, 1 GiB memory, 5 GiB volume | 0: sandboxes are off | 0 |
small.yaml | One node of 4 vCPU, at least 16 GiB of memory and 150 GiB of disk | 1 | 500m CPU, 3 GiB memory, 20 GiB Git cache | 250m CPU, 2 GiB memory, 20 GiB volume | 5 | 11, or 7 with backups on (auto) |
One host: small.yaml with k3s-values.yaml | The node of small.yaml | 1 | As small.yaml | As small.yaml | 4 | 9, or 6 with backups on (auto) |
cluster.yaml | Several nodes, with sandbox nodes that hold the review and conversation quotas | 2 | 500m CPU, 3 GiB memory, 70 GiB Git cache | 250m CPU, 2 GiB memory, 20 GiB volume | 12 | 8 |
Each server's memory request equals its limit, because the JVM sizes its heap from the limit.
The server's CPU limit is 4 cores, and 2 with dev.yaml and preview.yaml.
The Git cache is node storage of each server (Storage and retention).
The one host has fewer reviews and turns than small.yaml alone, because each gVisor sandbox Pod also takes 48 MiB and 10m CPU of RuntimeClass overhead.
The sandbox quotas hold these requests at full load, with any RuntimeClass overhead:
| Profile | Review quota | Conversation quota |
|---|---|---|
small.yaml | 5 Pods: 500m CPU, 2560 MiB memory | 11 Pods: 1100m CPU, 4224 MiB memory |
| One host | 4 Pods: 440m CPU, 2240 MiB memory | 9 Pods: 990m CPU, 3888 MiB memory |
cluster.yaml | 12 Pods: 1200m CPU, 6144 MiB memory | 8 Pods: 800m CPU, 3072 MiB memory |
Add the webapp (50m CPU, 64 MiB) and the agent image DaemonSet (1m CPU and 16 MiB on each sandbox node).
With cluster.yaml, also add the agent-sandbox controller (10m CPU, 64 MiB).
During an install or an upgrade, the migration Job requests 250m CPU and 1536 MiB more.
Check the rendered requests against the allocatable resources of your nodes before you install.
Raise capacity
| To get | Change | Also needs |
|---|---|---|
| More reviews at once | sandbox.reviewMemory: 512 MiB more for each review, plus the RuntimeClass overhead | That memory on the sandbox nodes. Each review also takes about 18 MiB of the server's memory and two database connections of its pool. |
| More conversation turns at once | sandbox.conversationTurns, or with auto a larger sandbox.budget | 384 MiB and 100m CPU on the sandbox nodes for each turn (Sandbox capacity budget) |
| More API capacity, and reception during a restart | server.replicas | A server request for each replica, and its database connections. More servers do not add reviews or turns. |
| Larger repositories | server.gitCache.sizeLimit | That much free ephemeral storage on each server's node |
| More database space | postgresql.persistence.size | Change a volume size |
| A larger review or conversation sandbox | sandbox.templates.<kind>.resources | The quotas follow the requests (Sandbox sizes) |
These signals show that a capacity limit holds work back:
| Signal | Meaning |
|---|---|
agent_queue_oldest_age_seconds, HephaestusAgentQueueStalled | Reviews wait for a review sandbox. |
mentor_turn_queued, mentor_turn_queue_wait_seconds | Questions wait for a conversation turn. |
hikaricp_connections_pending, HephaestusDatabaseConnectionTimeouts | Server threads wait for a database connection. |
HephaestusSandboxGatewayConnectionsHigh | The sandbox gateway is near sandbox.gateway.maxConnections. |
HephaestusGitCacheNearlyFull | The Git cache is near server.gitCache.sizeLimit. |
llm_pool_permit_wait_seconds | Calls wait for the model provider, not for the cluster (Call pool metrics). |
PostgreSQL memory
The bundled image calculates starting settings from its container limit.
postgresql.parameters adds explicit -c arguments after that calculation.
The small profile uses these values for a shared node:
| Setting | Value |
|---|---|
shared_buffers | 256MB |
work_mem | 2MB |
hash_mem_multiplier | 1.0 |
max_connections | 120, derived from the connection budget |
maintenance_work_mem | 64MB |
autovacuum_work_mem | 32MB |
autovacuum_max_workers | 3 |
effective_cache_size | 1GB |
max_parallel_workers_per_gather | 0 |
max_wal_size | 1GB |
work_mem applies to each sort or hash operation and can multiply across concurrent queries.
effective_cache_size is a planner estimate. It allocates no memory.
See PostgreSQL resource consumption.
The PostgreSQL container requests its full 2 GiB memory limit. With backups on, its pgBackRest sidecar also requests its full 512 MiB limit. At 120 connections, an allowance of 5 MiB for each backend, the shared buffers, autovacuum, maintenance memory and other memory total about 1512 MiB. Two work operations for each connection bring the estimate to about 1752 MiB. Both estimates fit the 2 GiB limit.
Database connections
Each chart role has databasePool.maximumPoolSize and databasePool.minimumIdle.
With maximumPoolSize: auto, the chart derives the server pool from the caps that it gives the server:
pool = inbox handlers + 2 × reviews at once + 2 × long-load preparations
+ permit maintenance threads + sync reserve + HTTP/control reserve
| Part | Connections | Source |
|---|---|---|
| Inbox handlers | 8 | server.inboxClaimers |
| Review captures | 2 for each review at once | sandbox.reviewMemory |
| Long-load preparations | 4, with sandboxes on | Fixed |
| Permit maintenance | 2, with sandboxes on | Fixed |
| Sync reserve | 8, for four syncs | Fixed |
| HTTP and control reserve | 4 | Fixed |
The result is 20 for dev.yaml and preview.yaml, 36 for small.yaml, 34 on one host, and 50 for cluster.yaml.
An explicit server.databasePool.maximumPoolSize must hold the derived count. The chart refuses a smaller one.
A model stream and a sandbox hold no pooled connection. Sandboxes reach the server through the gateway and have no database pool.
The chart derives max_connections for one rolling update, in which every old server can still drain beside its replacement:
| Client or reserve | small.yaml |
|---|---|
Servers at overlap: twice server.replicas | 2 |
| Server pools | 2 × 36 = 72 |
Five dedicated LISTEN sessions for each server | 2 × 5 = 10 |
| Migration Job | 2 |
| PostgreSQL metrics exporter, the monitor role's connection limit | 2 |
| Backup sessions | 2 |
| PostgreSQL probes and role setup | 3 |
| Client total | 91 |
superuser_reserved_connections | 3 |
| Operator margin | 4 |
| Required slots | 98 |
max_connections: rounded up to a multiple of 20, at least 120 | 120 |
The chart reserves the exporter and backup sessions also when they are off.
With cluster.yaml, four servers at overlap with pools of 50 give max_connections of 240.
An explicit postgresql.parameters.max_connections must hold the required slots. The chart refuses a smaller one.
An external database must hold the same budget beside its own reserved slots (PostgreSQL connection settings).
When all pooled connections are busy, a caller waits up to 45 seconds for one, then gets an error.
HephaestusDatabaseConnectionTimeouts fires on such errors.
Memory headroom
When the JVM starts, the buildpack's memory calculator sets the Java memory from the container's memory limit.
The default headroom (BPL_JVM_HEAD_ROOM) is 0 %, and the chart exposes no value for it.
Thus, a higher memory limit also gives the heap more memory at the next start.
Measure the server's total memory use before you change its memory limit.
See the Paketo memory calculator.
Disk
Budget the node's images, logs and sandbox scratch space separately from the PostgreSQL volume and the Git cache.
The sandbox templates bound their ephemeral storage (Sandbox sizes).
The small profile needs 150 GiB of disk on its node.