Measure node capacity

Use this procedure to measure the current chart and images on an isolated release. The workload uses typed TypeScript for conversations and the existing k6 scenarios. A passing request count alone does not qualify a node.

1. Prepare an isolated window

Agree on a window with the operator. Use synthetic accounts, workspaces and repositories only. Record the release, image digests, node capacity, allocatable memory, reserves and other workloads. Set a memory floor that leaves at least 1 GiB plus the eviction reserve. Stop new load before available memory crosses that floor.

The test-only fixture uses namespace capacity-bench-20261030 and a separate sandbox namespace. It must have its own PostgreSQL and generated credentials. Do not copy sessions, provider keys or application data from another release. Block public routes and external delivery. Retain the private paths that the application needs: DNS, database, sandbox gateway and Kubernetes API.

2. Prepare the replay

Generate the sensitivity inputs:

node scripts/capacity-provider.ts --grid > tmp/capacity-grid.json

Each case labels its provenance as assumed. The grid varies first-token time, output size and rate limiting. Its input size and stream speed are also assumptions. To use measured replay, supply paired samples with ttftMs, durationMs, inputTokens, outputTokens and status. Keep the source and provenance in the document. Do not reconstruct individual call distributions from attempt totals or independent percentiles.

Render the isolated provider:

node scripts/capacity-fixture.ts --replay tmp/replay.json --out tmp/provider.json

The provider never forwards a request. It records timestamps and synthetic correlation markers, without request bodies or credentials. It supports OpenAI chat completions and a bounded synthetic read-tool path. It is not a judge of practices or a complete review/composition fixture. Use --tool-mode none for a controlled single-call experiment. Keep that result separate from the default mixed-read case. Synthetic markers in conversation history can refer to an earlier turn. Validate every provider-to-turn timestamp join before you use it. The sweep sets CAPACITY_RUN_LABEL on the fake and includes stable user/turn keys in each question. Replay samples use that key and its call ordinal, so request UUIDs and arrival order do not change the sample. Use the logged replayKey and callOrdinal for phase joins in these runs.

--tool-mode roll scripts the turn of the k3s turn-roll drill (CI/CD), not a capacity case.

3. Install the test release

Use the chart's small profiles and the same image digests as the system to compare. The chart retains its normal value validation. The test-only Helm post-renderer lives under scripts/capacity-bench/. It refuses another Helm namespace, removes public routes and Internet paths, and substitutes the e2e profiles. It copies all application-prod.yml properties explicitly, except the listed synthetic auth settings. Record its output and all auth-only exceptions. Keep JVM resources, pool settings, timeouts, caches and logging equal to the compared release. The renderer restores the base values for every non-auth e2e property. A contract test covers the complete e2e property list. Production-only startup guards do not run in e2e; this fixture does not qualify those guards.

Use the test values in scripts/capacity-bench/values.yaml. Supply verified release image digests and replace its documentation API address with the qualified single IPv4 /32. Export that same address as BENCH_KUBERNETES_API_CIDR, and set HELM_NAMESPACE=capacity-bench-20261030 for Helm and the renderer. Use that environment variable to select the namespace; do not pass Helm's -n or --namespace flag. Provision this release's credentials and PostgreSQL TLS from the Kubernetes install procedure. Never use the existing installation's credentials. The renderer gives the three bench priority classes values below 1000 and preemptionPolicy: Never. If an earlier fixture created different classes, stop all its Pods before you replace those owned classes.

Helm 4 uses a postrenderer/v1 plugin. From the repository root, set HELM_PLUGINS="$PWD/scripts/capacity-bench" for the command. The supplied helm-plugin/plugin.yaml resolves its entrypoint beside that plugin directory. See the Helm plugin guide. Do not install this plugin in the operator's normal plugin directory.

Run the driver in the server Pod and use loopback for e2e sign-in. The fixture seeder creates synthetic sessions through the supported endpoint and completes the notice.

1. Run capacity-bench-seed.ts with CAPACITY_SEED_MODE=sessions and CAPACITY_USERS set to the cohort size.
2. Run scripts/capacity-bench/seed-identities.sql with psql -v ON_ERROR_STOP=1 inside bench-postgresql-0 in capacity-bench-20261030. It refuses a database with non-synthetic workspaces or accounts before it writes anything. It creates only the synthetic workspace and loopback identity provider, then links the synthetic memberships and identities.
3. Run the seeder with CAPACITY_SEED_MODE=models to configure the fake model.

A linked identity is required by the normal person-data write fence. The model and its binding must have the same declared privacy tier. Keep the session file private in the driver's ephemeral volume. The seeder configures both conversation and review purposes to the isolated fake. It uses no real provider credential.

4. Run conversations first

Copy the harness and the verified k6 binary into the driver container under /bench. Use the same k6 version as scripts/load-test.ts. Record its artifact checksum. Run a smoke before a recorded cohort.

The stream observer creates no questions:

node /bench/scripts/capacity-stream-observer.ts --seconds 1200

The first stream body byte can be a snapshot or heartbeat. Use the first nonempty answer text for user-visible first-token time. Keep both observations.

Run S3 after the operator confirms the quiet window:

node /bench/scripts/capacity-bench-run.ts --label s3-first --mode recorded \
  --quiet-window /bench/quiet-window.json

S3 offers forty independent session arrivals in one second. Set COURSE_USERS to change the cohort to 80 or 120 users. Each session asks three turns with think time. The initial batch does not wait for a completed session before another user arrives. Set COURSE_CHAT_RATE for a separate sustained open-arrival test. Exact completed counts and refused/dropped arrivals must reconcile. An empty completion rate must fail the run. Keep failed and interrupted evidence.

Sweep turn admission

Use COURSE_SSE=1 for the typed TypeScript driver. It opens one event subscription per identity and waits for terminal turn events without polling. It records arrival scheduling, request send, first turn event, first answer text and completion. The existing polling driver remains a diagnostic path; its extra HTTP requests affect CPU measurements.

Set the server's admission and its chart-derived sandbox quota together in the isolated release. Keep sandbox requests and limits unchanged. For each limit, use 40, 80 and 120 users with three turns and 30-second think time. Set the fake to 3-second first token, 500 output tokens and the recorded stream speed.

Create an operator state file with state and source fields. Set state to CLEAR only after the operator explicitly releases the window. Set it to QUIET when the operator requests a stop. Record the message or decision in source. Set stopAt to a future UTC timestamp that leaves enough time to restore the operator's replicas. The step checks this file and the merge queue before load and every 30 seconds. It also detects changes to main or the shared deployment generations.

node scripts/capacity-bench-step.ts --label admission-16-users-40 \
  --admission 16 --users 40 --operator-state tmp/operator-state.json

The sampler stops the driver below 1 GiB plus a 256 MiB eviction reserve. An unexpected sampler exit also stops the load. Use scripts/capacity-bench-stop.ts inside the driver container to cancel its synthetic turns through the owner API. The step verifies idle synthetic owners before load, then resets their conversations in the isolated database. After load, it verifies owner cleanup again and saves checksums for every receipt. The SSE driver uses snapshot recovery reads when the server drops notifications; it does not poll periodically. Export receipts before you stop the server Pod: the driver volume is ephemeral. Keep interrupted runs separate from completed capacity results.

The additional course presets describe review admission and publication loads. Direct review requests bypass provider-event selection. The existing synthetic webhook event measures publication, not downstream provider sync or duplicate-ID replay. Do not use these paths to claim end-to-end review capacity without the corresponding synthetic corpus and delivery receipts.

5. Collect and interpret

Use scripts/kubernetes-footprint.ts for synchronized node and container counters. It records charged memory, process RSS, cache, swap, CPU counters, pressure and OOM counters. Use --bench-postgres to include connection and lock-wait counts from the isolated database. Each sample is also written to JSONL so an interruption does not lose the trace. Process RSS can count shared pages more than once. Use cgroup charged memory for the budget and keep RSS as a separate observation. An absent kernel lifetime peak remains missing; sampled peaks can miss short events. Capture PostgreSQL connections/lock waits and inbox backlog alongside these samples.

Correlate request send, queue/claim, first provider call, provider token/end and turn end. Use matching tool logs to attribute tool spans. Gaps between provider calls contain non-model work, including scheduling and transport. Use interval unions when spans overlap. Do not add or subtract independently computed percentiles.

Increase offered load until a latency objective, the memory floor or a real provider budget fails. Name the binding resource at each step. For steady state, check Little's law with means over identical work boundaries. For a growing queue or finite burst, report backlog slope and drain instead. Run the two-hour soak only after the steady-state workload passes. Qualify the final artifact again when runtime or resource settings change. Restore every changed setting and release the operator's window when the run ends.

Check that it worked

Check that the expected users and turns completed, with no unexplained refusals or missing arrivals. Check that the resource trace covers the whole load and drain interval. Check that the operator state and merge queue stayed clear. Check that no synthetic turn remains active after cancellation or failure. Keep failed, interrupted and contaminated runs; do not use them as qualified capacity.

Stateful resource observations

The observation window runs from 2026-10-31T09:56:22Z to 2026-11-05T08:31:18Z, about 4.94 days. The metrics store contains no earlier container samples, including a query over 30 days. These observations cover normal activity across several releases, not a controlled review-width load or a backup qualification.

Container	Peak working set	Peak total memory charge	Retained request / limit
PostgreSQL	1,202.44 MiB	2,047.67 MiB	2 GiB / 2 GiB

Measure stateful memory gives the query method and the qualification criteria.

The separate conversation benchmark on 2026-10-30 had synthetic chat load and zero reviews. Its PostgreSQL maximum was 1,544.31 MiB RSS and 291.25 MiB charged, across the three admission cases.

The sampler sums process RSS, which can count shared pages more than once, as Collect and interpret explains. Do not substitute RSS for cgroup charge or working set, or add these independent maxima. Neither observation qualifies the lower PostgreSQL target. Keep the current ceilings until the reference workload includes reviews, vacuum, backups and inbox catch-up.

Measure stateful memory

The queries take each container's maximum over the window, then the maximum across its replacement containers. They exclude Pod-level cgroups and infrastructure containers. The working-set and charge maxima need not occur at the same time. Working set excludes some reclaimable file cache. The container's limit also covers that cache. Thus, working set alone does not justify a lower limit.

For example, query the working set with this PromQL expression:

max by (container) (max_over_time(container_memory_working_set_bytes{namespace="hephaestus",container=~"postgresql"}[14d]))

Use container_memory_usage_bytes for total charge, and tfirst_over_time to check the actual sample window in VictoriaMetrics. A 14-day range selector does not prove that 14 days of samples exist. Compare total charge with inactive file cache, memory pressure and OOM counters before you attribute a peak to reclaimable cache.

Qualify a lower PostgreSQL limit

The small profile keeps a PostgreSQL limit of 2 GiB. A limit of 1536 MiB is a qualification target, not a setting. Count the pgBackRest server sidecar separately, and count a backup or stanza Job in addition. Do not add the p95 values of single containers to claim a synchronized node peak.

Before you lower the process limit, run the reference load with the representative database, vacuum work, WAL archiving, and the permitted server surge. Read pg_settings to verify the effective values and their command-line source. Require process charge within 1536 MiB, no OOM, no Hikari timeouts, and connection-acquire p95 below 50 ms. Require WAL archive lag below 60 seconds during steady load and below five minutes during recovery. Require autovacuum to reduce dead tuples after the peak, and investigate idle transactions older than one minute. Keep the larger limit and lower concurrency if the trial fails.

The same load qualifies the twelve-connection reserve of the server pool. Measure hikaricp_connections_pending, acquisition timeouts and acquisition latency with inbox, capture, conversation and HTTP work together.

Recorded conversation baseline

This measurement ran on release 0.95.0, with 4 vCPUs and 15.61 GiB of memory, before conversation turns ran in sandboxes. It does not describe the sandbox runtime. Each case had 40 synthetic users, three turns each and 30 seconds between turns. The users arrived over one second. The deterministic provider used a first token at 3 seconds, 500 declared output tokens and 13 seconds per call. Each turn made one call, with no tools, reviews or provider throttles.

Turn admission	First text p50 / p95	Turn duration p50 / p95	First text within 10 s	Cohort drain, turns/h
8	25.78 / 55.97 s	35.56 / 65.76 s	8/120 (6.67%)	2,164
16	3.46 / 31.27 s	13.24 / 41.05 s	96/120 (80.00%)	3,332
32	3.57 / 17.88 s	13.37 / 27.66 s	112/120 (93.33%)	3,721

Each case misses the target of first answer text within 10 seconds for 95% of turns. The drain rate divides completed turns by the whole case duration. It is not a sustained arrival rate. The CSV results include the measured commit, cohort and provider assumptions.