CI/CD Architecture

Hephaestus uses trunk-based delivery through GitHub's merge queue. Pull requests provide early feedback, merge groups validate projected main, and pushes to main produce release images. Staging follows successful builds of main under the automatic promotion policy. Production runs a release only after an approved promotion.

Chart-only release

These instructions describe the chart-only release. For a 0.95.x deployment, use that release's versioned instructions and signed assets. The 0.x upgrade path requires 0.95.1 and its installed reconciler tooling before the cutover.

Architecture overview

Release flow

Every push to main publishes signed multi-architecture images for that commit and maintains the Version PR. Merging the Version PR cuts a release that promotes those images by digest — Release management. A pull request builds linux/amd64 only, so a push to it does not pay for a second architecture. A merge group builds linux/amd64 and linux/arm64, so an arm64 build failure or finding is found once per merge, before main. The merge queue squashes, so a merge group's SHA is the commit its push to main carries. That push reuses the images the merge group built and tested instead of building them again — Merge-queue image reuse.

The Version PR's branch is the exception. Every run builds every release image itself on both architectures, regardless of its start event. The branch's release evidence preflight requires each image on both published platforms. It can capture evidence only for images that its run published.

Run outline

The run uses stage names and static server names:

Changes
Workflow lint
Security / Dependencies, secrets and policy
Image advisories
Pinned upstream images
Quality / Server, Webapp, Extension, Tooling and docs, Tooling and docs tests (1 of 2), Tooling and docs tests (2 of 2), Windows, Clean install, Webapp stories
Server package
Build / Generated API, Database migrations, PostgreSQL drills
Server tests / Unit, Architecture, Integration (review), Integration (1 of 8) … Integration (8 of 8)
Browser tests / Webapp, Extension
Images / linux/amd64, linux/arm64, Manifest, Tag unchanged images
Kubernetes / k3s 1.37 amd64, k3s 1.35 amd64, k3s 1.35 arm64, kind 1.35 Calico
CI Status Gate

Server package, Image advisories and Pinned upstream images are independent top-level jobs. The package supplies the artifact to its consumers before their stages can start.

A path-filtered stage can be skipped. A selected stage shows its servers directly, including static names for skipped servers. The tooling gates and the two halves of the tooling tests remain three jobs. Each server keeps its isolated cancel callback, named Stop the merge group early.

The removed placeholder entries allocated no runner. Combining workflow linters saves one runner job when lint is selected. Folding chart lint into the latest k3s server saves one runner job when Kubernetes is selected. The checks remain the same.

GitHub displays every job of a reused workflow. A skipped job remains a skipped check. We use one stage workflow per group and YAML step aliases to share server steps without a second workflow level. The image signing steps stay in reusable-docker-build.yml, which is part of the signature identity.

CI Status Gate keeps its required context name. The workflow lint job remains a prerequisite of CI Status Gate. The gate continues to judge native results.

Quality gates

The local pre-push hook runs the focused checks in check:push. It does not replace the full quality gate or any CI check. Local verification defines its scope and exclusions.

Each quality leg runs its task under scripts/ci-deadline.ts, with a deadline of 15 minutes, under the job's 20. A task that hangs past it is listed with its processes and their command lines, in the log and in the job summary. Then the step stops the task and fails with exit code 124. The job's own limit cancels the run and can lose its log, so the list is the evidence of which task hung.

Job	Checks	Local command
Quality → Server	Spotless and PMD, NullAway policy, the test analysis (gate:server-test-analysis), the PMD canary when PMD inputs changed, source-use contracts, setting delivery per runtime role. On a pull request that changes Gradle inputs, the verification metadata check	vp run gate:server, gate:java-nullness, gate:contracts, gate:env, node scripts/check-gradle-verification-metadata.ts
Quality → Tooling and docs	toolchain and agent-runtime pins, the runner and lint contracts, gate:agents, load-test format and syntax, agent specs, instruction mirrors, docs format and docs:lint, Mermaid parse. The pinned ShellCheck on the shell scripts of deploy/platform/	vp run gate:agents, gate:docs, test:agents, gate:shell
Quality → Webapp	vp check, presentational components, story prose, docs color tokens, unit tests, Vite build with a routeTree.gen.ts diff and the bundle budget against the base build	vp run check:webapp, test:webapp, verification:webapp-build
Quality → Webapp stories	Storybook interaction tests, export:assets diff, Storybook build, Chromatic	vp run test:webapp:stories
Quality → Extension	vp check for extension/, format, unit and story tests, the production zip and its manifest check. Keeps the zip as extension-chrome-*	vp run ci:extension
Quality → Windows	Every gate that needs neither Gradle, Docker nor the agent sandbox, on windows-latest, uncached	vp run --no-cache ci:windows
Quality → Clean install	vp install --frozen-lockfile on a runner with Node and the launcher and nothing else, then gate:toolchain and the hooks path	vp install
Build → Server package	./gradlew :application:bootJar :application:testClasses. Uploads the classes and JARs once	—
Build → Generated API	generate:api diff against the packaged JAR. API compatibility with the previous stable release. db:check-drift, which replays master.xml and diffs the JPA model. ERD diff	vp run generate:api, gate:api-compatibility, db:check-drift, db:generate-erd-docs
Build → Database migrations	Gradle databaseTest task against PostgreSQL	—
Build → PostgreSQL drills	test:postgres-restore, then test:postgres-pitr, in one job beside databaseTest. Re-run failed jobs repeats both drills	vp run test:postgres-restore, test:postgres-pitr
Browser tests → Webapp	Playwright against the packaged JAR	vp run --filter webapp test:e2e
Browser tests → Extension	Playwright with the extension's e2e build — not the production zip — against the packaged JAR	vp run test:extension:e2e
Server tests → Unit, Architecture, Integration	the package job's compiled classes, without compiling again	vp run test:server:verification, test:server:integration
Dependency review → New dependency risk	informational dependency review. Pull requests that change a dependency input. Independent of CI Status Gate	—
Security → Dependencies, secrets and policy	Trivy filesystem scan, dependency vulnerability gate, TruffleHog, changelog immutability and compatibility guards, legacy-reference guards	—
Changesets → Verify changesets	policy tests, verified-revert detection, release-note presence, MIGRATION.md freeze, public contract	vp run gate:changesets, gate:public-contract
Images → linux/amd64, linux/arm64, Manifest, Tag unchanged images	One build job per platform builds every image of the run and scans it. The Dockerfile images use docker buildx bake. The Server image uses pack build from the packaged JAR. One manifest job merges, attests and signs them. Aliases for unchanged images	—
Workflow lint	workflow lint with ShellCheck warnings. Zizmor at medium confidence	—
PR → Validate PR	Conventional Commit title. Every commit’s primary author resolves to the PR author’s GitHub account (including bot accounts) or a listed review-policy maintainer. The title, body, and every commit message contain no Claude-Session: trailer, "Generated with" line, or session link (a Co-authored-by: trailer is allowed)	—

Build → Database migrations, the PostgreSQL drills, both E2E rows and the integration shards other than review are the queue tier. See Two verification tiers for migration PR exceptions.

The browser jobs retain webapp-e2e-results and extension-e2e-results, each including e2e-server.log, for seven days on success as well as failure. Inspect server startup and shutdown output even when browser assertions pass. A passing suite does not prove that background work started correctly. Its coverage summary names credential-gated live checks and their skipped reasons. The ordinary suite is not evidence of live-provider behavior.

Test checks show totals and failed suites/cases, not an inventory of successful tests. Source-file lookup runs only when failure annotations are needed. The complete unit, integration, database and Storybook JUnit reports remain downloadable for fourteen days, including successful runs.

Chromatic prints warnings/errors alongside the structured visual-coverage summary. Its JUnit report is retained with the Storybook reports. Do not publish its debug log or diagnostics file: they can contain credential-bearing signed URLs even when the project token is masked.

Reporter action inputs read job.status and explicit continued-step outcomes: status-check functions such as failure() are valid in a step's if condition, not in its with inputs. The CI contract parses those input expressions without status functions and tests successful and failed outcomes.

Image builds retain the complete buildpacks-log artifact for fourteen days. The console shows a first example and totals for repeated JVM archive-eligibility exclusions (unsupported location, already-linked class, signed JAR, JFR event class). These describe classes CDS cannot archive, not failed application operations. All preload and verification failures remain visible, including optional framework adapters: a previously investigated class can fail for a different reason after an update. Unfamiliar diagnostics and archive failures also remain visible. The reporting pipe cannot change the image build's exit status.

The quality servers in ci-quality-gates.yml share native YAML step aliases. Job-level conditions omit unchanged legs before a runner is allocated. In a pull request, selected legs do not cancel each other on failure. In a merge group, a failed leg cancels the run: Fail fast in the merge queue.

Each runs vp run ci:<leg>, then pipes vp run --last-details through scripts/report-task-run.ts. Each crossed task gets a notice: unpassed, cause unavailable, with its reproduction command. The runner's text report does not distinguish a task failure from a task that it stopped. The report order does not identify the cause. The job summary links the run attempt and keeps the report, including cache hits and miss reasons. Open the quality job and its Quality gates step to read the task output.

The quality step keeps its native failure verdict. The report also exits nonzero for unpassed tasks.

A leg runs when the change detection in cicd.yml says its scope changed. The Windows leg uses the tooling scope. Both it and the tooling leg prove the Git hooks on rejected and accepted commits.

A pull request that changes only server files, documentation, changesets or AGENTS.md and CLAUDE.md files usually defers the Windows leg. This includes the practice catalog and guidance. Changes to workflow or tooling files still select Windows through its path filter. The queue tier and ci:full also select it. Other paths select it when the tooling scope runs.

The merge group keeps the complete Windows leg for that scope. The ci:full label also runs it on the pull request. An unreadable label or a missing selection output runs it too. A Windows failure first found in the merge group removes the entry from the queue.

Public contract gate

vp run gate:public-contract protects what the compatibility policy promises outside the REST API. The Verify changesets job runs it for each pull request. It compares the checkout with the highest stable vX.Y.Z tag on origin, as API compatibility does.

Below v1.0.0 the gate refuses nothing. It only warns about a file that it can no longer find. Thus, the 1.0 cutovers can change these surfaces freely. A pre-release tag never becomes the base. To rehearse the gate, add --base v1.0.0-rc.1.

From v1.0.0, the gate fails on these changes, unless the change heads for a major release:

Surface	The gate fails on
Helm chart values	A value that is gone, a lower maximum, a higher minimum, a lost enum value, a lost type, or a lost pattern
Release Secret	A key that a chart template now reads without optional: a new key, or an optional key that became required
Shipped alert rules	An alert that is gone, or a metric that the rules no longer read
Practice definition schema	A field that is gone, or a field that accepts less, as for chart values
Catalog	A practice or group slug that is gone
Problem types	A slug of ProblemType that is gone
Practice script entry points	An export that is gone, or a changed contract version, outcome list or severity list
Links in posted text	A route in DEEP_LINKS that the web app no longer has

The gate cannot prove that one regular expression accepts more than another, so a changed pattern counts as lost. To widen a pattern, keep the old pattern in an anyOf branch beside the new one.

A change that moves one of these files updates the list in scripts/check-public-contract.ts. Otherwise, the gate warns below v1.0.0 and fails from v1.0.0.

API compatibility

A browser tab can run the SPA of one release against the server of the next release for days. The Chrome extension can be older. Thus, server/openapi.yaml must stay compatible with the previous stable release. vp run gate:api-compatibility compares the two specs with oasdiff breaking. It fails on each change at the ERR level. It shows WARN changes, but they do not fail it.

• Where it runs. Build → Generated API runs it on the spec that the job generates from the packaged JAR. The job runs for each pull request and merge group that changes the contracts. It also runs for the push of each version commit, which gates the release. The release workflow does not run the check again.
• The base. The base is the highest vX.Y.Z tag on origin. A release creates its tag only when it is published. The pattern does not select a pre-release tag such as v0.10.0-rc.1. The CI checkout has one commit and no tags, so the script fetches the base tag with --depth=1. To compare with a different release, add --base vX.Y.Z.
• The tool. The script downloads oasdiff from its GitHub release into a cache directory that Git ignores. Before each use, it verifies the archive against the SHA-256 that scripts/check-api-compatibility.ts pins for each platform. To update oasdiff, change the version and all checksums together. Compare each checksum in the release's checksums.txt with the asset digest on the release page.

The gate changes the level of some oasdiff rules. scripts/check-api-compatibility.ts lists them.

• A new value in a response enum is not a break, because response enums are open. Do not add an accepted break for one. The Chrome extension shows such a value as unknown. From 1.0.0, the web app also shows an unknown value without a route error. It offers a reload after an upgrade and keeps the current page until the reader chooses it.
• From a base of v1.0.0, a removed optional response property is an ERR change. A changed response media type is an ERR change too.
• From a base of v1.0.0, the gate fails on a removed operation or parameter. It also fails when the base release marks that operation or parameter deprecated. oasdiff alone accepts these removals, and the deprecation policy does not.

oasdiff does not compare operation IDs or tags. The policy does not promise them.

Accept a deliberate break

Before 1.0, a release can break the API deliberately. From 1.0, a break needs a major release. When the base is v1.0.0 or later, the gate fails on an entry that accepts a break, unless the change heads for a major release.

Before the Version PR, a pending changeset with "hephaestus": major marks the major release. From the Version PR on, the major of the package.json version is higher than the major of the base. The changeset stays on main until the Version PR consumes it. Thus, other changes can accept breaks for the same major release.

To accept a break:

1. Add a new server/openapi-accepted-breaks/<pr-or-slug>.txt file for each break. Choose a unique name for your pull request and change. Keep each entry in its own file, so separate pull requests can add entries without a shared-file conflict. The line is the message of the gate's annotation in CI: in API <method> <path> <change text>. The local report shows the same parts on two lines. Above it, write a # comment with the reason and the pull request.
2. Write a changeset that tells API clients what changed. See Writing changesets.
   • Before 1.0, use minor.
   • From 1.0, use major. The break must obey the version numbers and the deprecation policy of the compatibility policy.

The diff of each new file shows its accepted break in review. The gate reads all .txt files in the directory as one set. It joins them into a temporary ignore file for oasdiff. A duplicate entry accepts no additional break. The existing legacy-* filenames stay fixed. New entries do not continue their sequence.

An entry is obsolete when no break against the current base has that line. This occurs when the release that contains the break is published, because that release becomes the base. The gate then shows one warning that lists them. Remove their entry files in your next API change. They do not fail the gate, so a published release does not stop unrelated pull requests. The Version PR cannot remove them, because its own runs still compare with the previous release.

A published release moves the base, so a run that passed can fail when it runs again, for example in the merge queue. This occurs when main removes an endpoint that the new base added. List the break or restore the endpoint, then run the job again.

Merge policy

Pull requests targeting main merge through GitHub's merge queue after the GitHub Actions context CI Status Gate passes. Its prerequisites include all workflow lint checks. Required workflows run again for the merge group, which includes the current base and queued changes ahead of the pull request. Each required context is bound to the GitHub Actions integration rather than accepting a same-named external status.

A merge group is a diff, so it selects its jobs from the paths it touches — Path filters. It has no preview label, so Build and Images are filtered there too. any-code is the union of source filters and reaches docs/** through the tooling filter. Thus, a documentation-only group starts Quality, Security, Build, and Images, even when their internal jobs skip.

Read the CI Status Gate summary for each prerequisite's result. It lists every leg, including skipped legs. Read the Changes log for filters matched by the diff.

vp run pr:wait <number> uses the latest run number for each workflow, event, and app. A superseded run on the same commit does not replace the current result. The watcher reports cancellations separately from failures.

When a new event supersedes a pull-request run, its gate uses GitHub's native cancellation. It does not evaluate cancelled prerequisites as a failed validation.

Test report publishers stop on cancellation. A passing job must produce test reports. A job that already failed can lack them. The test command still owns its native failure result. Merge-group gates still evaluate after a fail-fast cancel, so a real worker failure blocks the queue.

A cancelled required check needs action. On a base with required checks, an optional cancellation does not block readiness. On a base without required checks, every reported check must pass. Checks with the same required name in different workflows or apps must all pass.

The watcher waits while GitHub reports BLOCKED or UNKNOWN, even when every required check passes. If a required check is absent, inspect the run before you rerun its gate. All CI Passed is a separate status. The ruleset still requires CI Status Gate.

A required context may be filtered because a ruleset accepts a skipped check as satisfied. The check still reports skipped, not success. A workflow that never triggers reports nothing. A pending required context blocks the queue indefinitely. merge_group supports no paths: filter, so each job's if supplies the filter.

The repository ruleset controls how many merge groups the queue builds in parallel through max_entries_to_build. Read the active value in the ruleset's merge queue configuration. Each group starts a CI/CD run. GitHub's runner concurrency limits apply across workflows, so queue parallelism competes with pull-request validation for capacity.

A failed merge group drops its pull requests from the queue without failing their checks. Thus, CI updates a comment with the run link on the pull request named by the merge group's head ref. It does not requeue automatically: the result does not say whether the failure was transient or deterministic. Regrouping the queue cancels the runs it dissolves, and a canceled run is reported to nobody. A failed group also cancels its own run, but its gate verdict stays failure, so CI reports it.

Review is a native ruleset requirement with stale approvals dismissed on every push. Authors listed in REVIEW_POLICY_MAINTAINERS receive an automated approval. Every other author needs a human with write access, including Renovate. Green checks and Dependency Dashboard approval do not satisfy this review requirement. Renovate includes that reminder in each pull request's body.

Repository administrators own the allow-list. Do not add dependency bots to it. Remove the automation when independent maintainer review is available.

After approving the current commit, the workflow attempts to minimize superseded automatic reviews. A minimization failure is a warning, not a failed approval.

The approval job runs in pull-request-policy.yml with pull-requests: write, as Pull request policy describes. It loads the evaluator from scripts/ on the default branch, reads current pull-request metadata and reviews from GitHub, and executes no pull-request code. An empty allow-list approves nobody.

It has no branches: filter. A stacked layer gets approval while it still targets the layer below. After that layer merges, retargeting to main must not depend on another event. The edited trigger also re-evaluates explicit base changes.

The stacked trigger re-evaluates a layer when it joins a stack. GitHub applies the stack base's review rules to each layer. An earlier approval can remain APPROVED on the same commit while GitHub requires a new review in that context. The policy checks GitHub's current review decision before it reuses an automatic approval. It posts a fresh approval for a listed maintainer when GitHub reports REVIEW_REQUIRED.

Pull request policy

The title, commit identity and attribution checks, the maintainer approval and the author assignment need write access to the pull request. They do not run on pull_request_target. GitHub blocks that event by default in public repositories from 2026-12-02 (GitHub changelog). Instead, two workflows split the work, as GitHub Security Lab recommends (Preventing pwn requests):

1. pull-request.yml runs on pull_request, with the pull request's own workflow files and no permissions. It records the pull request number and the event action as the pull-request artifact. It checks out nothing.
2. pull-request-policy.yml runs on workflow_run when that run succeeds. GitHub runs it from the default branch, with the permissions that each job declares. It checks out only the default branch's policy files and runs no pull-request code.

The artifact is untrusted data. scripts/pull-request-policy.ts accepts only a number and a known action. It reads the pull request from the API and acts only while the pull request's head is the commit that the recording run tested. A run for a replaced head does nothing, and the newer run decides.

First-time fork contributors need a maintainer to approve the recording run. A merge conflict prevents pull_request runs, so policy feedback waits until the conflict is resolved. A record can point only at a pull request with the tested head commit. It cannot change the policy or what the policy reads. The judging job checks the head on every page of commits and again before it posts the verdict.

A workflow_run job reports on the default branch, not on the pull request. Thus, the judging job posts its verdict on the pull request's head commit as the Validate PR check, with the Checks API. The ruleset does not require Validate PR or the approval job. The native review requirement needs the approval, so a maintainer's pull request cannot merge without this workflow.

Zizmor reports every pull_request_target and workflow_run trigger as dangerous-triggers, and the Zizmor job fails on any finding. pull-request-policy.yml carries an ignore comment with its reason. scripts/ci-contract.test.ts also refuses any workflow that runs on pull_request_target, so an ignore comment cannot admit one.

A merge group uses a synthetic ref, not the Version PR branch name. If its aggregate tree changes package.json's version relative to the group's base, it is a release candidate. It builds every release image on both platforms and must pass release evidence preflight on that exact tree. Missing base/version data fails detection rather than silently skipping release validation.

Each successful push run of CI/CD on main starts version-pr.yml. It versions the newest first-parent main commit whose push run succeeded, not the tip. main takes a merge more often than CI/CD completes, so the tip is rarely validated when a run ends. Thus, only a validated commit changes the Version PR, and maintenance does not wait for a quiet main.

The run makes no change when the Version PR already versions that commit. It also makes no change when main released a newer version after that commit, because the release consumed its changesets. It makes no change when the Version PR already versions a newer validated commit. It checks the release and the branch again just before it moves the branch.

scripts/maintain-version-pr.ts runs from the tip of main and versions the selected commit in a separate worktree, without a token. It makes the commit on a staging branch, then moves the Version PR's branch to it. Resetting the branch to a main commit first would make GitHub mark the pull request merged.

A failed maintenance run opens or comments on the Version PR maintenance failed issue, and the next successful run closes it. A timeout or a manual cancel of a started run does the same. A waiting run that the concurrency group replaces reports nothing, because the newer run does the same work. A stale Version PR alone gives no signal.

The Version PR carries checks like any other pull request. Its updates use GITHUB_TOKEN. GitHub creates approval-required workflow runs for pull_request opened, synchronize and reopened events triggered by that token. Nothing dispatches CI/CD onto the Version PR automatically.

When GitHub requests approval for the pull request's run, approve that run. If a run fails, rerun its failed jobs. Nothing retries a failure automatically.

Maintaining an unchanged head need not start another run. If the current head has no run, or only cancelled runs, dispatch one deliberately:

gh workflow run cicd.yml --ref changeset-release/main -f release-preflight=true

A cancelled run cannot authorize a release. The pull request's run and a dispatch on its branch share one concurrency lane, so a newer run cancels an active one. Dispatch only when no run for the head is active or awaiting approval.

Either run reports CI Status Gate on the branch head, which is the pull request's head commit, and requires its own release evidence preflight. The preflight also runs the verifiers of the earlier releases that publish-release runs, against the head's lock (Release Management § Joint lock with the chart). Thus, validation precedes the merge that cuts a release.

Verify changesets is the one leg it skips, and deliberately: the Version PR is the pull request that consumes changesets rather than adding one. Release management.

Fail fast in the merge queue

The queue removes an entry only when a required check fails. CI Status Gate needs every leg, so it cannot fail before the slowest leg ends. Two rules end a failed merge group early and free its runners. Pull requests keep independent legs running after failure. The shared Security scan is a prerequisite on every event, so its failure blocks dependent work.

A failed server cancels the run. Each build, test and quality server has a dependent cancel job that runs if the server fails in a merge group. That job needs only its server, so it starts without waiting for other servers in the workflow. Each server has a static name in its stage workflow. Shared step sequences use YAML aliases, so each skipped server still has a readable name. Every cancel job is named Stop the merge group early.

• The gate waits for every stage, and none of its prerequisite lists include cancel jobs.
• Changesets runs only in pull requests, so it has no cancel job.

After the cancel, cancelled() is true for the whole run. Thus, Report unsuccessful merge group reads the gate's verdict output, not cancelled(). A leg can also fail while a regroup cancels the run. Thus, the reporter comments only after the queue dropped the entry, and it waits up to two minutes for that. A run that cancelled itself after a failed leg has the verdict failure, so the reporter is meant to comment on the pull request. A run that a regroup cancelled has the verdict cancelled, and CI does not comment.

The merge-queue experiment did not exercise the reporter after a cancel. The first failed merge group on main confirms it. The gate does not publish the All CI Passed status after a cancel. No ruleset requires that status.

The cheap legs run first. The heavy legs start only after Security / Dependencies, secrets and policy succeeded or skipped, in every event. The heavy legs are Server package, Quality, Server tests and Images. Build, Kubernetes and Release evidence preflight wait through Server package or Images. Thus, an advisory against the whole base ends the group before a heavy leg takes a runner. Workflow lint, Image advisories and Pinned upstream images run beside Security.

One Security job calls the scan in every event. The image advisory snapshot and pinned upstream image scan remain independent of that prerequisite.

Never force-cancel a merge-group run. A force-cancel skips the always() jobs, so CI Status Gate never reports. The queue then keeps the entry with no verdict until its check timeout. Use only the graceful gh run cancel command in the isolated cancel servers.

Only the cancel jobs can write to Actions. The actions: write permission also lets a job dispatch workflows, rerun runs and delete caches. Reusable-workflow callers forward this permission as a ceiling to their stage workflows. A call job sets a permission ceiling. Each build and test server sets its own narrower permissions. Only each isolated cancel server receives a token that can cancel runs.

That job runs on ubuntu-slim, with no checkout and no action, so no other code holds the token. Each workflow keeps permissions: {} at the top level. ACTIONS_WRITE_GRANTS in scripts/ci-contract.test.ts lists each other job with this permission, and the reason.

The cancel job needs a runner, so a busy runner pool adds its wait to the eviction time.

Eviction time. In the merge-queue experiment, the time from the first failed leg to the removal of the entry was 1 minute 27 seconds. With a busy runner pool it was 4 minutes 9 seconds, because the cancel job waited for a runner. Without the cancel jobs it was about 6 minutes, and on main the gate failed 13 to 19 minutes after the first failed leg. The time on main is not measured yet. Record it here from the first merge group that fails after this rule reached main.

scripts/ci-contract.test.ts holds each rule:

• Each leg that runs in a merge group has its cancel job, for merge groups only. The job's condition has always() and the leg's failure result.
• No workflow, action or script force-cancels a run.
• Only the cancel jobs, and the listed exceptions, have actions: write.
• The heavy legs wait for the same Security scan in every event.
• The reporter reads the gate's verdict output.

Security and supply-chain controls

Supply-chain security in CI lists each control, when it runs, what it blocks, and where it reports.

Gradle dependency inventory and the Vulnerabilities exclusion

Supply-chain security in CI owns this exclusion.

Environments

Environment	Eligibility	Deploys on	Approval
Staging	qualified commits of main	automatic promotion policy	none
Production	a published release	manual promotion	required reviewer

After each successful main push, continuous-staging.yml dispatches one automatic Release / Promote run for Staging and ends. That run decides, signs, publishes and observes the promotion in one job, so a promotion waits for one runner. Its conclusion and the Staging deployment record report the outcome. GitHub notifies nobody of a run that GITHUB_TOKEN dispatched. Thus, a failed automatic promotion opens the Staging did not follow main issue, or comments on it while it is open. Close it after triage.

GitHub environment setup

1. Settings → Environments → New environment
2. Create Staging. Limit its deployment branches to protected branches.
3. Create Production with Required reviewers. Limit its deployment branches to protected branches.
4. Do not add a custom deployment protection rule. promote.yml records its own deployment and sets deployment: false. A custom rule needs the record that this setting stops, so the job would fail.

The reconciler applies one Helm chart. The chart's migration Job applies the schema. The server role waits until the database holds its change sets. Helm does not enforce a stopped cutover: Recover a stopped cutover gives that procedure. The one-host platform owns ports 80 and 443.

The preview label

The label builds images only. There is no deployed preview until previews move to the staging cluster.

On a same-repository pull request, the label makes CI build, scan and publish every image at the head commit. Without it, CI builds a pull request's images only for the Kubernetes cluster jobs or release evidence. The merge group builds every other changed image before it lands. Thus, the label shows an image build or scan failure on the pull request, before the merge queue finds it. Changes reads the label live, as for ci:full (Two verification tiers). If it cannot read the label, a pull request that is not a draft builds its images.

If you add the label after the run started, use Re-run all jobs on that run, or push a commit. Re-run failed jobs keeps the earlier Changes result, so it does not see the label.

Images rebuilds images whose inputs changed. Its Tag unchanged images job aliases the base commit's verified digests for the rest. Thus, every image exists at the head SHA.

An alias is never release evidence. If the base commit's image is still publishing, it uses the merged pull request's candidate, which carries only linux/amd64. Either way, the alias is an image that another commit's run built. That is why the Version PR's branch builds every image itself instead.

A draft with the label also builds its images.

Workflows

Each display name starts with its area: CI, Release, Security, Docs, Cache or Repo, as in Release / Promote. The Actions sidebar sorts by name, so it groups each area. A workflow that a workflow_run trigger names keeps its name, because the trigger matches the display name. So cicd.yml stays CI/CD. The required checks are job names, so a display name does not change them.

Workflow	Trigger	Purpose
cicd.yml	PR, merge group, push to main, manual	Validates changes before merge and publishes final-SHA images after merge. The release evidence gate runs over the images the run built on the Version PR's branch, and wherever the release-preflight dispatch input asks for it
pull-request.yml	PR opened, edited, synchronized, reopened, ready for review	Records the pull request number and event action as an artifact, with no permissions. Checks out and runs nothing
pull-request-policy.yml	pull-request.yml completed successfully	From the default branch: validates the title against the exported policy in commitlint.config.ts, applies the commit identity and attribution policies, and posts the Validate PR check. Approves a listed maintainer's pull request. Assigns the author when the pull request opens
ci-quality-gates.yml	Called by cicd.yml	Selects quality legs, clean-install verification and story tests
cache-toolchain.yml	Main push changing install inputs or setup	Seeds trusted Linux and Windows install caches without repeating quality gates
cache-gradle.yml	Main push changing Gradle inputs, or dispatch	The only Gradle cache producer: builds what the server jobs restore
ci-browser-tests.yml	Called by cicd.yml	Webapp and extension browser suites from the packaged server
ci-build.yml	Called by cicd.yml	Runs the OpenAPI, schema, database and PostgreSQL drill gates against the server artifact packaged by cicd.yml
ci-tests.yml	Called by cicd.yml	The server unit, architecture and integration suites, on the package job's compiled classes
reusable-docker-build.yml	Called by cicd.yml	Builds every requested image in one job per platform, natively, and blocks on the release vulnerability policy there. One job merges each index, attests, signs and tags it
ci-security-scan.yml	Called by cicd.yml	Trivy dependency scan, secret detection and repository policy checks
dependency-review.yml	PR changing a dependency input	Reports introduced dependency advisories independently of the required CI verdict
dependency-graph.yml	PR changing a dependency input, push to main	Resolves and uploads the verified Gradle dependency graph with read-only permissions
submit-dependency-graph.yml	Successful dependency-graph workflow	Submits snapshot data without executing code from the originating run
verify-changesets.yml	Called by cicd.yml	Tests changeset and version-sync policy, enforces release-note presence, and requires a note that names each removed chart value
retarget.yml	PR edited so that its base becomes the default branch	Runs the head's CI/CD run again, so a stacked layer leaves the light tier. See Light tier for stacked layers
openapi-autocommit.yml	autocommit-openapi label	Regenerates the OpenAPI spec and client and commits them to the pull request
security-mutation.yml	PR touching the suite's inputs, manual	Runs the security mutation suite
practice-guidance-links.yml	PR touching practice guidance, weekly, manual	Checks the web links in bundled practice guides
scorecard.yml	Push to main, branch-protection change, weekly	Publishes the OpenSSF Scorecard
semgrep.yml	Pull request, push to main	Tests and scans the repository's own TLS rule pack
kubernetes-nightly.yml	Nightly, manual	Runs every Kubernetes leg and the sandbox footprint benchmark on the newest main build that passed CI, with its chart and agent image. Reports a failure on one tracking issue
extension-provider-slots.yml	Nightly, manual	Checks the actual extension adapter against six public GitHub and GitLab report layouts and keeps each result
platform-pins.yml	PR touching platform pins or their validator	Verifies the installer checksum and the images that both upstream charts render
rescan-images.yml	Daily, manual	Rescans main's published images against the release vulnerability policy and keeps one tracking issue in step with the findings. A second job rescans the latest published release's images and reports drift on the vulnerability response issue
ci-diagnostics.yml	Completed CI/CD runs	Reports exact-attempt job and step timings without changing the required CI verdict
scheduled-failures.yml	Completed scheduled workflows	Notifies maintainers once per failed attempt and covers unavailable specific reporters
ci-profile.yml	Weekly, manual	Reports advisory performance budgets and profiles server test tiers
delivery-metrics.yml	Weekly, manual	Reports lead time, deployment frequency, and change fail, recovery and rework proxies. See Delivery Metrics
ci-server-clean-reference.yml	Weekly, manual	Records cold server phases and compares generated JARs
scorecard-ratchet.yml	Push to main, branch-protection change, weekly, manual	Fails when a Scorecard check drops below the committed baseline. A push fails only for the checks its own commit decides
cd-docs.yml, cd-docs-teardown.yml	docs/** on push and pull requests. A published release. PR closed	Publishes the docs site, the latest release at / and main at /next/ (Versions), and its pull-request previews
version-pr.yml	Successful CI/CD push run on main	Maintains the accumulating Version PR from the newest validated main commit, and tracks failures on one issue. Its own pull-request CI/CD run, release evidence preflight included, validates it
release.yml	Successful CI/CD push on main	Requires empty architecture ratchets for the planned stable version before publication, then promotes the moving series and latest image tags
release-upgrade.yml	Called by release.yml, by a merge group that changes a migration, weekly	Boots the previous stable release with seeded data, then the candidate against the same database. See Seeded release upgrade
promote.yml	Manual dispatch, or automatically for staging	Records the GitHub deployment of the promoted source and signs the channel a host follows. Hosts pull and apply it themselves

Commits a workflow makes

Actions holds no signing key, so a commit pushed with git from a workflow is unsigned and a required-signature rule rejects it. Every automated commit here uses GitHub's createCommitOnBranch mutation, which GitHub signs. Renovate uses the same path for dependency bumps.

scripts/commit-via-api.ts holds the one implementation. openapi-autocommit.yml commits the regenerated spec and client to the pull request's branch through it. promote.yml commits the channel to deploy-state, and scripts/maintain-version-pr.ts commits the Version PR. scripts/ci-contract.test.ts fails any workflow that pushes to a real ref.

The mutation carries whole file contents, not a diff. Thus, the script enumerates every added, modified, or deleted work-tree path and sends each complete file. A rename is its source deleted and its destination added. It commits against the head used for the work-tree comparison. If the branch has moved, the run fails instead of overwriting the new changes. The credential is still GITHUB_TOKEN, so these commits start no further workflow run.

Shared setup actions

Action	Contract
setup-toolchain	Installs the Node.js version pinned in package.json. With install set to none, it installs nothing else, so node scripts/… jobs reach no package registry. frozen adds pinned pnpm and restores its store cache. It caches the pnpm executable by platform and version, with two download retries on a miss. After the frozen install, it checks that the installed vite-plus is the version that package.json pins. Then it puts the lockfile's vp on PATH. Only the clean-install job installs the global launcher
setup-caches	Sets up the JDK from .java-version, validates the wrapper and uses the official Gradle action for dependency and build caches. The trusted package job publishes caches. Consumers are read-only
restore-server-build	Downloads and validates the packaged JARs, compiled tests and resources. Outputs the executable JAR path without recompiling or installing local artifacts
setup-browsers	Pulls the digest-pinned Playwright image that matches the package version
setup-release-security-tools	Installs Cosign, Trivy, and optionally Syft for release-evidence jobs
download-trivy-db	Downloads with retries and mirror fallback, or restores the CI run's immutable advisory artifact. Checks freshness at consumption and records metadata as requested — database identity owns the rerun contract
ghcr-login	Authenticates Docker to GHCR

Browser commands run through node scripts/ci-browser.ts <command> [arguments...] in the official Playwright image. .github/playwright-image.json pins its digest and version. The version must match webapp/package.json. The image supplies Chromium and its system libraries. The runner supplies the pinned Node.js binary and the installed dependencies.

Host networking lets tests reach the packaged JAR and PostgreSQL through their existing loopback URLs. The container runs as the runner user and writes reports into the checkout. See Playwright's Docker guidance for the image, --init, and shared memory.

CI jobs do not install system packages. Tooling jobs install ripgrep from the release archive in .github/ci-tools/ripgrep.json. The shared resolver checks its SHA-256 before the installer puts rg in /usr/local/bin, on the tools server's fixed PATH.

Repository-local actions require checkout first. Jobs without checkout use the external SHA-pinned action directly rather than checking out the repository only to reach a wrapper. The reusable image builder also uses the external login action because its registry is a caller input, whereas ghcr-login intentionally authenticates only to GHCR.

Browser E2E boots the packaged JAR without Gradle, so it sets up Java directly. OpenAPI autocommit builds the JAR and uses setup-caches.

Commit, branch and pull request image tags are published only after their immutable digest passes signature and provenance verification. Index assembly uses a run-specific staging tag. Final tag publication preserves the verified digest.

Image verification stays beside publishing. The merge job signs recursively, then verifies the index. The alias job verifies each immutable subject before moving its tags. Merge and alias jobs need no checkout or repository toolchain just to verify registry evidence. Provenance verification reads attestation bundles from the image registry and keeps the existing identity and predicate policy.

scripts/ci-provenance.test.ts executes all three shell paths and pins the same repository/workflow signature identity, GitHub OIDC issuer and owner-constrained attestation verification, including rejection before alias publication. Signing remains in reusable-docker-build.yml, so a refactor cannot silently change the certificate identity.

Caches

A run restores caches from its own branch and from main, never from a sibling branch. Task-cache entries are content-addressed, so a pull request saves them for its own reruns.

cache-gradle.yml is the only Gradle cache producer. Every other job is a read-only consumer. The producer runs after a main push changes a Gradle input. The inputs are the build scripts, the lockfiles, the version catalog, the wrapper, the generated-client sources and the JDK pin.

The producer resolves every dependency configuration of the server build. It also runs the Gradle work of the server jobs: packaging, the test classes, Spotless, PMD and a test inventory that does not execute tests. The cache keeps raw POM and module metadata with the resolved binaries. Consumers need those files for dependency verification, even when a warm build uses binary metadata. The action does not prune files after each job.

A main push does not build the server, because it reuses the merge group's packaged server (§ Build once). Thus, without this producer, nothing would refresh the dependencies that consumers restore. Consumers would download them from Maven Central in every job.

The Gradle action prefers an older entry of the same job to a newer entry of another job (restore order). Thus, a second writer would pin consumers to its stale snapshot, even if it writes only when reuse fails.

The pnpm executable cache also saves only from main. The JDK is pinned once, in .java-version.

The package store and pnpm's lockfile-verification cache are separate. setup-toolchain also restores lockfile-verified.jsonl from the directory reported by pnpm cache path. Without it, a warm store still needs registry metadata for release-age and resolution checks. Only successful default-branch push installs publish this record, keyed by platform, pnpm version, lockfile, workspace policy and any repository .npmrc. pnpm's native verification cache validates the content and policy before reusing a verdict.

The frozen install still runs with scripts disabled and package integrity checks intact. A miss performs normal verification, not an offline fallback.

The action verifies the native cache file exists on each runner, including Windows. A changed upstream cache contract must be reviewed rather than silently losing reuse.

Registry metadata itself is not cached by this action. A dependency update is cold under its new key. Only later runs with the same inputs can reuse the default branch's verdict.

cache-toolchain.yml performs a frozen install on Linux and Windows after a main-branch push changes these inputs or the setup workflow/action. Ordinary main CI skips quality legs, so this dedicated producer keeps both platforms warm without duplicating their gates. It is separate from release validation: a failed or expired cache is a normal cold install, not a reason to block a release.

Rerunning the original main-push cache workflow can replenish an expired entry. Manual dispatch and pull requests cannot publish verification verdicts.

Java quality verdicts are not stored in the Vite task cache: Gradle owns their task inputs and outputs. GraphQL generation opts into Gradle's build cache using the plugin's declared inputs and outputs. Its optional timestamped @Generated class marker is disabled so unchanged schemas produce identical sources and compilation can also be restored. Test tasks always execute. Their results are not cached. The local verification contract describes the shared local/CI command and how to distinguish Vite, Gradle build and Gradle configuration caches.

Gradle's official action explicitly uses its enhanced cache provider. Its proprietary caching helper is free for public repositories. It retains dependency fallback and deduplicated build outputs. The OSS basic provider's immutable, exact-match cache cannot refresh build outputs after source-only changes. Configuration-cache reuse remains within a job: no encryption key is configured, so the action does not persist configuration-cache entries across jobs.

Two verification tiers

A pull request outside the light tier uses the fast tier. A merge group, a push and a dispatch also enable the queue tier. detect-changes publishes the decision once as queue-tier, and each queue-tier job adds it to its path condition. The numbered integration shards require the queue tier. The review shard does not.

The table lists eligible jobs. Path filters and event guards still select which jobs run.

Tier	Eligible jobs
Fast (pull request and queue)	Changes, Quality, Unit, Architecture (migration exception), review, packaging, generated-artifact drift, images, Security, Release notes, Workflow lint
Queue (merge group, push, dispatch)	Numbered integration shards, Webapp and Extension browser suites, Database migrations, PostgreSQL drills (migration exception)

Path filters and the main-push rules still select within each tier. For PRs and merge groups, the application-server or test-config filter must match for server-tests to select server suites. The light tier skips those suites. Thus, a docs-only diff that matches neither filter runs no review or numbered integration shard. Database tests, drills and browser suites have their own path conditions.

Kubernetes qualification stays on same-repository pull requests, because it boots this run's own images. A draft does not select it.

The kubernetes path filter selects the cluster legs for chart, platform, sandbox, gateway and agent-image changes. A selected ordinary PR runs the latest-minor amd64 install, upgrade, isolation probes and adapter qualification. It also runs the replacement-storage recovery drill when its inputs change.

The queue tier and nightly runs add Kubernetes / k3s 1.37 amd64 drills on a separate cluster. It starts alongside the install/qualification job after the images are ready. The caller needs only the image producer and change metadata. The image producer already waits for the packaged server.

The merge group runs the conversation roll, which installs the release and rolls the server and loop. Eviction runs nightly, on release and on a CI dispatch with eviction-drill=true. It runs no eviction on a PR or merge group. The nightly and dispatch clusters run roll and eviction as native background steps, and wait-all requires both results. Eviction fixtures use their own namespace and a tainted disposable node. The qualification cluster owns chart lint, upgrade checks, adapter probes, recovery and new-node image prepull checks, so the drill cluster does not repeat them.

If this split increases the measured merge-group verdict median or p90, revert the split or move the roll to nightly. A failed roll fails the Kubernetes stage and blocks the merge group through CI Status Gate. Use ci:full to reproduce the queue legs on a PR, including the separate roll job, lowest-minor k3s and Calico.

The nightly run adds arm64 and eviction. A failed nightly eviction fails the Kubernetes stage. Its existing Report a failure job opens or updates the issue "The nightly Kubernetes run failed" and links the run. The release evidence preflight also requires that the newest nightly run on main passed its cluster legs within 48 hours (scripts/check-nightly-kubernetes.ts).

The release also runs eviction against its captured candidate images before final publication. Release qualification and recovery defines its gate and failure recovery.

The review shard holds ReviewEndToEndIntegrationTest: one review of a real pull request, from its request to its admitted results. Only the sandbox container and the model provider are stubbed. It runs when the server suites are selected under the tier and path rules above. A change to capture, the gateway, the proxy or admission can stop every review while each unit test still passes. The stub fixes the model's judgement, its read receipts and its coverage, so the test does not check how the runner derives them.

On a selected PR, adapter qualification deliberately exceeds a review container's memory limit through the production sandbox adapter. It requires exit code 137 and the recorded oom_killed reason, with no evicted increment.

Another drill uses a tainted, disposable node and a raised soft memory threshold to exercise the kubelet without exhausting the runner. Three synthetic Pods stay below equal memory requests and use the chart's priorities. Creation order and memory use oppose the expected eviction order. The drill requires memory evictions in review, conversation, then role order, with no OOMKilled container. This checks priority within one request-usage group. It does not qualify loaded node capacity or enable soft eviction in the installation.

Main changes only through the merge queue, and the queue runs the required checks on the merged result (GitHub: managing a merge queue). bors and Prow's Tide apply the same rule. They test the merge candidate and advance the branch only when it passes (Rust forge: bors and rollups, Tide). The merge group runs the selected queue-tier checks before main advances. A PR normally defers those checks. ci:full and the migration exception select them earlier.

Google's presubmit applies the same split. It runs only fast, reliable tests before submission and the larger tests after (Software Engineering at Google, ch. 23).

When both tiers ran on each pull request, a merged change paid for the queue tier twice. The queue tier is most of a run's runner time, and GitHub's runner concurrency limit is shared by the queue and every pull request.

A queue-tier failure removes the entry from the queue, and the failure comment links the run. So does an image build or scan failure, because a pull request builds images only for the preview label, Kubernetes qualification or release evidence. Fix the failure and queue the pull request again.

A draft pull request keeps its selected tier. Without preview or release evidence, it builds no image and gets no cluster qualification or image scan. The image selection rules apply to drafts too.

ci:full selects the queue tier on a draft too. CI Status Gate still reports, so required checks resolve on a draft. Marking the pull request ready for review (ready_for_review) starts an ordinary pull-request run, with the images that such a run builds (The preview label). That run is never skipped as a duplicate of the draft's run.

A re-run replays the event of its run, including the draft state. Re-run the ready run to use its ready state.

Changes to workflows, build scripts, shard durations or browser tests do not select the full queue tier on a pull request. The merge group checks the complete selected queue tier before the change reaches main. Release evidence selection stays separate. A Version PR does not select the full queue tier unless it carries ci:full.

Migration checks on PRs

A PR outside the light tier that changes db/** runs database tests and PostgreSQL drills, including on a draft. The light tier defers these checks to the merge group. If it changes only server/application/src/main/resources/db/** and its release or schema notes, it runs review without Unit or Architecture. The notes can be changesets, .migration/*.md files or generated ERD files. Generated-artifact drift, quality and security checks stay selected. A migration with a Java, build or configuration change keeps Unit and Architecture too.

The migration-specific integration classes remain in the numbered shards. They first run in the merge group, where a failure removes the group from the queue. The Database tier and review do not replace those classes. An explicit full run checks them on the PR too.

Reproduce a queue-tier failure

Use ci:full to reproduce a queue-tier failure on the pull request. It enables the queue tier but keeps path selection. It is not the default for a workflow or migration change. Remove the label after the reproduction. Then push, or use Re-run all jobs on the latest CI/CD run.

Re-run failed jobs keeps the earlier Changes result, so it does not see the label. Changes reads the label from the pull request when it runs, not from the event. If the full label output is missing, the queue tier runs.

There is no labeled trigger, because a label event would cancel the running CI. A run whose jobs all skip would also report CI Status Gate as passed on the head commit. That report would replace the verdict of the run that it cancelled. Locally, these commands run the queue tier's suites:

• vp run test:server:integration for the integration shards.
• The database contract tests: from server/, ./gradlew :application:databaseTest with the SPRING_DATASOURCE_* settings of a running PostgreSQL (server/AGENTS.md § Test tiers).
• vp run test:postgres-restore and vp run test:postgres-pitr for the drills.
• vp run --filter webapp test:e2e and vp run test:extension:e2e for the browser suites.

Revert criterion. report:ci-latency reports the share of merge groups in the last 14 days in which a queue-tier job failed. It counts the groups in which only fast-tier jobs failed apart, and it gives no share for fewer than 20 groups. If the queue-tier share exceeds 10 %, the queue tier is catching too much that a pull request should have caught. Then run the queue tier on pull requests again: change queue-tier in detect-changes to true for every event.

Light tier for stacked layers

A layer of a stack is a pull request whose base is another pull request's branch. It never merges on its own: it enters the queue as part of its stack. The merge group runs the checks selected for the combined diff.

Thus, such a layer runs the light tier: Changes, the quality legs except Windows and Webapp stories, Security, Release notes and Workflow lint. These are format, lint, types, prose and the quality legs' own tests. It packages no server, so it runs no server suite, no artifact gate, no image build and no cluster leg. CI Status Gate evaluates the selected checks.

Changes reads the base from the pull request when it runs, not from the event, and publishes light-tier. The ci:full label runs the full tier on a layer too. A release candidate leaves the light tier and builds its release evidence. An unreadable pull request also runs the full tier.

When the layer below merges and the layer is retargeted to the default branch, CI / Retarget runs the head's CI/CD run again. The new attempt reads the new base and runs the ordinary pull-request tier. A re-run is never skipped as a duplicate, because the earlier run of the same tree may have run only the light tier. A push to the retargeted pull request runs the ordinary pull-request tier as well.

Path filters

The kubernetes filter includes docker/**, application*.yml, *Properties.java and Application.java. ci-contract.test.ts checks that it covers every configuration-binding source.

detect-changes in cicd.yml selects legs from changed paths for two diff events. A pull request compares with its base. A merge group compares merge_group.base_sha..head_sha. A push to main and a dispatch publish unfiltered selection flags rather than filtering a diff.

Selection does not override outer event guards. Ordinary main pushes omit Quality, Server tests, Security and Workflow lint. They skip API, schema, database, and browser jobs inside Build.

Release-version pushes repeat those checks. Main still packages and publishes images.

A same-repository pull request with the preview label, or with Kubernetes qualification, also runs Build and Images regardless of paths. The label and the cluster tests need every image at the head SHA. No other event reads the label. Any other pull request builds no image. The merge group builds every changed image before it lands.

Selects	Paths
Quality → Webapp	webapp/**, docs/images/readme/**, docs/src/css/custom.css, the root oxlint and oxfmt configs, the four scripts/check-* files it runs, root package files
Quality → Extension	extension/**. Webapp files imported by path: vocabulary registries, trace formatting, status definitions, sign-in providers, text helpers, and their imports. Webapp lint plugin and stylesheet. Root oxlint and oxfmt configs, scripts/ci-browser.ts, .github/playwright-image.json, and root package files
Quality → Server	server/** except AGENTS.md and CLAUDE.md, scripts/run-gradlew.ts, scripts/check-gradle-verification-metadata.ts, root package files
Quality → Tooling and docs	scripts/**, docker/agents/**, server/application/src/{main,test}/resources/agent/**, load-tests/**, docs/**, tsconfigs, oxlint and oxfmt configs, .changeset/**, commitlint.config.ts, instruction files, .github/renovate.json
Every quality leg	vite.config.ts, scripts/lib/push-gates.ts, .vite-hooks/**, .java-version, cicd.yml, ci-quality-gates.yml and the setup actions: the task graph, the hooks and the JDK decide every leg's verdict, so a change to one runs all of them
Build artifact gates	contracts: the Server paths, docker/postgres/**, or either generated client and its generator config (api-clients). E2E: webapp source and the server main sources. Extension E2E: extension/**, the vocabulary registries and the server main sources. Image: server/application/src/main/**, the Gradle build scripts, catalog, locks and wrapper, server/application/project.toml, server/generated-clients/**
Docker images	webapp-image: the webapp build inputs and non-test source. agent-images: docker/agents/**. postgres-image: docker/postgres/**
CI config	one filter per leg naming its own workflow files and actions. Any .github/workflows/** or .github/actions/** change runs Actionlint and Zizmor

docker/agents/** is in both the tooling and the agent-image filters: test:agents and typecheck:agents cover the practice script tree, and the same tree builds the agent image. test:agents also covers the agent runtime, so its resources select the tooling filter.

Docker layer caching

The Dockerfile images are targets of docker/docker-bake.hcl, built for one platform in one docker buildx bake call. Each imports <image>:cache-main-<platform> from ghcr.io. Only pushes to main and merge groups write it, with every intermediate layer. A pull request reuses those layers and writes nothing.

amd64 builds on ubuntu-24.04, arm64 on ubuntu-24.04-arm. No emulation. The Server image, built with Buildpacks, does not use this cache. Dockerfile builds always bypass the cache for the os-packages and exported runtime stages. OS package updates owns that contract and the local build commands.

Registry authentication

Image builds authenticate to ghcr.io with the workflow's GITHUB_TOKEN. Base images are pulled from Docker Hub anonymously, which GitHub-hosted runners may do without a rate limit. A job that reads a first-party image back — the scan, the preflight, the cluster tests — logs in as well, because packages: read alone buys nothing.

A pull request from a fork gets a read-only GITHUB_TOKEN, so cicd.yml passes publish: false to both image workflows. That build compiles the image and stops: no login, no push, no signature, no attestation, no policy scan and no alias moves. Forks therefore get no published images and no cluster qualification. The release path exercises their code once it is on main.

When release preflight is required, it owns image vulnerability scanning for both architectures. The image-build workflows omit their duplicate scans. Other published builds scan every platform they built. Each platform retains its own report and policy result. The final gate rejects a required preflight that is skipped, canceled or unsuccessful.

Each platform's build job scans the digest it pushed, on its own runner, before it hands that digest to the manifest job. The manifest job needs every build, so only scanned digests reach an index. Before publishing verified tags, it checks that the final index leaves equal those scanned subjects. A scan therefore costs no job and no runner queue wait of its own. It does not run in the manifest job, because that job holds the signing token and runs no repository code.

The platform jobs run independently, so one platform's refusal keeps the other platform's scan and evidence. A broken build thus lets its sibling finish, which costs a runner for that time.

The scan resolves the immutable image reference through scripts/resolve-image-scan-subject.ts, using the same platform resolver in scripts/lib/image-scan.ts as scheduled rescans. An index must contain the requested platform. A missing platform or malformed index fails instead of falling back to whichever architecture the scanner's host would choose. A single manifest is resolved through that same immutable reference.

Image identity

Every consumer resolves an image through its build commit. reusable-docker-build.yml publishes exactly that tag for the run's start event. Pushes, dispatches, and merge groups use github.sha. A push that reuses its merge group's image publishes the same digest under that tag. Pull requests use their head SHA because github.sha is a merge commit with no image tag.

Within a run, the vulnerability scan ignores tags. It reads the digest that its own build job exported. Re-running a failed build job builds and scans the platform again, and the manifest job reads that new digest.

ci-contract.test.ts asserts the whole contract. The build must publish every consumer-resolved tag for that consumer's event. No job resolves an image by the run that built it. A run-scoped tag names the attempt rather than the artifact, and GitHub's re-run failed jobs gives the retried job a new attempt number.

Build once

Server package in cicd.yml packages the server once and uploads its JARs, compiled tests and resources. OpenAPI and browser checks boot that executable JAR. Buildpacks consume it without another build. Schema gates, database contract tests and the Server tests suites use the restored classes with -PpackagedServer=true, which bypasses compilation. ci-tests.yml sets it as ORG_GRADLE_PROJECT_packagedServer, because its commands are vp run tasks. Thus, every server suite tests the classes that the JAR ships.

The suites never compile. Test compilation is the slowest server build step, and it is rarely a cache hit. A change alters the test sources or the main classes they compile against. So the suites wait for packaging rather than compile the same sources in each of their jobs. The package artifact is kept for one day, so a failed job re-run after that day fails on its download. Re-run all jobs instead.

The package job compiles the tests without Error Prone and NullAway (-PtestAnalysis=false). The analysis can fail the build, but it does not change the bytecode that javac writes. Thus, the suites start sooner and still test the same classes.

The server quality leg compiles the tests with the analysis (gate:server-test-analysis), and a finding fails that leg. That leg runs whenever a Java compile input can change: server/**, .java-version or the setup actions. A push to main runs Quality only for a version bump. Its merge group ran the analysis on the same tree. Locally, every server test task compiles the tests with the analysis.

The PostgreSQL backup-restore and PITR drills apply the migrations from that same executable JAR. Spring Boot's launcher runs the Liquibase bundled in the JAR against the changelog in the JAR, from an empty directory and with a minimal environment. Thus, a drill restores the schema that the release ships, not the checkout's resources. In CI, a drill without the JAR fails before it starts a container. A local drill without HEPHAESTUS_APPLICATION_JAR migrates through Gradle from the checkout.

API and ERD drift checks use scripts/assert-generated-clean.ts against their own generated paths. It includes added, modified, deleted and staged files without staging anything or refreshing the Git index. Unrelated changes cannot contaminate either verdict. The API and schema steps still report independently, and a failure in either fails the artifact gate.

The application image and Build artifact gates depend on packaging independently. The Kubernetes cluster jobs and release preflight wait for the image producers, not the API, database or browser checks. The final CI gate still requires all of them. An early image verdict cannot approve a failed artifact gate.

A default-branch push reuses the packaged server when a successful merge group validated the exact same commit. The source must be this repository's cicd.yml run. Its artifact gates, tests, security checks, both platform image builds and final gate must have succeeded. Its Manifest job must have attested the Server image. Release preflight does not change the packaged Java artifact, so its absence in ordinary merge groups does not force a rebuild. Release candidates still require release preflight.

The latest attempt of each job counts, so an arm64 image build that flaked and passed on a re-run does not block reuse. A queue run whose arm64 build stays failed fails its gate, so its commit does not merge. main packages the server again when the merge group did not rebuild the Server image, because that queue run attested no Server image.

The immutable artifact ID and its SHA-256 digest bind the download to that run. Missing, expired or unavailable source evidence falls back to packaging. A corrupt download fails rather than being used. Main still runs its own validation and publishes its images under main-only tags. Release publication still generates fresh vulnerability evidence and signatures.

Merge-queue image reuse

A default-branch push reuses each image that its successful merge group published for the same commit. detect-changes resolves that run with scripts/resolve-queue-run.ts. The run must have a successful CI Status Gate, which covers its image builds and per-platform scans. The Manifest job in reusable-docker-build.yml accepts the image at <image>:<commit> only when all these checks pass:

• Its platform leaves are exactly the platforms that the push requests.
• Its cosign signature comes from reusable-docker-build.yml on the queue branch, with the merge_group trigger, at this commit.
• Its build provenance attestation names this commit and was made by that queue run.

The push then signs the index and each leaf again, verifies them, and moves the main tags forward. They stay when they already name a newer commit. It does not rebuild, re-attest, or rescan the image. The release verifier accepts only the default-branch signing identity, so the push signs the image again. The provenance stays the queue run's, because that run built the image.

A keyless signature fetches a GitHub OIDC token, and that request can fail with a passing 5xx. The manifest job signs each image up to three times, because verification accepts any valid signature. scripts/keyless-sign.ts signs the release evidence and the staging channel. It retries a token 5xx and the two transparency-log failures that its tests name.

An image that the queue run did not publish is built and scanned as before. Examples are a direct push, an image that the merge group did not rebuild, and a lookup that failed. An image that the queue run published but that fails a check fails the push, which then publishes no image. Thus, main never ships a different image in place of one that its merge group built and tested.

After a transient failure, such as a registry or API timeout, Re-run failed jobs repeats the manifest job and the jobs that follow it. After a persistent failure, merge a new commit through the queue. Its merge group builds, scans and signs the images again, and its push publishes them for main, staging and the release.

resolve-queue-run.ts also lists the images whose attestation step succeeded in the queue run's Manifest job. The push reuses only those images, and its build jobs build the rest at once. When the queue run published every image, a push that is not a version bump builds and scans nothing. It then skips Server package and Image advisories. It runs Changes, Images / Manifest and CI Status Gate. Thus, it holds few of the runners that a waiting merge group needs.

A merge group adds one arm64 build job, which builds and scans that platform for every image that it rebuilds. The push reuses those images and no longer builds or scans them on either architecture. It still builds and scans both platforms of each image that the merge group did not rebuild, as before. The webapp records its branch at build time, so a merge group builds it with the default branch's name. Merge groups also write the Dockerfile layer cache, because the push no longer refreshes it for reused images.

An unchanged pull-request image is not rebuilt. Tag unchanged images verifies an earlier run's published digest and retags it to this run's head SHA. It aliases from the nearest default-branch commit found through the chain of bases. A stacked pull-request layer uses another branch's head, which published nothing. If no such default-branch commit exists, the run builds every image itself. detect-changes resolves that commit once, with scripts/resolve-alias-base.ts, and hands it to the image jobs as alias_base.

A pull request aliases only from a commit whose CI/CD push run has succeeded, because that run publishes the images under the commit's tag. Right after a merge, that run is still building, so the pull request builds every image itself.

Parallelism and concurrency

• Quality legs start from source. Server tests suites and Build gates start when Server package uploads its artifact. In every event, Quality and Server package start after Security: Fail fast in the merge queue.

• The unit and the architecture tier run in separate jobs, Unit and Architecture. The unit tier runs in two test JVMs, which is the cap that org.gradle.workers.max sets. In one job, its JVMs would share those two servers with the architecture tier. Architecture also runs vp run test:server:selection.

• Integration has the review shard and numbered shards. Integration shards says how the classes are split. Each shard runs in its own JVM and Testcontainers database. The static shard jobs run independently. No matrix cancels a sibling. A failed merge-group server calls the isolated cancel helper. CI Status Gate requires the aggregate result. Locally the tier runs whole. The weekly server profiles run the same shards.

• Pull requests build linux/amd64. Merge groups, pushes to main, dispatches, and every run on the Version PR's branch build linux/amd64 and linux/arm64. Both lists take the same build, merge, signing, attestation and per-platform scan path. Pull requests build only changed images and alias the rest. Merge groups build only changed images and alias nothing. A merge group that runs the Kubernetes legs builds every image. Those legs install every image under the group's commit. The Version PR's branch builds them all. detect-changes decides both once, as platforms and all-images, and hands them to every image build in the run. A run that builds every image also packages the server for the Server image. It does so even when its diff holds no code, such as a bot's pin bump in ci-kubernetes.yml. scripts/ci-image-set.test.ts evaluates the workflow's own conditions and path filters. It covers every event and every change of up to three representative files. It also covers a push that reuses part of its merge group's images. It covers a dispatch for the release preflight. It covers a fork, a draft with and without its preview label, and a skipped run. It also covers a failed package and a failed image build. A job that installs every image must run where its event needs it. It must not run while some image is not built, reused or aliased. No image may be both built and aliased, because the alias would replace the build's tag. The Kubernetes legs also wait for Server package, so a failed package is reported once.

• A Kubernetes cluster leg is named for its distribution, minor and architecture, such as k3s 1.37 amd64. Its check history thus survives an image update. The job summary names the pinned node image. Renovate updates the image within its minor. A new minor is a workflow change. Before cluster creation, each leg removes unused SDKs from its GitHub-hosted runner. scripts/prepare-kubernetes-runner.ts checks free space on Docker's filesystem before each cluster starts.

• Before its vulnerability scans, each image build job removes the same unused SDKs and logs the free space. Trivy copies each image layer to a temporary file after the builds have filled the root volume. Thus, scripts/prepare-image-scan-runner.ts gives the scans' temporary files the volume with the most free space.

• A pull request that matches the kubernetes filter runs the latest k3s on amd64. Its server first lints the chart and validates its schemas. The merge queue also runs the lowest supported minor on amd64 and kind 1.35 Calico. Only the nightly run adds the lowest supported minor on arm64. Runners limit the merge queue, so the queue does not wait for an arm64 runner. The kind leg removes kind's bind mounts over /sys/devices/virtual/dmi/id on each node, because with them no sandbox Pod with hostUsers: false can start. It tests the commit that main receives, so a pull request does not repeat them. A dispatch runs the same legs as the queue. On the latest minor on amd64, the separate queue/nightly drill job runs kubernetes-cluster.ts turn-roll. In this drill, a conversation turn runs bash in its conversation sandbox while the servers roll. The sandbox driver keeps the run lease and uses the gateway of the replacement server. The turn must end DONE on its first attempt in the same sandbox. The fake model provider of the capacity benchmark scripts the turn. arm64 runs only the lowest minor, the minor of the arm64 host. A push to main that a merge group tested runs no leg, because its commit is the group's commit. A push with no merge-group run, such as a direct push, runs every leg. kubernetes-nightly.yml runs every leg each night on the newest main build that passed CI. It catches a change that the kubernetes filter does not select. Such a change reaches staging, and can reach a release, up to a day before the nightly run tests it. A failed nightly run updates one tracking issue.

• The same workflow's Sandbox footprint job measures the review sandbox's memory. ReviewFootprintStagingLiveTest stages a review of a large repository at a pinned change: SwiftFormat, 37 practices. scripts/sandbox-footprint.ts then runs two reviews at once on the agent image that the tested main build published. It pulls that image by its digest. Only the model is a stand-in, so the job needs no credential. The sampler container runs privileged in the host's process namespace, because it reads every container's cgroup and processes. The job has no secret that this could expose. A run fails unless each review submitted every practice, finished, uploaded its result and was sampled. The memory result is informational: no budget is enforced until the sandbox requests file holds the measured need. The two reviews start together. The concurrent peak sums their samples per tick, so drift between them lowers it. A busy merge queue skips the night with a notice, because a busy queue is not a defect in main. The job waits for the queue for up to 45 minutes. Then it works for about 15–20 minutes, inside a 90-minute limit. A failed run updates its own tracking issue, "The nightly sandbox footprint run failed".

• To read a footprint failure, open the run's sandbox-footprint artifact:

  1. Read verdict.reasons in footprint.json. Each reason names the review, counted from 0, and the check that failed.
  2. A runtime exit of 137 means that the kernel killed the container. Usually it reached its memory limit.
  3. Open the review's container logs: reviews[].initializeLog and reviews[].log give their paths in the artifact.
  4. Compare the review's runtime peak and processes with an earlier green run to find what grew.

• A job's timeout-minutes is a hang detector of at least 15 minutes. This holds whatever the job's usual duration is. GitHub documents only that it cancels a job that runs longer than its limit (jobs.<job_id>.timeout-minutes). On 2026-10-24, the steps of one job finished in seconds. About 6 minutes later, the job was marked cancelled for exceeding its 5-minute limit. The cause is unknown. A cancelled prerequisite fails CI Status Gate and removes a pull request from the merge queue. The gate cannot tell that cancellation from a real hang. The floor of 15 minutes is a chosen margin over that one delay, not a value that GitHub documents. Revisit it when more delays are measured. The 15-minute limit of ubuntu-slim jobs (GitHub-hosted runners) is a ceiling for those jobs, not the basis of the floor. This rule covers job limits only. A step limit stops the step's process on the runner. ci-contract.test.ts holds every job to the floor, and every number in an expression limit too.

• Action and composite setup steps run sequentially within each job. The runner writes the shared event file before each action. The action SDK reads that file. Concurrent action starts can expose incomplete event data. Jobs and matrix entries still run independently. Standalone shell work that does not read action context can use native background: true with a join before its consumer.

• Security starts from Changes. Only Pinned upstream images reads the advisory snapshot, so only it waits for Image advisories.

• Quality legs run independently. In a merge group, a failed leg cancels the run.

• The Tooling and docs leg runs in three jobs: the gates, and two halves of test:tooling and test:agents. Those two suites took 9 of the leg's 12 minutes in one job. HEPHAESTUS_NODE_TEST_SHARD (1/2 or 2/2) passes Node's --test-shard to both suites, so each test file runs in exactly one half. Locally the variable is unset, and each suite runs whole. The two native platform jobs run independently. No matrix cancels a sibling. A failed platform cancels a merge group through the isolated helper. Other events collect both scans.

• All images of a run share one build job per platform and one manifest job. Each job pays a runner's setup. The Server image is built from the packaged JAR. Thus, the build jobs start when Server package uploads it, and the Dockerfile images wait with them.

• A push to a pull request cancels the earlier runs of that pull request, and no other runs. A superseded run validates a commit that nobody merges, and it holds runners that other pull requests wait for. Each workflow that runs on pull_request declares this at the top level:

  concurrency:
    group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
    cancel-in-progress: ${{ github.event_name == 'pull_request' }}

  Equivalent groups are permitted, such as the pull request number in pull-request.yml. A group must not use the head branch name alone, because two forks can push branches with the same name. A reusable workflow declares no lane of its own. In a called workflow, github.workflow is the caller's name, so a group there would join the caller's lane. The caller's lane covers the jobs that it calls.

  Some workflows do not cancel, because a run cancelled halfway leaves state that no later run replaces. That state is a deploy, a teardown, a commit, or an approval. The policy runs only after a successful pull-request.yml run, so a cancelled run leaves no verdict, and that workflow cancels. UNCANCELLED_PULL_REQUEST_RUNS in scripts/ci-contract.test.ts lists the workflows that do not cancel, with their reasons.

  The test evaluates each group. It fails when a workflow lacks the rule or when one pull request cancels another. It also fails when a run can cancel a run on main or in the merge queue. It also fails when a workflow-level or job-level lane of a listed workflow cancels a pull-request run.

• release.yml never cancels. The same-repository Version PR shares a concurrency lane with a dispatch on its branch. A racing trigger replaces redundant validation. Either survivor must pass release preflight. The merge queue separately validates the final merge tree. Ordinary PRs, forks, main pushes and merge groups retain separate lanes.

• A run on main that leaves a per-commit record — semgrep.yml, scorecard.yml, scorecard-ratchet.yml — does not cancel an in-progress predecessor when another commit arrives. A missing record can become a gap in a later score. Those workflows write cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}, and scripts/ci-contract.test.ts holds every main push to it. A workflow whose newest run simply replaces the last, such as the docs deploy, is named there as the exception. GitHub can still replace a pending run in the same concurrency group: cancel-in-progress: false protects running work, not every queued commit. Post-merge verification must use the actual completed run for the converged main head, not infer success from this setting. GitHub's native queue: max can retain more pending runs, but the Workflow lint parser does not yet support it. Adopt it when a supported validator release is available. Do not suppress workflow syntax checks or add a custom queue.

• cicd.yml gives each push to main a lane of its own, so no push run waits for or replaces another. A replaced pending run left its commit without a verdict. It also showed as the latest completed run, and the status badge counts a cancelled run as failing. Push runs on main can thus finish out of order. Two rules keep that safe. A staging promotion for an older build promotes the newest successful build that descends from it. A replaced promotion thus strands no build. The image build moves the main tags only forward, and never leaves them behind a published commit.

Integration shards

The review shard holds ReviewEndToEndIntegrationTest. The tier and path rules select it and the numbered shards.

The numbered shards split every other integration class by the seconds that it took. Gradle sorts the classes from the longest to the shortest and puts each class on the shard with the least total so far. A class without recorded seconds, such as a new class, goes to shard hash(class) mod n + 1. A @Nested class stays with its top-level class. This is the idea of Develocity's Test Distribution, which balances partitions with historical test times, without its server.

The number of shards and the default seconds live in server/application/integration-test-durations.json. The CI matrix lists review and each numbered shard, and scripts/ci-contract.test.ts pins the matrix to the file. To change the number of shards, change shards in the file and the three matrices.

The file is committed, so every run of a commit partitions the same way, locally too. A partition that changed on each push would move hundreds of classes into new JVMs at once. A test that relies on a row of an earlier test would then fail in some unrelated merge group.

When a shard drifts towards its timeout, refresh the seconds from a recent merge group with node scripts/integration-test-durations.ts refresh <run-id>. The command keeps the mean of each class's committed and measured seconds and drops classes that no longer ran. When selected, the Architecture check verifies the partition on the pull request. A merge group with selected server suites runs every shard with the new partition.

vp run test:server:selection proves that review and the numbered shards hold each integration test exactly once. When server suites are selected, the merge group runs every shard, so a partition never removes a test from that run. Each shard's job summary lists its classes and their seconds. To repeat a failure locally, run its class with --tests, or run the shard with HEPHAESTUS_INTEGRATION_SHARD=<n>.

Recovery tests

The latest-minor amd64 leg runs the replacement-storage recovery drill nightly and on pull requests that change its inputs. The kubernetes-recovery path filter owns those inputs: the chart, PostgreSQL image, drill script and qualification workflows. Merge groups skip this destructive drill.

The drill backs PostgreSQL up to a disposable S3-compatible server and archives a later WAL-only row. It removes the disposable source claim, provisions a fresh claim and volume on a replacement k3d node, and restores through the chart. The drill requires removal of the old volume, checks the replacement node and requires both rows after replay. It applies clone lockdown before application roles restart. It preserves the cluster configuration and Secrets. It does not rehearse recovery of those files or an off-node host failure.

Workflow validation

Workflow lint runs GitHub's parser, Actionlint and Zizmor in one job and blocks through CI Status Gate.

1. scripts/check-workflows.ts loads GitHub's workflow parser and language service. It validates every workflow and composite action: schema, expressions, contexts, needs, step references, and the native background, wait, wait-all, cancel and parallel steps. A warning fails the check. It compares each local ./ action step with the inputs that the action declares. It gives the steps context the outputs that a local action declares. It reads a workflow, an action or a local reusable workflow only inside this repository, through links as well. It runs ShellCheck at warning severity on the bash and sh scripts of composite actions and of workflows that use native steps. It runs Pyflakes on their python scripts. It does not check PowerShell. Its pinned tool downloads use the release-asset path below. An unavailable or corrupt archive fails the check. The SHA-256 and executable checks still apply.
2. Actionlint checks the workflows that use no native steps, because it cannot read native steps. It keeps its own ShellCheck and its metadata for popular remote actions.

For a workflow that uses native steps, the first validator also reads the action.yml of each public action pinned to a commit from raw.githubusercontent.com, without credentials. It checks that action's inputs and outputs, and fails when the file is unavailable. Thus, this check needs network access. Nothing it reads is stored in the repository. vp run test:tooling runs the validator's own tests.

Pull request policy runs cancel earlier runs only when a new commit arrives. An edit therefore cannot replace a successful policy check on the same head with a cancelled check. Each run reads the current title, body, and commits from the API. The identity check walks every GraphQL commit page. The attribution check reads that same complete list. It refuses harness and tool markers only: a Co-authored-by: trailer that names a model is allowed.

Monitoring CI

Use GitHub's organization-level Actions metrics for workflow and job run time, queue time, failure rate, and runner usage. CI Status Gate evaluates every direct prerequisite and the required release evidence preflight. It publishes the separate commit status without checking out source. Its logs and local summary retain each prerequisite result. A local summary write failure does not change the verdict.

After CI finishes, CI / Diagnostics reads that run's exact attempt with read-only Actions permissions. It reports the completed verdict, job times, and step times in a separate workflow summary.

Partial reruns can leave incomplete job metadata. Those timings and a missing gate remain unavailable. A partial record does not fail the diagnostics workflow. API errors and unrelated run records still fail it. Cancelled CI runs allocate no diagnostics runner.

Attempt-to-gate time starts when that attempt starts, so it excludes earlier attempts. Creation-to-start time includes dependency waiting. Missing timestamps remain unavailable. A reporting failure cannot change CI, Version PR maintenance, or release eligibility. JUnit results appear as Test Results - … check runs.

Scheduled failures

CI / Scheduled failures reports failed scheduled workflows from trusted run metadata. It executes no upstream source or artifacts and installs no dependencies. One issue per workflow notifies the maintainer team. Each later failed attempt adds one comment. Replays of an already reported attempt add no notification.

The Kubernetes nightly and image rescan workflows keep their specific reports and notify the same team. The shared reporter checks those attempts and reports a failure only when the specific owner did not notify maintainers. This also covers job timeouts, which can prevent an in-job report step from running. A test checks that every scheduled workflow has a report owner. Manual runs and pull request runs do not enter the shared reporter.

Full-verification latency

The full-verification target is a six-minute median and a seven-minute p90, measured from workflow creation to completion of CI Status Gate. A merge group always runs both verification tiers, so the sample is merge groups. On the default branch, CI / Profile lists merge-group runs created in the last 28 days. It reads the first attempt, including when a later attempt is still running. Both outcome shares below use that first attempt, so a successful rerun cannot hide a failure. Run listings use UTC days and fail at the API limit rather than report an incomplete window.

It selects the ten most recent successful first-attempt runs with full server and browser verification. Lightweight changes and release-candidate groups, which also generate release evidence, cannot satisfy this cohort.

The report also selects the ten most recent successful first-attempt PR runs of the fast tier. Unit or review must have run. No numbered integration shard or browser suite can have run. Migration PRs remain in that cohort when they run Database and the drills.

It reports the median and p90 as an observation, with no target. It counts the merge groups of the last 14 days in which a queue-tier job failed, for the revert criterion. A failed worker still counts when fail-fast cancellation gives the workflow a cancelled conclusion. A cancelled group with no failed worker, or only a failed gate, its reporter or a cancellation helper, does not count as a failure.

The same weekly report shows all cancellations over seven days, split by cause:

• Native worker failure, including fail-fast cancellation after that failure.
• Predecessor drop, matched through the queue branch's predecessor SHA and queue removal events.
• Manual removal, matched to the exact merge-group SHA.
• Timeout, from a native job timeout.
• Unknown, when the available evidence cannot identify the cause.

A successful intermediate group does not break the predecessor chain. Traversal stops when a predecessor merged before the cancelled run began. A native worker failure takes priority if a predecessor also drops. The guard separates groups with a native worker failure from cancellations without one.

The report retains the matching removal event or failed job names with each cancelled run. These records identify the cause, not the identity of the caller that cancelled a run. The observer needs read access to pull requests for queue removal events.

Target: at most 10 % of completed merge-group first attempts in seven days are cancelled for a cause other than a native worker failure. The denominator includes successful and failed groups as well as cancelled groups. Unknown causes count against the target and make the classification incomplete. This includes a cancelled first attempt with no native job records. The guard applies without a minimum sample and always shows the denominator.

A breach or an unknown cause produces a warning in this report. The observer adds no alert channel and changes no queue, tier or required check.

Budget misses produce warnings and a job summary, not workflow failures. Fewer than ten qualifying runs produces an insufficient-data notice, not a pass or a regression.

All observations retain the JSON report. Historical latency includes runner waiting and earlier commits, so it is not a correctness verdict on the profiled commit.

Performance observations are advisory in this diagnostic workflow. Test failures, unreadable recordings, invalid metrics, and reporting/API errors still fail it. Do not use continue-on-error to hide those failures. Investigate sustained warnings using the retained run timelines and profiles.

A blocking performance gate needs a comparable workload and runner environment, an established noise model, and an attributable regression. A historical fleet-wide target miss is insufficient. This separation follows SRE guidance on actionable, low-noise monitoring and uses native GitHub warnings and job summaries.

Reproduce the observer with an authenticated GitHub CLI:

GITHUB_REPOSITORY=owner/repository vp run report:ci-latency

To inspect one completed run, download its metadata and every page of jobs for the same attempt. Replace RUN_ID and ATTEMPT in both requests:

mkdir -p tmp/ci-timings
gh api repos/{owner}/{repo}/actions/runs/RUN_ID/attempts/ATTEMPT > tmp/ci-timings/run.json
gh api --paginate 'repos/{owner}/{repo}/actions/runs/RUN_ID/attempts/ATTEMPT/jobs?per_page=100' |
  jq -s '{total_count: .[0].total_count, jobs: [.[].jobs[]]}' > tmp/ci-timings/jobs.json
vp run report:ci-timings tmp/ci-timings/run.json tmp/ci-timings/jobs.json

The report separates time before job creation, time before the first runner, per-job start delay, and execution time. Start delay includes dependency waiting, so it is not a pure runner-queue metric. Summed execution times overlap: they measure runner demand, not verdict latency. Retries retain the original run creation time and must not enter first-attempt percentiles. Unlike the gate's in-flight summary, this report uses its completed verdict and rejects incomplete job lists or mixed attempts. Keep raw API evidence under tmp/, not in committed performance snapshots.

Daily security rescan

Security / Rescan images runs daily and on dispatch. It rescans the images from main, the supported release, dependency locks, and the one-host platform pins. Platform image scans cover both release architectures and resolve each subject to a digest. Dependency and platform findings go to access-controlled code-scanning alerts, without public report artifacts.

Dependency and platform scans run only on the default branch. Select that branch for a manual rescan. An unavailable scan fails the scheduled job. It does not count as a clean scan.

The merge queue keeps the full vulnerability policy, including findings against unchanged dependencies. A daily rescan reduces the time before maintainers see new advisories. It does not guarantee an advisory arrives before a merge group. A finding requires an update or a reviewed policy exception. Cancellation or a retry does not waive it.

Renovate tracks the platform chart versions, image tags and digests, and the k3s release. These updates require dashboard approval and human review. A k3s update also needs an independently verified installer checksum. The Platform pins check and scheduled rescan check that pair. The pull request check renders both upstream charts and compares their image subjects with the scan plan.

The rescan also reads the selected k3s release's complete image list. It scans each image of that list except the images of the add-ons that k3s-config.yaml disables, because the host never pulls them.

Flaky tests

CI / Flaky test report runs every Monday and on dispatch. It covers the CI/CD runs of the last 14 days, including merge groups. The report counts cancelled runs separately by event. A cancellation is not a flaky test. Each test and job count includes a commit once, even if several runs repeat that commit. It lists the jobs and tests that failed in one attempt of a run and then passed in a later attempt.

A job or test that fails again in the run's last attempt is a real failure, so the report leaves it out. If a job that never passed does not fail again in the last attempt's own rerun, the run's tests stay unknown.

The report reads run attempts and the retained JUnit reports, and it retains flakes.json for 90 days. Reproduce it with an authenticated GitHub CLI:

GITHUB_REPOSITORY=owner/repository vp run report:flakes

Testing Guide § Flaky tests has the policy.

Server profiles

PR and merge-queue test jobs retain JUnit XML reports for 14 days, separately for each suite, shard and attempt. Use those reports and GitHub's step timings before rerunning tests with instrumentation. Deep profiling stays in the manual/weekly workflow, not on the PR critical path. Select integration, verification (unit and architecture, including coverage), or all when dispatching CI / Profile. Weekly runs profile both tiers sequentially so diagnostics do not occupy two heavy runners at once.

The integration tier is profiled one shard at a time, with the shards that Parallelism and concurrency defines. A run of the whole tier no longer fits in one test task's timeout. The shard list is mirrored in both weekly workflows and pinned to the CI matrix by scripts/ci-contract.test.ts.

Each tier retains its JFR recording, Gradle task profile, resource usage and JUnit results for 14 days. Verification profiles are diagnostic artifacts, not integration-history samples.

Profile invocations use a single-use Gradle process (--no-daemon) for repeatable JVM startup. Gradle streams test output only in these profiles so Spring context diagnostics reach the retained log. History uses one ci-profile-history-gradle-jfr-<shard> artifact for each shard and compares wall time, test time and context measurements. Each sample names its shard. A baseline accepts only samples of the same shard, so the earlier whole-tier history is not used. The profiles partition by the committed seconds, so a shard's classes change only when the committed file changes.

GNU time's CPU/RSS accounting does not reliably include Gradle's detached JVMs, so those values are not reported as suite metrics or compared against budgets. Inspect the retained JFR recordings for JVM CPU, allocation and GC diagnostics instead.

The manual runner-comparison selection runs integration shard 1 twice each on standard Ubuntu 24.04 x64 and ARM64 runners, one job at a time. Preparation is a separate step so cold dependency and compilation caches are not mistaken for test execution speed. Each sample retains runner hardware, JUnit results and Gradle task profiles. Compare actual job ordering, test counts and execution time as well as end-to-end latency. Matrix declaration order is not a scheduling guarantee. These samples do not enter profile history and do not change the runners used for required CI.

The integration profile also records Spring context-cache metrics. It warns about a sustained regression after three consecutive profiles exceed variance limits against five earlier default-branch profiles or absolute Spring-context limits. Only successful test runs with a readable JFR recording and valid summaries enter history, including out-of-budget samples.

History is restored from this repository's default-branch JFR history artifacts, retained for 90 days. A rerun replaces its own sample. Uninstrumented runs are not baseline candidates.

Without seven earlier profiles, the sustained-regression check has no verdict. Branch dispatches produce diagnostic artifacts without changing or comparing against the baseline.

The workflow's Run integration shard with JFR step owns the invocation. Additional test-JVM flags use -PtestJvmArgs=.... Gradle composes them with the configured memory limits and JaCoCo agent. -PprofileTests=true writes a separate JFR recording for each test task, so unit and architecture recordings never overwrite each other.

Missing recordings, Gradle reports or resource metrics fail the profile rather than becoming zero-valued baselines. Integration profiles also require Spring-context measurements. Unit/architecture profiles do not require a Spring application to boot.

For local shard profiling, use the same HEPHAESTUS_INTEGRATION_SHARD selector as the CI matrix and the Java version pinned in .java-version. Compare the same suite, runtime and cache state. Compare a local shard only with the weekly history of the same shard.

SpringTestContextArchitectureTest enforces the reviewed Spring context keys. The weekly CI / Server clean-build reference workflow profiles generation, compilation, and unit and architecture execution in one job, with the build cache disabled. It compares two clean generated-client JARs. When that job ends, each integration shard runs in a separate job, one at a time. Each shard job compiles from source with the build cache disabled. The Gradle profile (--profile) of each job is the phase breakdown: it lists the time of every task.

Running CI locally

vp run format && vp run check

When relevant to the change, also run vp run build:webapp, vp run test:server:verification and vp run test:server:integration. vp run verify runs the credential-free set.

# Webapp
vp run check:webapp          # vp check: format and lint; the lint is type-aware and type-checks
vp run fix:webapp           # check:webapp, applying every safe fix
vp run test:webapp           # Unit tests

# Chrome extension
vp run check:extension       # vp check for extension/
vp run test:extension        # Unit tests; test:extension:stories for the stories
vp run verification:extension-build  # Production zip, then its manifest check

# Application Server (Java)
vp run format:java:check     # Check formatting
vp run test:server:unit      # Unit tests

# Agent runtime (Node) — the Pi runner and the practice scripts
vp run test:agents           # Runner + practice script specs
vp run check:agents           # Format check, oxlint, both typechecks, release-image inventory, scripts/*.test.ts
vp run fix:agents            # check:agents, applying every safe fix
vp run typecheck:agents      # Agent + practice script TypeScript

Issue	Solution
Formatting errors	Run vp run format
Lint errors (agent runtime, practice scripts, scripts/)	Run vp run fix:agents, then fix what remains
TypeScript errors	Run vp run typecheck to see details
OpenAPI out of sync	Run vp run generate:api
Breaking API change	See API compatibility
Database schema drift	Run vp run db:draft-changelog

Adding a new service

Add a service to CI gives the steps.

Static preview comments

Storybook and documentation previews list every published story or page from changed source files. They use the built Storybook index and Docusaurus metadata, not guessed routes. The lists are not a dependency-impact report: changing a component without changing its story file does not add that component's stories. Unpublished or deleted entries cannot link to a preview page.

Stories are grouped by their full Storybook component title (including its sidebar path), and pages by their source directory. Each group shows its count and all its links, with no collapsed sections or item cap. Headings and wrapping link paragraphs keep the paths readable without repeating them for every variant or introducing a wide table.

scripts/render-preview-comment.ts writes a JSON array of complete Markdown comments. The shared renderer keeps whole links together and repeats the group heading when a group spans comments. Large lists continue in numbered comments below GitHub's 65,536-character limit. Every part retains the preview link, total count, checked-out commit and build logs. An individual link too large to fit fails explicitly rather than dropping content.

scripts/publish-preview-comments.ts uses the GitHub SDK provided by actions/github-script to update the preview's existing bot comments in place. It validates the entire artifact before writing, paginates existing comments, skips unchanged bodies, and deletes surplus parts only after publishing every current part. GitHub does not offer atomic multi-comment updates. A failed run fails the publication step and a retry converges without duplicate comments. Teardown replaces the main comment and deletes continuation parts for both previews. A delayed publisher that observes a closed pull request also retires its comments instead of restoring live links.

Documentation preview deployment

Docs build errors still fail the build job. The optional preview has its own result. Only the Surge command can fail without failing the job. A failed upload writes an error status under Preview / Docs, with a link to the workflow log, and adds a warning and a step summary. API, artifact, build, and status-report failures still fail their steps.

Forks still build. Their publication job is skipped because it has no hosting credential or status-write permission.

cd-docs.yml compares documentation trees before it selects publication. For a synchronize event, it compares the previous head with the new head. Thus a code-only push does not redeploy documentation that an earlier commit changed. For an opened or reopened pull request, it compares the merge base with the head. The workflow's path filter alone cannot establish this: GitHub filters pull requests on their whole three-dot diff.

Build jobs cancel superseded builds. Upload and teardown share the docs-preview-<PR number> concurrency group, keep active mutations running, and retain queued mutations with queue: max. After acquiring this lock, upload re-reads the pull request and requires an open state and the exact tested head. Teardown requires a fresh closed state. A delayed close event cannot remove a reopened pull request's previews.

scripts/docs-preview.ts owns status transitions. Before Surge starts, the upload records a pending claim on the exact head. Only this attempt can turn that claim into success or error, from its actual upload outcome.

An existing Preview / Docs status, including pending or error, prevents another upload on that head. A rerun preserves and reports that status. It never turns an earlier pending claim into success. A later head that changes docs can claim its own upload.

The diagram shows writes by the attempt that creates a claim. Runs that find an existing status send no transition, including an existing provider failure status.

Signal	Meaning	Action
Preview / Docs is error	The preview was not published.	Read its linked workflow log. The required docs build has its own result.
Preview / Docs stays pending	An attempt claimed this head but did not report an outcome.	Read the claim's linked run. A rerun preserves the claim and does not upload again.

GitHub documents job concurrency and its queue, commit statuses, and path-filter comparisons.

Chromatic visual coverage

The Stories job reports visual coverage separately from interaction tests and the Preview / Storybook publication link. Visual coverage never decides the job. Storybook interaction tests are the gate. Chromatic's verdict appears in a step summary, with a warning for unverified coverage. Neither a preview link nor a passing job is visual approval. Review the linked build.

Verdict	Meaning and action
tested-build	The returned build has captured snapshots and a terminal successful JUnit report with no unfinished or failed test cases. It may be a reused approved build, not new captures in this CI run.
inherited	The returned build reused baseline snapshots without new captures. Valid TurboSnap coverage, but not a newly tested recovery baseline.
quota-skipped	An account limit blocked coverage. The job still passes with a warning. An account owner resolves the limit.
failed	Visual differences or component/interaction errors need review. Fix errors. Accept only intentional, inspected differences.
unavailable	Chromatic failed or returned insufficient coverage evidence, for example an unfinished build after an account limit. The job still passes with a warning. Inspect the build, action logs and account limits.
policy-skipped	Fork PRs and dependabot/ or renovate/ branches do not run credentialed visual tests. Interaction tests still run. This is not visual approval.

scripts/report-chromatic.ts requires both the action's public outputs and its fresh JUnit report. The workflow removes the previous report before invoking Chromatic and uses one explicit report filename for validation and artifact upload. The report must identify the same returned build, have status PASSED or ACCEPTED, and contain only successful terminal test cases with no reported failures. Missing, malformed, mismatched, incomplete, or nonterminal evidence is reported as unverified.

CLI exit codes and positive capture counters alone are insufficient: publish-only mode can return zero even after an account-limit warning, while snapshot records are still IN_PROGRESS. Check project UI testing/review settings as well as billing when the log says no UI tests or UI review are enabled. Publication is not a completed visual run.

Counts describe the returned build, which Chromatic can reuse. They do not establish when snapshots were captured. Accepted builds can retain nonzero change counts. Chromatic's JUnit writer marks both skipped and preserved records as skipped, so that marker alone does not reject a successful inherited baseline. Nor do snapshot and test counts share a promised equality: the gate checks terminal structured evidence instead of subtracting counters. Review this contract when updating the action.

For visual verification of a fork contribution, a maintainer can review it and run a trusted same-repository branch. Do not expose project secrets to forks. No workflow runs on pull_request_target (Pull request policy).

Recovering quota and the baseline

Recover the Chromatic baseline gives the steps.

Snapshot consumption and global changes

TurboSnap reuses unchanged baseline snapshots. webapp/chromatic.config.json owns external-file invalidation: global styles, public files, assets and build configuration intentionally trigger full snapshots. Budget for those runs. Do not narrow externals or broaden untraced to reduce quota use or latency. Follow Chromatic's external-file guidance when changing these boundaries, and preserve dependency stats from the single Storybook build.

Release tool downloads

The tool pins are in .github/ci-tools/*.json and .vale/toolchain.json. Renovate-managed GitHub pins name the repository in githubRelease. Each asset pairs its full release tag with its SHA-256. The resolver derives the URL and replaces {version} in the asset name with the tag's version. All assets of a tool must use the same release tag.

Renovate's release-attachment datasource updates each tag and digest together. Keep the v prefix: the datasource looks up the exact upstream release tag. The manager keeps each asset separate, including checksum files and Cosign signature bundles. The Cosign public key comes from the release attachment with the existing pinned SHA-256.

Renovate obtains digests from the same upstream release as the assets. A matching digest proves download integrity. It does not prove publisher authenticity. Before merge, independently verify every changed asset checksum and Cosign trust material. The Renovate PR notes state this requirement for automated updates too. Cosign's existing bootstrap and signature checks remain required.

The other pins are manual: Helm, Kind, k3d, Kubeconform, oasdiff, ShellCheck, Gateway API, Vale, ripgrep, Liberica, bootstrap tools and buildpack dependencies. For a manual update, change the version, source URL and checksum together. Apply the same independent verification before merge. The resolver never selects a newer release.

setup-ci-tool restores the GitHub Actions cache with a key made from the tool, version and SHA-256. It verifies the restored file before use. A valid hit needs no upstream download. A missing or corrupt cache entry falls back to the GHCR mirror, the upstream GitHub API, then the direct URL. Every source must match the pinned SHA-256. A mismatch refuses that file and tries the next source.

The log names the source that supplied each verified tool. If no source supplies the expected bytes, the gate fails with the tool identity and source failures. Cache restore and save are best effort. A verified download is saved for later runs.

Mirror and upstream requests make at most three attempts, with waits of one and two seconds. The timeout is ten seconds for API metadata and 30 seconds for each asset attempt, including its body.

The asset API uses the job token first, then tries the public endpoint without credentials. A denied authenticated request can still succeed through the public API. The token stays on api.github.com and github.com, including redirects between those origins. Storage redirects retain their signed URL credentials. Other hosts receive no repository token.

Cosign keeps the upstream installer's trust chain. Its 3.1.3 bootstrap and release public key have the same pinned SHA-256 values as before. The bootstrap verifies the selected binary with both its keyless bundle and its KMS bundle. Syft and Trivy archives also match their pinned release checksum files before extraction.

The image build installs pack through setup-ci-tool, for linux/amd64 and linux/arm64. Pack publishes a SHA-256 file for each archive, but no signature. Its pins are the published SHA-256 values.

The Server image build downloads no buildpack dependency. Each .github/ci-tools/buildpack-*.json pin holds a file that a buildpack would download, with the SHA-256 from its buildpack.toml. scripts/buildpack-dependencies.ts resolves the files and writes a Paketo dependency-mapping binding, which maps each SHA-256 to its file. The buildpack verifies the file against that SHA-256 before it uses it. If a buildpack still downloads a file, the build fails and the log names the URI. After a builder or buildpack update, pin each file that the log names.

The mirror is ghcr.io/hephaestus-build/ci-tools/<tool>:<version>. An OCI tag cannot contain +, so a pin writes a + in an upstream version as _. Pulls use the job token with packages: read. The resolver verifies the selected blob against the repository pin.

The Cache / Mirror pinned CI tools workflow runs on main when pins change, each week, or by manual dispatch. It reads the same pin files, verifies every platform asset, then publishes the version with packages: write. A failed group leaves the existing version tag unchanged. Other tool groups can still publish.

Mirror errors name the registry operation and HTTP status, without response bodies or credentials. If authentication or publication is denied, check the job's packages: write grant and the package's Actions access. An organization owner controls private package creation under Organization settings → Packages → Package Creation. See package creation permissions.

Its upstream requests allow four attempts of 60 seconds, with waits of five, ten and 15 seconds. This longer retry runs outside the pull request gates.

Trivy's advisory databases are changing data, not executable tools. They retain the GHCR and ECR fallbacks and the existing freshness checks. No pinned tool source uses Docker Hub.

Signing jobs use the Cosign installer action at an immutable commit, with no checkout of the build branch. The installer also pins its shared resolver action to an immutable commit. Signing jobs disable Actions cache archives for Cosign, security tools, Helm and pack. Archive extraction happens before the tool checksum check and can change other runner files. These jobs use GHCR, API and direct downloads, with verified files reused only within the job. Publish and review a new installer commit before updating those references.