Testing Guide

Choose the tier by the behavior under test. Gradle recipes lists the commands. JUnit tags select tests independently of filenames. Use descriptive *Test and *IntegrationTest names for readers.

Unit tests

• Extend BaseUnitTest for isolated component tests.
• Use Mockito (@Mock, @InjectMocks) for dependencies.
• Keep unit tests independent of Spring contexts and Testcontainers.

Integration tests

• Extend BaseIntegrationTest for cross-component behavior.
• Reuse the shared PostgreSQL Testcontainer.
• For webhook tests, deserialize recorded fixtures into the handler's event DTO.

Controller integration tests

Test controllers at the HTTP boundary so access-control configuration is exercised. Reuse the domain's integration-test base and fixture factories.

• Extend AbstractWorkspaceIntegrationTest or a service-specific base. Each controller IT then cleans the database automatically. It provides helper factories such as persistUser and exposes workspaceService for setup.
• Use WebTestClient. The full Spring Boot context already runs. Thus, tests verify access-control rules exactly as the frontend receives them (401 unauthenticated versus 403 non-admin).
• Reach for the custom security annotations (@WithAdminUser, @WithUser) together with TestAuthUtils.withCurrentUser() so the generated JWT matches our resource-server configuration.
• Assert on status code, payload shape, and repository side effects. A minimal test covers "happy path", "validation failure", and at least one access-control branch.
• Keep factories close to the domain. Prefer createWorkspace("slug", "Display", ...) to manual repository seeds. This keeps each test focused on behavior instead of boilerplate.

Example skeleton:

@AutoConfigureWebTestClient
class FooControllerIntegrationTest extends AbstractWorkspaceIntegrationTest {

    @Autowired
    private WebTestClient webTestClient;

    @Test
    @WithAdminUser
    void endpointHappyPath() {
        User owner = persistUser("owner");
        Workspace workspace = createWorkspace("slug", "Display", "login", AccountType.ORG, owner);

        webTestClient
            .get()
            .uri("/workspaces/{workspaceSlug}", workspace.getWorkspaceSlug())
            .headers(TestAuthUtils.withCurrentUser())
            .exchange()
            .expectStatus()
            .isOk();
    }
}

Cross-tenant isolation tests

Tenancy bugs silently leak data. The endpoint answers 200 with plausible data, so nothing fails visibly. Every new workspace-scoped read path gets an isolation case in CrossTenantIsolationIntegrationTest (in workspace/). Its class Javadoc documents the fixture and reason: two workspaces share an account_login and have one overlapping member.

Checklist for a new read path:

• List endpoint — assert A's collection under A's slug never contains B's rows (length() + value, not just "my row is present").
• Detail endpoint (anti-IDOR) — assert that B's entity id through A's slug returns 404. Add a positive control: A's own id returns 2xx. Thus, the 404 cannot pass against a broken endpoint.
• Query keyed on anything but workspace_id is a warning sign. Examples are an org login, a login string, or an external native id. Before shipment, scope by workspace_id or add the provider/workspace predicate.
• SCM work scoped through repository_to_monitor? A monitor stores only the repository name, and another provider can host the same name. Match the name, then call monitor_covers_repository(...) = true in JPQL or append MonitoredRepositoryScope.NATIVE_RTM_R in native SQL. Test two workspaces on different instances of the same provider kind. Give both monitors the same name and native id. Include the unconnected inverse and verify that removal preserves the other instance's repository.
• @WorkspaceAgnostic repository? The SQL statement inspector is off for it. Thus, the integration test is the only automated guard. Do not skip it.

The hephaestus.tenancy.enforcement: throw setting in application-test.yml fails tests that read a workspace_id-bearing table without a workspace_id predicate. This safety net cannot see @WorkspaceAgnostic queries or string-keyed scopes. Thus, HTTP-level assertions remain necessary.

Webhook fixtures and tooling

Reusable webhook JSON lives in server/application/src/test/resources/github and …/gitlab, named after the event it records. Load one with a ClassPathResource, or extract new samples with:

vp run schema:nats

Available examples include label.created, repository.created, create, push, and more – consult the folder before recording new payloads.

Before you commit a recording, replace each person's identity with a synthetic one of the same shape. This includes the name, username, email address, numeric user ID and avatar URL. Use one synthetic identity for each person across all fixtures, so that a lookup that the tests exercise still matches. The test organization's own accounts (HephaestusTest and the maintainer account that recorded the payloads, FelixTJDietrich) can stay.

vp run schema:nats only sees events an environment already subscribes to. To record any other GitHub event, add a temporary repository webhook in the HephaestusTest organization that subscribes to that event only. Its URL does not need to answer. Cause the event, and then save each delivery's request.payload from gh api repos/<owner>/<repo>/hooks/<hook_id>/deliveries/<delivery_id>. Delete the webhook and whatever you created when you are done. The issue_dependencies fixtures were recorded this way.

A fixture ending in .derived.json is not a recording. It starts with a recorded payload. For fields without a recordable event in the test organization, it uses the provider's documented payload. Replace it with a recording when one exists.

Each row lists every field the fixture adds, changes or removes against its recorded base.

Fixture	Recorded base	Fields that differ
github/pull_request.review_requested.reviewer.derived.json	pull_request.review_requested, a team request	requested_team replaced by requested_reviewer, a user, since GitHub sends one or the other (octokit/webhooks example). pull_request.requested_reviewers lists that user and the Copilot reviewer bot as /users/copilot-pull-request-reviewer[bot] reports it. pull_request.requested_teams is as recorded.
gitlab/merge_request.update.reviewers.derived.json	merge_request.update	Adds reviewers with each state (approved, requested_changes, reviewed, unreviewed) and re_requested: false, assignees, and changes.assignees (none before). Removes changes.labels and changes.updated_by_id. Moves object_attributes.updated_at and changes.updated_at on to 19:05:00. No changes.reviewers: nobody's state changes in this hook. It carries all four states. Before 19.3, a review or request for changes runs no merge request hook of its own.
gitlab/merge_request.update.review_rerequested.derived.json	merge_request.update	The re-request's own hook: adds reviewers as in reviewers.derived but with user1 unreviewed and re_requested: true, assignees, and changes.reviewers from the reviewers.derived list to this one, as the only changes entry. Moves object_attributes.updated_at on to 19:08:00, which the re-request's system note touches before the hook runs (request_review_service.rb).
gitlab/merge_request.update.reviewers_removed.derived.json	merge_request.update	Adds changes.reviewers from the review_rerequested list to none and changes.assignees from ab12cde to none, with no reviewers or assignees key, as GitLab leaves them out when there are none (issuable.rb). Removes changes.labels and changes.updated_by_id. Moves object_attributes.updated_at on to 19:15:00, changes.updated_at from 19:08:00.
gitlab/merge_request.update.reviewers_stateless.derived.json	merge_request.update	A hook from GitLab before 18.6, whose reviewers carry User#hook_attrs only (issuable.rb): adds reviewers as in reviewers.derived without state and re_requested and without user3, assignees, and changes.reviewers from the four to the three. Removes changes.labels and changes.updated_by_id. Moves object_attributes.updated_at on to 19:06:00 and changes.updated_at from 19:05:00 to 19:06:00.

The GitLab fixtures other than reviewers_stateless follow GitLab 18.6 and later, which states each reviewer's state and re_requested (reviewers_hook_attrs) and lists changes.reviewers whenever those attributes change (hook_reviewer_changes). From 19.3 a submitted review runs merge request hooks of its own as well (update_reviewer_state_service.rb). The GitLab fields follow merge request events.

Gradle recipes

From server/:

./gradlew :application:test                             # Unit tests
./gradlew :application:architectureTest                 # ArchUnit and Modulith
./gradlew :application:verification                     # Unit, architecture and coverage floors
./gradlew :application:integrationTest                  # Spring and Testcontainers
./gradlew :application:test --tests SomeTest            # One unit test class
./gradlew :application:test --tests 'SomeTest.method'    # One unit test method
./gradlew :application:bootJar                          # Executable JAR, without running tests

Generation and compilation dependencies are automatic. Test tasks always execute when requested. Compilation remains incremental and cacheable. --no-build-cache prevents output restoration. Combine it with clean when measuring a cold build.

The shared integration PostgreSQL container runs the docker/postgres image. Locally, each test JVM builds that image from its Dockerfile. CI integration jobs instead pull the image that CI published for the commit their images alias from. They name it in HEPHAESTUS_TEST_POSTGRES_IMAGE. When a change edits the image, or no published image exists, CI builds the Dockerfile once per job before the tests start. That build downloads packages, as a local build does.

Contexts built on BaseIntegrationTest reach GitHub's GraphQL endpoint through ScriptedGitHub. The application's own client, senders, schedulers and listeners run unchanged, and only the HTTP exchange answers from the script. A test reads what the application posted from ScriptedGitHub.threads(), keyed by repository and number. A GitHub document that the script does not answer fails as a GraphQL error.

No JVM client in a test resolves a real host. OfflineNetwork replaces InetAddress host-name resolution in every test JVM. A child process such as git, a native DNS client or a container uses its own resolver, so the guard does not cover it. Loopback names and the names the host gives its own addresses resolve. A name reserved by RFC 2606 or RFC 6761 fails as an unknown host. Any other name is refused and recorded, and OfflineNetworkExtension fails the test class that tried it.

A reverse lookup of a public address fails too. It is recorded unless one of the host's own interfaces carries the address. A test whose code only classifies an address declares the name with OfflineNetwork.answer. The live tier resolves every name.

The test database starts with silent mode engaged, as a new instance does. A test that asserts a posted copy turns silent mode off and restores it afterwards.

The shared integration PostgreSQL container keeps its ephemeral data directory on tmpfs. Database cleanup still truncates every application table transactionally. SQL constraints and isolation checks remain enabled. This container does not prove storage durability across container restarts.

vp run test:server:selection runs dedicated Gradle/JUnit inventory tasks without executing tests. They use the same tier and shard selectors as the execution tasks. The command proves that non-live tests belong to a required tier and that integration shards are disjoint and complete. Required PR verification and vp run verify run this proof before the real suites. Discovery writes only to application/build/test-selection/, leaving execution reports and coverage untouched.

Those discovery reports are not evidence that tests passed. Execution tasks do not accept the removed -PtestSelection property. Use the selection command for discovery.

Flaky tests

A flaky test fails and then passes on the same commit. CI does not retry Gradle or Vitest tests. Playwright retries a failed test once to record a trace, and then it fails the run as flaky. You can rerun a failed job, but the rerun does not hide the flake. The weekly flaky test report lists every job that passed only on a rerun. Maintainers read it each week.

When a test flakes:

1. Open an issue for the test and assign an owner.
2. The owner finds the cause and fixes it, or deletes a test that asserts nothing new. Do not add a retry, a longer timeout, or a sleep to hide it.
3. Quarantine the test only while the flake blocks merges and the cause is not fixed yet. The skip names the issue, the owner's GitHub login and an end date at most 14 days ahead. In Java, write @Disabled("quarantine #123 @owner until 2026-11-07: free-port race"). In TypeScript, put // quarantine #123 @owner until 2026-11-07: free-port race on the line directly above it.skip(.
4. On the end date, the owner fixes or deletes the test.

vp run check fails on a skip that names no issue, owner and end date, and on an expired quarantine. A skip that depends on the environment, such as missing live credentials, is not a quarantine.

Live external-service tests

Live tests use real provider APIs, LLM endpoints or sandbox containers. They carry @Tag("live") and are excluded from every other tier. From server/, explicitly opt in with:

./gradlew :application:liveTest

Use --tests ClassName to select a suite. An unmatched filter fails.

To send only some catalogue criteria cases to the model, name their id values from src/test/resources/practices/criteria-cases.json:

./gradlew :application:liveTest -PcriteriaCases=dependency-label,dependency-import-without-declaration-edit

The property first runs the selected fixture-staging checks, then those cases of PracticeRunnerLiveLlmTest and no other live test. Each case is reported under its id, not its position in the file. A blank, repeated or unknown id fails the run instead of sending every case. Without the property, the live tier runs every case.

To also limit the upstream model requests that each selected case sends, add the optional criteriaRequestLimit property:

./gradlew :application:liveTest -PcriteriaCases=dependency-label -PcriteriaRequestLimit=12

The limit applies to each case. Each forwarding attempt reserves one slot, including retries and compaction requests. Failed or cancelled attempts keep their slots. A request past the limit is refused, and that case fails, even if its result matches.

The limit needs criteriaCases and a positive integer. Without the property, this harness adds no request-count limit. Record provider token usage separately.

To measure the same cases with other model settings, add one or more of these optional properties:

./gradlew :application:liveTest -PcriteriaCases=dependency-label \
  -PcriteriaReasoningEffort=HIGH -PcriteriaMaxOutputTokens=32768 -PcriteriaContextWindow=131072

Property	Value	Default
criteriaReasoningEffort	A reasoning effort: NONE, MINIMAL, LOW, MEDIUM, HIGH, XHIGH or MAX. The names are case-sensitive.	none sent
criteriaMaxOutputTokens	A positive integer, less than the context window	4096
criteriaContextWindow	A positive integer	131072

Each property needs criteriaCases. Gradle refuses a token count that is not a positive integer before any test starts. The live test refuses a lower-case or unknown reasoning effort, and an output allowance that does not fit the context window, before any model request. A live test that is skipped because no credentials are set checks none of these values. Each case prints the model settings it uses.

First check the same selection without credentials or a model:

./gradlew :application:integrationTest -PcriteriaCases=dependency-label,dependency-import-without-declaration-edit

This runs only the selected cases in PracticeCriteriaCaseStagingIntegrationTest. Each invocation stages the evidence and shipped criteria used by the live harness. It proves selection and fixture staging, not model outcomes.

Gradle's test dry run does not execute a parameterized method's argument source. Use the staging results to count selected case IDs. A dry-run report cannot prove that count. The live credential gate remains in effect.

Do not combine criteriaCases with HEPHAESTUS_INTEGRATION_SHARD. The tasks reject that combination instead of adding unrelated shard tests.

Meta-annotation cheat sheet

Use one of these composite annotations on the test class (or method) rather than wiring @Tag("live") + @EnabledIfEnvironmentVariable by hand. They keep gates uniform and put the env-var contract in one place.

Annotation	Gate	Use for
@LiveLlmTest	HEPHAESTUS_LIVE_LLM_API_KEY	Tests that drive a real OpenAI-compatible endpoint. Set HEPHAESTUS_LIVE_LLM_BASE_URL, HEPHAESTUS_LIVE_LLM_API_KEY, and HEPHAESTUS_LIVE_LLM_MODEL explicitly. Read through LiveLlmCredentials.fromEnv().
@LiveGitHubTest	GH_APP_INSTALLATION_ID	Tests that hit github.com through the installed GitHub App. BaseGitHubLiveIntegrationTest runs an extra runtime check on private-key material.
@LiveGitLabTest	HEPHAESTUS_LIVE_GITLAB_TOKEN	Reserved for GitLab-side replay/live-sync tests.
@LiveDockerTest	none — runtime probe in @BeforeAll	Tests that need a Docker daemon. Combine with @LiveLlmTest when the sandbox payload also needs LLM credentials.

The annotations use Execution(SAME_THREAD). Gradle also runs one test JVM at a time.

Authoring a new live test

1. Pick the correct meta-annotation. If the environment variable does not exist yet, add an annotation alongside the existing four. Do not reuse one with a misleading name.
2. Put the test under server/application/src/test/java/.../live/. The directory marker keeps live and non-live tests in different code-review buckets.
3. Read credentials from LiveLlmCredentials.fromEnv() or the provider helpers. Never inline secrets or commit local credential files.
4. Per-test wall-clock timeouts: 90s is reasonable for a single conversation turn on a warm host. The first turn also starts the conversation host. 5 minutes for a full practice run.

Conversation turn live test

HephTurnLiveTest runs one conversation turn through the real path: the ask endpoint, the worker's claim, the conversation host container, the LLM proxy and a model. It needs Docker and the agent image built from docker/agents/pi:

docker buildx build -f docker/agents/pi/Dockerfile -t hephaestus-agent-pi:local --load docker/agents
HEPHAESTUS_AGENT_IMAGE_REFERENCE=hephaestus-agent-pi:local \
HEPHAESTUS_LIVE_LLM_BASE_URL=... HEPHAESTUS_LIVE_LLM_API_KEY=... HEPHAESTUS_LIVE_LLM_MODEL=... \
  ./gradlew :application:liveTest --tests HephTurnLiveTest

The host reaches the test's sandbox gateway at host.docker.internal. The gateway binds a free port unless SANDBOX_API_PORT names one. If a host firewall admits containers to one port only, set SANDBOX_API_PORT to that port.

Hephaestus voice eval

server/application/src/test/resources/agent/heph-voice.live.ts sends nine questions to a real model through the conversation host. The eval composes the prompt as HephHostSpec does, from system.md and the writing guide, and a fake worker serves a small context folder. Each case asserts what an answer must never do and prints one line: eval <case> outputTokens=<n> words=<n> fieldsRead=<true|false>.

The file name does not match *.spec.ts, so vp run test:agents and vp run check do not run it. It reads the @LiveLlmTest credentials in the meta-annotation cheat sheet. Without them, every case skips. From the repository root:

HEPHAESTUS_LIVE_LLM_BASE_URL=… HEPHAESTUS_LIVE_LLM_API_KEY=… HEPHAESTUS_LIVE_LLM_MODEL=… \
  node --test server/application/src/test/resources/agent/heph-voice.live.ts

To compare another prompt, set HEPHAESTUS_LIVE_HEPH_SYSTEM_PROMPT_FILE to a file with the whole system prompt. It replaces system.md and the writing guide.

Live models for practice scripts

Two live tests run a practice program that uses every direct model slot against real models.

Test	Runs	What it proves
docker/agents/practice-scripts/live.test.ts	Custom programs through the program host, docker/agents/practice-scripts/lib/core/program-host.ts. The proxy stand-in forwards each call to the real endpoint.	Each bound slot answered every call, and one review request went through IPC. A program can return its own result. Each call names its practice and carries the bearer token.
PracticeScriptLiveLlmTest	The same fixture behind the real LLM proxy on the sandbox gateway, with the job token and the models file of the job	The proxy routes each slot and records the usage of the attempt. The end of the attempt adds one ledger row for each bound decision, embedding and reranking model.

The server test carries @LiveLlmTest. Both tests read the same variables:

Variable	Slot	Required
HEPHAESTUS_LIVE_LLM_BASE_URL, HEPHAESTUS_LIVE_LLM_API_KEY, HEPHAESTUS_LIVE_LLM_MODEL	chat	Yes. Without them, the test skips itself.
HEPHAESTUS_LIVE_EMBEDDING_MODEL	embedding, on the chat endpoint	No
HEPHAESTUS_LIVE_RERANKING_MODEL	reranking, on the chat endpoint	No
HEPHAESTUS_LIVE_DECISION_BASE_URL, HEPHAESTUS_LIVE_DECISION_API_KEY, HEPHAESTUS_LIVE_DECISION_MODEL	decision, on an OpenAI Decisions endpoint	No. Without them, the host test lets the chat model serve the decision slot through its log probabilities. The server test leaves the slot unbound.

A slot without its variables is not bound. The host test does not assert it. The server test asserts that the program reports it as not available. The chat model must return top_logprobs when no decision endpoint is set. The tests assert what the host guarantees, never the wording of a model.

To run the host test from the repository root:

HEPHAESTUS_LIVE_LLM_BASE_URL=… HEPHAESTUS_LIVE_LLM_API_KEY=… HEPHAESTUS_LIVE_LLM_MODEL=… \
  node --test docker/agents/practice-scripts/live.test.ts

To run the server test, from server/:

./gradlew :application:liveTest --tests PracticeScriptLiveLlmTest

vp run test:agents also runs live.test.ts, which skips itself without the chat variables.

GitHub live API tests

We ship a focused suite that exercises the live GitHub App installation and verifies end-to-end sync behavior (repository metadata, labels, milestones, and teams).

Prerequisites

1. Sandbox installation – the Hephaestus IntegrationTests GitHub App must be installed in a sandbox organization you control. The tests create and delete repositories, milestones, labels, and teams on each run.

2. Credentials – provide both a GitHub App private key and a Personal Access Token with the following scopes:

   • repo (full)
   • admin:org
   • read:packages

3. Local config file – copy the template that lives alongside the tests:

   cd server/application/src/test/resources
   cp application-live-local.example.yml application-live-local.yml

   Fill in the placeholders with the sandbox organization slug, the installation id, and either an inline PEM key (github.app.privateKey) or a readable privateKeyLocation. Keep this file out of version control – it is already listed in .gitignore.

   Alternatively, export the matching environment variables:

   export GH_APP_ID=2250297
   export GH_APP_INSTALLATION_ID=93512943
   export GH_APP_PRIVATE_KEY="$(cat /path/to/private-key.pem)"
   export GH_APP_PAT=ghp_xxx...             # PAT with the scopes above
   export GH_APP_ORGANIZATION=HephaestusTest

   GH_APP_INSTALLATION_ID must be exported even when the other credentials come from the local file: @LiveGitHubTest checks the environment before Spring loads that file. Missing credentials skip the suite. A successful Gradle exit alone does not prove live tests executed.

The run prints the GitHub artifacts it provisions. Clean-up is handled automatically, but if a failure interrupts execution you can safely delete any hephaestus-it-* repositories, milestones, or teams that remain in the sandbox.

Authoring new GitHub sync tests

1. Extend AbstractGitHubLiveSyncIntegrationTest (or fall back to BaseGitHubLiveIntegrationTest when repositories are not needed). These bases set up credential checks, provide the workspaceRepository, and expose helpers such as createEphemeralRepository, registerRepositoryToMonitor, createEphemeralTeam, and seedOrganizationMembers.
2. Use the supplied helpers to create and track temporary GitHub artifacts. They automatically register clean-up handlers through @AfterEach, so add new resources to the provided lists instead of implementing manual deletion logic.
3. Keep tests deterministic. Rely on databaseTestUtils.cleanDatabase() in @BeforeEach, which the base already invokes. Generate unique slugs through nextEphemeralSlug("suffix") for repositories, branches, or teams.
4. If a scenario needs extra Spring configuration, extend application-live-local.yml in server/application/src/test/resources/. The checked-in .example file documents every property. Copy it on demand and keep secrets out of version control.

Troubleshooting

• Skipping because of missing credentials – export GH_APP_INSTALLATION_ID, then check the console for the base class's App id, private-key and PAT checks.
• GitHub API rate-limit failures – the suite creates several entities per run. Prefer a dedicated sandbox organization so you do not clash with production automation limits.
• Longer runtimes – each suite bootstraps a Testcontainers PostgreSQL instance and provisions GitHub resources. Expect higher runtimes than the pure Testcontainers integration tests. Avoid running them on every PR and instead use them before releases or when touching the GitHub sync layer.

Practice cutover proof

Database-tier tests seed the complete released schema and realistic retained data before applying the cutover. Fresh-schema tests cannot replace that upgrade proof.

Focused integration tests prove account claim and absorption, result ordering, mark reverses and uncertain-write recovery. Scan tests distinguish withheld from unreviewed pairs and preserve reused membership after job deletion. They test complete inventory, the 20-percent removal boundary and comparison under a changed method. The migration procedure owns the database proof steps.