Testing Guide

Choose the tier by the behavior under test. Gradle recipes lists the commands. JUnit tags select tests independently of filenames. Use descriptive *Test and *IntegrationTest names for readers.

Unit tests

• Extend BaseUnitTest for isolated component tests.
• Use Mockito (@Mock, @InjectMocks) for dependencies.
• Keep unit tests independent of Spring contexts and Testcontainers.

Integration tests

• Extend BaseIntegrationTest for cross-component behavior.
• Reuse the shared PostgreSQL Testcontainer. The application connects as hephaestus_app, and Liquibase migrates as hephaestus_owner, as in production (Database roles). The application role cannot change the schema. A fixture that changes it uses PostgreSQLTestContainer.ownerJdbc().
• For webhook tests, deserialize recorded fixtures into the handler's event DTO.

The test profile starts no background work: no scheduled task, provider-event claimer, startup thread or LISTEN subscriber that acts on its own. hephaestus.background-work.enabled is false there, as in the specs and cds-training profiles. Thus no background write races a test or the cleanup that truncates its tables. A LISTEN connection that wakes the waits and streams of requests still runs, since the tests of those requests need it.

A test runs the work by calling it, for example ProviderEventConsumer#claimAndHandle or a sweeper's method. A test that receives over HTTP and needs claimer threads starts them with StoredEventHandling.runningClaimers and stops them after the test. A bean that starts background work carries @ConditionalOnBackgroundWork. BackgroundWorkArchTest finds each such start and fails without the condition.

The integration tier fails when a virtual thread is pinned to its carrier for 20 ms or more. On Java 21, a virtual thread that blocks under a monitor keeps its carrier. A few such threads can stall every virtual thread, the waiters of a connection pool among them. The JDK records each such park, and the failure prints its call path.

A common cause is a Caffeine get(key, loader) whose loader blocks, since Caffeine runs the loader under a lock. Load such an entry with CacheLoads.get. PinnedParks.during asserts in a unit test that a call pins no virtual thread.

Controller integration tests

Test controllers at the HTTP boundary so access-control configuration is exercised. Reuse the domain's integration-test base and fixture factories.

• Extend AbstractWorkspaceIntegrationTest or a service-specific base. Each controller IT then cleans the database automatically. It provides helper factories such as persistUser and exposes workspaceService for setup.
• Use WebTestClient. The full Spring Boot context already runs. Thus, tests verify access-control rules exactly as the frontend receives them (401 unauthenticated versus 403 non-admin).
• Reach for the custom security annotations (@WithAdminUser, @WithUser) together with TestAuthUtils.withCurrentUser() so the generated JWT matches our resource-server configuration.
• Assert on status code, payload shape, and repository side effects. A minimal test covers "happy path", "validation failure", and at least one access-control branch.
• Keep factories close to the domain. Prefer createWorkspace("slug", "Display", ...) to manual repository seeds. This keeps each test focused on behavior instead of boilerplate.

Example skeleton:

@AutoConfigureWebTestClient
class FooControllerIntegrationTest extends AbstractWorkspaceIntegrationTest {

    @Autowired
    private WebTestClient webTestClient;

    @Test
    @WithAdminUser
    void endpointHappyPath() {
        User owner = persistUser("owner");
        Workspace workspace = createWorkspace("slug", "Display", "login", AccountType.ORG, owner);

        webTestClient
            .get()
            .uri("/workspaces/{workspaceSlug}", workspace.getWorkspaceSlug())
            .headers(TestAuthUtils.withCurrentUser())
            .exchange()
            .expectStatus()
            .isOk();
    }
}

Cross-tenant isolation tests

Tenancy bugs silently leak data. The endpoint answers 200 with plausible data, so nothing fails visibly. Every new workspace-scoped read path gets an isolation case in CrossTenantIsolationIntegrationTest (in workspace/). Its class Javadoc documents the fixture and reason: two workspaces share an account_login and have one overlapping member.

Checklist for a new read path:

• List endpoint — assert A's collection under A's slug never contains B's rows (length() + value, not just "my row is present").
• Detail endpoint (anti-IDOR) — assert that B's entity id through A's slug returns 404. Add a positive control: A's own id returns 2xx. Thus, the 404 cannot pass against a broken endpoint.
• Query keyed on anything but workspace_id is a warning sign. Examples are an org login, a login string, or an external native id. Before shipment, scope by workspace_id or add the provider/workspace predicate.
• SCM work scoped through repository_to_monitor? A monitor stores only the repository name, and another provider can host the same name. Match the name, then call monitor_covers_repository(...) = true in JPQL or append MonitoredRepositoryScope.NATIVE_RTM_R in native SQL. Test two workspaces on different instances of the same provider kind. Give both monitors the same name and native id. Include the unconnected inverse and verify that removal preserves the other instance's repository.
• @WorkspaceAgnostic repository? The SQL statement inspector is off for it. Thus, the integration test is the only automated guard. Do not skip it.

The hephaestus.tenancy.enforcement: throw setting in application-test.yml fails tests that read a workspace_id-bearing table without a workspace_id predicate. This safety net cannot see @WorkspaceAgnostic queries or string-keyed scopes. Thus, HTTP-level assertions remain necessary.

Webhook fixtures and tooling

Reusable webhook JSON lives in server/application/src/test/resources/github and …/gitlab, named after the event it records. Load one with a ClassPathResource.

Available examples include label.created, repository.created, create, push, and more – consult the folder before recording new payloads.

Before you commit a recording, replace each person's identity with a synthetic one of the same shape. This includes the name, username, email address, numeric user ID and avatar URL. Use one synthetic identity for each person across all fixtures, so that a lookup that the tests exercise still matches. The test organization's own accounts (HephaestusTest and the maintainer account that recorded the payloads, FelixTJDietrich) can stay.

To record a GitHub event, add a temporary repository webhook in the HephaestusTest organization that subscribes to that event only. Its URL does not need to answer. Cause the event, and then save each delivery's request.payload from gh api repos/<owner>/<repo>/hooks/<hook_id>/deliveries/<delivery_id>. Delete the webhook and whatever you created when you are done. The issue_dependencies fixtures were recorded this way.

A fixture ending in .derived.json is not a recording. It starts with a recorded payload. For fields without a recordable event in the test organization, it uses the provider's documented payload. Replace it with a recording when one exists.

Each row lists every field the fixture adds, changes or removes against its recorded base.

Fixture	Recorded base	Fields that differ
github/pull_request.review_requested.reviewer.derived.json	pull_request.review_requested, a team request	requested_team replaced by requested_reviewer, a user, since GitHub sends one or the other (octokit/webhooks example). pull_request.requested_reviewers lists that user and the Copilot reviewer bot as /users/copilot-pull-request-reviewer[bot] reports it. pull_request.requested_teams is as recorded.
gitlab/merge_request.update.reviewers.derived.json	merge_request.update	Adds reviewers with each state (approved, requested_changes, reviewed, unreviewed) and re_requested: false, assignees, and changes.assignees (none before). Removes changes.labels and changes.updated_by_id. Moves object_attributes.updated_at and changes.updated_at on to 19:05:00. No changes.reviewers: nobody's state changes in this hook. It carries all four states. Before 19.3, a review or request for changes runs no merge request hook of its own.
gitlab/merge_request.update.review_rerequested.derived.json	merge_request.update	The re-request's own hook: adds reviewers as in reviewers.derived but with user1 unreviewed and re_requested: true, assignees, and changes.reviewers from the reviewers.derived list to this one, as the only changes entry. Moves object_attributes.updated_at on to 19:08:00, which the re-request's system note touches before the hook runs (request_review_service.rb).
gitlab/merge_request.update.reviewers_removed.derived.json	merge_request.update	Adds changes.reviewers from the review_rerequested list to none and changes.assignees from ab12cde to none, with no reviewers or assignees key, as GitLab leaves them out when there are none (issuable.rb). Removes changes.labels and changes.updated_by_id. Moves object_attributes.updated_at on to 19:15:00, changes.updated_at from 19:08:00.
gitlab/merge_request.update.reviewers_stateless.derived.json	merge_request.update	A hook from GitLab before 18.6, whose reviewers carry User#hook_attrs only (issuable.rb): adds reviewers as in reviewers.derived without state and re_requested and without user3, assignees, and changes.reviewers from the four to the three. Removes changes.labels and changes.updated_by_id. Moves object_attributes.updated_at on to 19:06:00 and changes.updated_at from 19:05:00 to 19:06:00.

The GitLab fixtures other than reviewers_stateless follow GitLab 18.6 and later, which states each reviewer's state and re_requested (reviewers_hook_attrs) and lists changes.reviewers whenever those attributes change (hook_reviewer_changes). From 19.3 a submitted review runs merge request hooks of its own as well (update_reviewer_state_service.rb). The GitLab fields follow merge request events.

Gradle recipes

From server/:

./gradlew :application:test                             # Unit tests
./gradlew :application:architectureTest                 # ArchUnit and Modulith
./gradlew :application:verification                     # Unit, architecture and coverage floors
./gradlew :application:integrationTest                  # Spring and Testcontainers
./gradlew :application:test --tests SomeTest            # One unit test class
./gradlew :application:test --tests 'SomeTest.method'    # One unit test method
./gradlew :application:bootJar                          # Executable JAR, without running tests

Generation and compilation dependencies are automatic. Test tasks always execute when requested. Compilation remains incremental and cacheable. --no-build-cache prevents output restoration. Combine it with clean when measuring a cold build.

The shared integration PostgreSQL container runs the docker/postgres image. Locally, each test JVM builds that image from its Dockerfile. CI integration jobs instead pull the image that CI published for the commit their images alias from. They name it in HEPHAESTUS_TEST_POSTGRES_IMAGE. When a change edits the image, or no published image exists, CI builds the Dockerfile once per job before the tests start. That build downloads packages, as a local build does.

Contexts built on BaseIntegrationTest reach GitHub's GraphQL endpoint through ScriptedGitHub. The application's own client, senders, schedulers and listeners run unchanged, and only the HTTP exchange answers from the script. A test reads what the application posted from ScriptedGitHub.threads(), keyed by repository and number. A GitHub document that the script does not answer fails as a GraphQL error.

No JVM client in a test resolves a real host. OfflineNetwork replaces InetAddress host-name resolution in every test JVM. A child process such as git, a native DNS client or a container uses its own resolver, so the guard does not cover it. Loopback names and the names the host gives its own addresses resolve. A name reserved by RFC 2606 or RFC 6761 fails as an unknown host. Any other name is refused and recorded, and OfflineNetworkExtension fails the test class that tried it.

A reverse lookup of a public address fails too. It is recorded unless one of the host's own interfaces carries the address. A test whose code only classifies an address declares the name with OfflineNetwork.answer. The live tier resolves every name.

The test database starts with silent mode engaged, as a new instance does. A test that asserts a posted copy turns silent mode off and restores it afterwards.

The shared integration PostgreSQL container keeps its ephemeral data directory on tmpfs. Database cleanup still truncates every application table transactionally. SQL constraints and isolation checks remain enabled. This container does not prove storage durability across container restarts.

vp run test:server:selection runs dedicated Gradle/JUnit inventory tasks without executing tests. They use the same tier and shard selectors as the execution tasks. The command proves that non-live tests belong to a required tier and that integration shards are disjoint and complete. Required PR verification and vp run verify run this proof before the real suites. Discovery writes only to application/build/test-selection/, leaving execution reports and coverage untouched.

Those discovery reports are not evidence that tests passed. Execution tasks do not accept the removed -PtestSelection property. Use the selection command for discovery.

Web app requests

MSW answers every request of a web app test (webapp/src/mocks/handlers.ts). A request that no handler answers gets a network error and fails the test that sent it. Thus, no test request reaches a real host, such as another checkout's server on the API port.

• Answer each request in the test that depends on it. Use a fixture in the wire shape from webapp/src/mocks/fixtures.
• Add a shared default only for a read that every signed-in page sends. Its answer must be neutral, such as no waiting product feedback.
• /workspaces and members/me have no default. Each test states the reader's workspaces and role.

webapp/src/test/unanswered-request-debt.ts lists the test files that do not fail on an unanswered request yet. The list only shrinks. A listed file that runs in full with every request answered fails until you remove its entry.

Flaky tests

A flaky test fails and then passes on the same commit. CI does not retry Gradle or Vitest tests. Playwright retries a failed test once to record a trace, and then it fails the run as flaky. You can rerun a failed job, but the rerun does not hide the flake. The weekly flaky test report lists every job that passed only on a rerun. Maintainers read it each week.

When a test flakes:

1. Open an issue for the test and assign an owner.
2. The owner finds the cause and fixes it, or deletes a test that asserts nothing new. Do not add a retry, a longer timeout, or a sleep to hide it.
3. Quarantine the test only while the flake blocks merges and the cause is not fixed yet. The skip names the issue, the owner's GitHub login and an end date at most 14 days ahead. In Java, write @Disabled("quarantine #123 @owner until 2026-11-07: free-port race"). In TypeScript, put // quarantine #123 @owner until 2026-11-07: free-port race on the line directly above it.skip(.
4. On the end date, the owner fixes or deletes the test.

vp run check fails on a skip that names no issue, owner and end date, and on an expired quarantine. A skip that depends on the environment, such as missing live credentials, is not a quarantine.

A local test server does not own its port. Another process on the host can send a request as soon as the port opens. Check the method, path or credential that the test sends. Answer any other request with a 404 or a 401, and do not count it.

Live external-service tests

Live tests use real provider APIs, LLM endpoints or sandbox containers. They carry @Tag("live") and are excluded from every other tier. From server/, explicitly opt in with:

./gradlew :application:liveTest

Use --tests ClassName to select a suite. An unmatched filter fails.

To send only some catalogue criteria cases to the model, name their id values from src/test/resources/practices/criteria-cases.json:

./gradlew :application:liveTest -PcriteriaCases=dependency-label,dependency-import-without-declaration-edit

The property first runs the selected fixture-staging checks, then those cases of PracticeRunnerLiveLlmTest and no other live test. Each case is reported under its id, not its position in the file. A blank, repeated or unknown id fails the run instead of sending every case. Without the property, the live tier runs every case.

To also limit the upstream model requests that each selected case sends, add the optional criteriaRequestLimit property:

./gradlew :application:liveTest -PcriteriaCases=dependency-label -PcriteriaRequestLimit=12

The limit applies to each case. Each forwarding attempt reserves one slot, including retries and compaction requests. Failed or cancelled attempts keep their slots. A request past the limit is refused, and that case fails, even if its result matches.

The limit needs criteriaCases and a positive integer. Without the property, this harness adds no request-count limit. Record provider token usage separately.

To measure the same cases with other model settings, add one or more of these optional properties:

./gradlew :application:liveTest -PcriteriaCases=dependency-label \
  -PcriteriaReasoningEffort=HIGH -PcriteriaMaxOutputTokens=32768 -PcriteriaContextWindow=131072

Property	Value	Default
criteriaReasoningEffort	A reasoning effort: NONE, MINIMAL, LOW, MEDIUM, HIGH, XHIGH or MAX. The names are case-sensitive.	none sent
criteriaMaxOutputTokens	A positive integer, less than the context window	4096
criteriaContextWindow	A positive integer	131072

Each property needs criteriaCases. Gradle refuses a token count that is not a positive integer before any test starts. The live test refuses a lower-case or unknown reasoning effort, and an output allowance that does not fit the context window, before any model request. A live test that is skipped because no credentials are set checks none of these values. Each case prints the model settings it uses.

First check the same selection without credentials or a model:

./gradlew :application:integrationTest -PcriteriaCases=dependency-label,dependency-import-without-declaration-edit

This runs only the selected cases in PracticeCriteriaCaseStagingIntegrationTest. Each invocation stages the evidence and shipped criteria used by the live harness. It proves selection and fixture staging, not model outcomes.

Gradle's test dry run does not execute a parameterized method's argument source. Use the staging results to count selected case IDs. A dry-run report cannot prove that count. The live credential gate remains in effect.

Do not combine criteriaCases with HEPHAESTUS_INTEGRATION_SHARD. The tasks reject that combination instead of adding unrelated shard tests.

Meta-annotation cheat sheet

Use one of these composite annotations on the test class (or method) rather than wiring @Tag("live") + @EnabledIfEnvironmentVariable by hand. They keep gates uniform and put the env-var contract in one place.

Annotation	Gate	Use for
@LiveLlmTest	HEPHAESTUS_LIVE_LLM_API_KEY	Tests that drive a real OpenAI-compatible endpoint. Set HEPHAESTUS_LIVE_LLM_BASE_URL, HEPHAESTUS_LIVE_LLM_API_KEY, and HEPHAESTUS_LIVE_LLM_MODEL explicitly. Read through LiveLlmCredentials.fromEnv().
@LiveGitHubTest	GH_APP_INSTALLATION_ID	Tests that hit github.com through the installed GitHub App. BaseGitHubLiveIntegrationTest runs an extra runtime check on private-key material.
@LiveGitLabTest	HEPHAESTUS_LIVE_GITLAB_TOKEN_FILE	GitLab CE commit identity re-resolution.

The annotations use Execution(SAME_THREAD). Gradle also runs one test JVM at a time.

GitLab commit identity proof

GitLabCommitIdentityLiveTest uses a disposable GitLab CE instance and the normal test database. It creates one user and a private group tree per case. It schedules that tree for deletion and permanently deletes the user, even if group cleanup fails.

The setup credential needs the GitLab api scope and instance-admin access to create test users and group tokens. The Hephaestus connection uses a disposable, non-admin group token. The setup token is never stored in the connection. Keep it in a private file and set HEPHAESTUS_LIVE_GITLAB_TOKEN_FILE to that path. Set HEPHAESTUS_LIVE_GITLAB_URL to the fixture's HTTPS origin.

Set hephaestus.integration.gitlab.default-server-url to the same origin for the test JVM. For a private instance, admit only its host or address with hephaestus.scm.egress.private-hosts. Use its current CA in the JVM truststore, with normal hostname and certificate verification. The fixture hostname must resolve to its service address, rather than loopback.

HEPHAESTUS_LIVE_GITLAB_URL=https://gitlab-fixture.example.test \
HEPHAESTUS_LIVE_GITLAB_TOKEN_FILE=/private/fixture-token \
./server/gradlew -p server :application:liveTest --tests '*GitLabCommitIdentityLiveTest' \
  -PtestJvmArgs='-Dhephaestus.integration.gitlab.default-server-url=https://gitlab-fixture.example.test -Dhephaestus.scm.egress.private-hosts=gitlab-fixture.example.test -Djavax.net.ssl.trustStore=/private/fixture-trust.p12 -Djavax.net.ssl.trustStorePassword=changeit'

The example truststore holds public certificates only. The test calls the configured repair job and respects its one-minute ShedLock minimum between cases. It does not enable background scheduling in the test profile.

The cases rename a user, change an email and remove the old address, and transfer a project between subgroups. The transfer case fetches the project by its stable ID, persists its reported path, then reconciles the monitor, as the scheduler does. Each case checks the queue result, changed commit link, correction receipt and matching Activity actor. The fixture waits for an ID read to confirm the new namespace.

A successful transfer response describes the project before its asynchronous transfer. GitLab retains scheduled group deletions for its configured retention period (Groups API). The provider mutations use GitLab's Users API, Commits API and project transfer API.

Authoring a new live test

1. Pick the correct meta-annotation. If the environment variable does not exist yet, add an annotation alongside the existing provider annotations. Do not reuse one with a misleading name.
2. Put the test under server/application/src/test/java/.../live/. The directory marker keeps live and non-live tests in different code-review buckets.
3. Read credentials from LiveLlmCredentials.fromEnv() or the provider helpers. Never inline secrets or commit local credential files.
4. Per-test wall-clock timeouts: 90s is reasonable for a single conversation turn. 5 minutes for a full practice run.

Kubernetes sandbox isolation

The platform gate qualifies each actual SandboxTemplate on the installed cluster. It starts synthetic sandboxes and verifies Pod settings, the runtime boundary, and network isolation. It reaches no model provider.

Reserve the shared qualification release, start it with vp run dev:kubernetes, then run:

KUBECONFIG="$HOME/.hephaestus/kubernetes-cluster/kubeconfig" \
  node scripts/kubernetes-cluster.ts verify

The Kubernetes CI job runs verify after installation and upgrade. scripts/sandbox-isolation.ts owns the checks and accepts individual template and address arguments. The old Java tools-Pod live suite and its image overlay are removed. The provider live tests remain separate.

Deterministic sandbox release qualification

Release preflight and the release workflow run the native review runner in the selected agent image. Qualification must pass before the release is published. Run the same qualification on a local Linux Docker host:

node scripts/sandbox-security.ts --image <agent-pi-reference> \
  --application-image <application-server-reference>

Use image digests from the same candidate release for both references. Qualification reads agent/ from the Paketo image's /workspace/runner.jar without starting the image. The qualification report records both image references.

Docker's default bridge must permit traffic between containers. The probe collector publishes no host port. The replay provider runs on loopback inside each container. No live model or provider credential is required. The harness gives each workspace a separate Docker container with a read-only root, no network, and bounded memory and processes.

The corpus injects instructions into a pull request body, a comment, and a repository file. The replay reads the injected text, then attempts cross-workspace reads and a network post. The gate scans model requests, tool transcripts, review outputs, and process logs for another workspace's planted marker. A positive control requires the marker to reach its own workspace's model requests.

Two mutation controls expose the other workspace's files and permit network access. The gate must detect the resulting read and post before it can pass.

A failure stops publication. Detailed evidence stays in the local temporary directory, whose path the command reports. Do not publish those files before review. CI retains a summary with no captured content. This qualification tests the Docker host's native image architecture and the container and tool boundaries.

It does not test the Kubernetes SandboxTemplate. The Kubernetes sandbox isolation gate qualifies the production templates. The database tenancy tests remain required.

Offline feedback quality evaluation

The feedback quality harness replays stored evidence and exact outputs without provider credentials. It checks selected claims and reports provisional labels, human judgments, and advisory rubric ratings separately. The labelling guide gives the independent human review procedure.

Live review runner image

The live review suites use the prebuilt agent image, with its SDK and practice toolkit. Set HEPHAESTUS_LIVE_AGENT_IMAGE to the image to test. Otherwise, prepare hephaestus/agent-pi:live first. The harness never pulls it or installs dependencies during a run.

The runner executes in one non-root container with a read-only root, its owned fixture workspace, and temporary storage. Testcontainers forwards only the stand-in proxy port through host.testcontainers.internal. The container receives the scoped proxy token. It receives no provider key. The harness removes its container after success, failure, or timeout.

This checks real tools and model integration. The separate Kubernetes gate checks production network isolation.

Hephaestus voice eval

The eval in server/application/src/test/resources/agent/heph-voice/ sends questions to a real model through the conversation runner. The eval composes the prompt as HephTurnContexts does, from system.md and the writing guide, and a fake server serves a small context folder. Each case asserts what an answer must never do and prints one line: eval <case> outputTokens=<n> words=<n> fieldsRead=<true|false>.

The case file names do not match *.spec.ts, so vp run test:agents and vp run check do not run them. The eval reads the @LiveLlmTest credentials in the meta-annotation cheat sheet. Without them, every case that asks the model skips. The oracle checks still run. From the repository root:

HEPHAESTUS_LIVE_LLM_BASE_URL=… HEPHAESTUS_LIVE_LLM_API_KEY=… HEPHAESTUS_LIVE_LLM_MODEL=… \
  node --experimental-test-module-mocks --test --test-concurrency=1 'server/application/src/test/resources/agent/heph-voice/*.live.ts'

--test-concurrency=1 runs one case file at a time. By default, node --test runs the files in parallel, and parallel model turns can hit the provider's rate limit or the turn timeout.

Each case, or each group of cases for one finding, is its own <id>.live.ts file. harness.ts holds the model setup, the shared context folders and the oracles. To add a case, add a file. Do not append it to the file of another case. The glob in the command above finds each case file, so no list of cases needs an update.

To compare another prompt, set HEPHAESTUS_LIVE_HEPH_SYSTEM_PROMPT_FILE to a file with the whole system prompt. It replaces system.md and the writing guide.

Live practice script review

PracticeScriptLiveLlmTest runs a stored script behind the real LLM proxy on the sandbox gateway, with the job token. It proves that the script opens one review, makes no direct model call, and adds no usage row of its own. It carries @LiveLlmTest and needs HEPHAESTUS_LIVE_LLM_BASE_URL, HEPHAESTUS_LIVE_LLM_API_KEY and HEPHAESTUS_LIVE_LLM_MODEL. Without them, it skips itself. It asserts what the host guarantees, never the wording of a model.

To run it, from server/:

./gradlew :application:liveTest --tests PracticeScriptLiveLlmTest

GitHub live API tests

We ship a focused suite that exercises the live GitHub App installation and verifies end-to-end sync behavior (repository metadata, labels, milestones, and teams).

Prerequisites

1. Sandbox installation – the Hephaestus IntegrationTests GitHub App must be installed in a sandbox organization you control. The tests create and delete repositories, milestones, labels, and teams on each run.

2. Credentials – provide both a GitHub App private key and a Personal Access Token with the following scopes:

   • repo (full)
   • admin:org
   • read:packages

3. Local config file – copy the template that lives alongside the tests:

   cd server/application/src/test/resources
   cp application-live-local.example.yml application-live-local.yml

   Fill in the placeholders with the sandbox organization slug, the installation id, and either an inline PEM key (github.app.privateKey) or a readable privateKeyLocation. Keep this file out of version control – it is already listed in .gitignore.

   Alternatively, export the matching environment variables:

   export GH_APP_ID=2250297
   export GH_APP_INSTALLATION_ID=93512943
   export GH_APP_PRIVATE_KEY="$(cat /path/to/private-key.pem)"
   export GH_APP_PAT=ghp_xxx...             # PAT with the scopes above
   export GH_APP_ORGANIZATION=HephaestusTest

   GH_APP_INSTALLATION_ID must be exported even when the other credentials come from the local file: @LiveGitHubTest checks the environment before Spring loads that file. Missing credentials skip the suite. A successful Gradle exit alone does not prove live tests executed.

The run prints the GitHub artifacts it provisions. Clean-up is handled automatically, but if a failure interrupts execution you can safely delete any hephaestus-it-* repositories, milestones, or teams that remain in the sandbox.

Authoring new GitHub sync tests

1. Extend AbstractGitHubLiveSyncIntegrationTest (or fall back to BaseGitHubLiveIntegrationTest when repositories are not needed). These bases set up credential checks, provide the workspaceRepository, and expose helpers such as createEphemeralRepository, registerRepositoryToMonitor, createEphemeralTeam, and seedOrganizationMembers.
2. Use the supplied helpers to create and track temporary GitHub artifacts. They automatically register clean-up handlers through @AfterEach, so add new resources to the provided lists instead of implementing manual deletion logic.
3. Keep tests deterministic. Rely on databaseTestUtils.cleanDatabase() in @BeforeEach, which the base already invokes. Generate unique slugs through nextEphemeralSlug("suffix") for repositories, branches, or teams.
4. If a scenario needs extra Spring configuration, extend application-live-local.yml in server/application/src/test/resources/. The checked-in .example file documents every property. Copy it on demand and keep secrets out of version control.

Troubleshooting

• Skipping because of missing credentials – export GH_APP_INSTALLATION_ID, then check the console for the base class's App id, private-key and PAT checks.
• GitHub API rate-limit failures – the suite creates several entities per run. Prefer a dedicated sandbox organization so you do not clash with production automation limits.
• Longer runtimes – each suite bootstraps a Testcontainers PostgreSQL instance and provisions GitHub resources. Expect higher runtimes than the pure Testcontainers integration tests. Avoid running them on every PR and instead use them before releases or when touching the GitHub sync layer.

Practice cutover proof

Database-tier tests seed the complete released schema and realistic retained data before applying the cutover. Fresh-schema tests cannot replace that upgrade proof.

Focused integration tests prove account claim and absorption, result ordering, mark reverses and uncertain-write recovery. Scan tests distinguish withheld from unreviewed pairs and preserve reused membership after job deletion. They test complete inventory, the 20-percent removal boundary and comparison under a changed method. The migration procedure owns the database proof steps.