# Practice model cutover checks

The [release migration guide](https://github.com/hephaestus-build/Hephaestus/blob/main/MIGRATION.md) owns the upgrade order, historical-proof rule, script preparation, and rollback boundary. The [production runbook](/admin/upgrade.md) owns isolation and recovery.

## Rehearse the pilot upgrade

1. Follow [Validate the actual production data](/admin/upgrade.md#validate-the-actual-production-data) on a verified pilot restore.
2. Use the exact candidate release without diagnostic changes.
3. Record the migration duration, locks, disk use, and WAL growth.
4. Complete the history checks below before the production outage.
5. After a failure or repair, repeat the rehearsal on a fresh restore.

One synthetic rehearsal with 40,000 work items and 896,019 results took 15 minutes 1.214 seconds. This exceeded ten times the assumed three-month pilot volume. The run added diagnostic source indexes. Its duration does not prove production performance for an unmodified release. Use the measured pilot restore to set the outage budget.

## Check retained pilot history

Keep the [silent deployment barriers](/admin/production-operations-runbook.md#silent-deployment-checklist) in place throughout these checks. Do these checks before any deliberate new review or scan.

1. Open a few retained results from different practices and work items.
2. Compare their IDs, pinned versions, outcomes, text, and citations with the pre-upgrade sample under authorized source access.
3. Check developer attribution on author and reviewer results.
4. Check the exact review and parent pull request for each reviewer result.
5. Open retained feedback at each destination used by the pilot.
6. Check separate private cards through the developer's authorized view.
7. Keep private text out of operator exports and incident records.
8. Compare posted-copy receipts, URLs, and appearance times with the pre-upgrade sample.
9. Check a feedback record with both posted and Unconfirmed copies.
10. Confirm that the Unconfirmed copy stays fenced against a repeat POST.
11. Check retained incorrect-result marks, restorations, and pending corrections.
12. Check that a former member retains history without active workspace access.
13. Check an account with multiple provider identities for the combined preferences and membership role.
14. Confirm that one disabled delivery preference keeps account delivery off.
15. Compare review, job, and send records before and after startup.
16. Confirm that no new reviews or provider writes started during inspection.

The migration guide explains why retained results cannot establish current standing. A source-access restriction can withhold text without removing history. Use the [observation text contract](/admin/practice-review-operations.md#observation-text-availability) when text is unavailable. Record expected transformations and relationships instead of requiring equal counts across old and new tables. Existing send or correction recovery can change state during startup. A state change alone does not prove a provider write.

## Diagnose a refused cutover

Keep the previous signed release and verified backup available. Preserve the failed changeset identity, error, and [Liquibase diagnostics](/admin/upgrade.md#if-migration-fails). Use a restricted restore for diagnosis. Do not bypass a guard, delete product history, clear receipts, or invent source facts.

The table maps the guard categories to source records and supported actions. The queries below apply to the complete released schema before the cutover. They return identifiers and state, not feedback bodies or credentials. A clean query does not prove that every guard in its category passes.

| Guard category                          | Meaning and row diagnosis                                                                                                                                                                   | Supported action                                                                                                                                                                                                                                                       |
| --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| PostgreSQL or baseline                  | PostgreSQL 18 or the required baseline cut-point is absent. Check [release and schema](#release-and-schema).                                                                                | Follow the [baseline prerequisites](/admin/liquibase-baseline-runbook.md#before-deployment). An installation already at the complete released tail needs no repeat synchronization.                                                                                    |
| Released tail or table state            | Required released history/tables are absent, or final `item`/`result` tables already exist. Check [release and schema](#release-and-schema).                                                | Confirm the installed release and migration history. For unexplained schema changes, keep the old release and escalate. Do not force the cutover.                                                                                                                      |
| Dispatch ownership                      | A `lease_owner` or `projection_owner` remains, even after containers stop. Use the [ownership query](#dispatch-ownership-and-job-purpose).                                                  | Keep inputs paused. Let the old server's delivery recovery settle actual writes and projection. Follow the migration guide's drain/retry procedure. Never clear ownership manually.                                                                                    |
| Unsupported job purpose                 | A retained job has a non-null purpose other than `PRACTICE_REVIEW`. Use the [purpose query](#dispatch-ownership-and-job-purpose).                                                           | Keep the old release and escalate with job IDs and purpose values. Stopping containers does not convert retained jobs.                                                                                                                                                 |
| People and ownership                    | A person reference is missing, erased, nonhuman, or assigned to conflicting accounts. A sync requester can also be invalid. Use [person diagnostics](#people).                              | Use the old release's supported identity or erasure workflow only when authoritative records establish the action. For ambiguous ownership or residual erased-person history, keep the old release and escalate.                                                       |
| Item routing and reviewer ancestry      | An exact reviewer ID/parent is absent, keys disagree, a provider namespace is ambiguous, or a source crosses workspaces. Use [item diagnostics](#items-and-reviewers).                      | Reconcile mutable provider mirrors through supported source sync where authoritative facts remain. For absent or conflicting historical facts, keep the old release and escalate. Never assign a guessed reviewer or workspace.                                        |
| Proof, capture, and reuse               | Outcomes, evidence, pinned versions, capture/reuse references, or chat links cannot convert safely. Use [proof diagnostics](#proof-and-feedback).                                           | Reproduce on the restore. Keep the old release and escalate with affected IDs. Obtain a supported conversion before retry. Do not rewrite immutable proof.                                                                                                             |
| Feedback support and copies             | Support, copies, approved text, receipts, write keys, or accepted inline anchors disagree. Use [feedback diagnostics](#proof-and-feedback).                                                 | Compare retained proposals and actual provider writes under authorized read access. Keep uncertain copies fenced. Without a supported reconciliation, keep the old release and escalate. Never delete receipts or invent text.                                         |
| Scripts and current definitions         | An active/catalog/comparison-base script is unsupported, or the current definition is incomplete. The refusal identifies a practice ID or catalog slug.                                     | Follow **Custom practice scripts** in the migration guide. Save complete definitions first. For an incomplete definition without authoritative content, keep the old release and escalate.                                                                             |
| Native objects, preservation, and locks | An auth partition template is absent, a native constraint/preservation assertion fails, or lock acquisition exceeds 30 seconds. Use [native-object diagnostics](#native-objects-and-locks). | Keep writers stopped. For lock contention, identify remaining writers and repeat the rehearsal after a normal drain. For schema or preservation failures, keep the old release and escalate through [failed-migration recovery](/admin/upgrade.md#if-migration-fails). |

### Release and schema

```
BEGIN READ ONLY;

SELECT current_setting('server_version_num') AS server_version_num;

SELECT id, author, filename, exectype

FROM databasechangelog

WHERE id IN ('1791941746633-2', '1791486276211-v1-cutover');

SELECT to_regclass('public.observation') AS old_results,

       to_regclass('public.identity_link') AS old_identities,

       to_regclass('public.item') AS final_items,

       to_regclass('public.result') AS final_results;

SELECT id, locked, lockgranted FROM databasechangeloglock;

ROLLBACK;
```

The required tail uses author `hephaestus` and filename `db/changelog/1791941746633_changelog.xml`. Check the older baseline with its [own procedure](/admin/liquibase-baseline-runbook.md). Do not release a lock by hand except as [One migration at a time](/admin/runtime-roles.md#one-migration-at-a-time) allows.

### Dispatch ownership and job purpose

```
BEGIN READ ONLY;

SELECT id, workspace_id, agent_job_id,

       lease_owner IS NOT NULL AS has_lease,

       projection_owner IS NOT NULL AS has_projection_owner

FROM feedback_dispatch

WHERE lease_owner IS NOT NULL OR projection_owner IS NOT NULL;

SELECT id, workspace_id, purpose FROM agent_job

WHERE purpose IS NOT NULL AND purpose <> 'PRACTICE_REVIEW';

ROLLBACK;
```

### People

```
BEGIN READ ONLY;

SELECT provider_id, array_agg(id ORDER BY id) AS identity_ids,

       array_agg(DISTINCT account_id) AS account_ids

FROM identity_link

GROUP BY provider_id, subject, COALESCE(team_id, '')

HAVING count(DISTINCT account_id) > 1;

SELECT s.id, s.triggered_by_user_id FROM sync_job s

LEFT JOIN account a ON a.id = s.triggered_by_user_id

WHERE s.triggered_by_user_id IS NOT NULL AND a.id IS NULL;

SELECT s.id FROM person_suppression s

JOIN public_activity_objection o ON o.id = s.id;

WITH people AS (

  SELECT about_user_id AS user_id FROM observation

  UNION SELECT recipient_user_id FROM feedback

  UNION SELECT about_user_id FROM feedback

  UNION SELECT reactor_user_id FROM reaction

  UNION SELECT user_id FROM chat_thread WHERE user_id IS NOT NULL

)

SELECT p.user_id, u.provider_id, u.type, a.id AS account_id,

       a.status AS account_status, s.id AS suppression_id

FROM people p

LEFT JOIN "user" u ON u.id = p.user_id

LEFT JOIN identity_link i ON i.provider_id = u.provider_id

  AND i.subject = u.native_id::text AND COALESCE(i.team_id, '') = ''

LEFT JOIN account a ON a.id = i.account_id

LEFT JOIN person_suppression s ON s.provider_id = u.provider_id

  AND s.subject = u.native_id::text AND s.team_key = ''

WHERE u.id IS NULL OR COALESCE(upper(u.type), '') <> 'USER'

   OR a.status IN ('DELETING', 'DELETED') OR s.id IS NOT NULL;

ROLLBACK;
```

The person query samples product-history references. For other person failures, inspect the guard's source relation with the same provider/key/team comparison. Additional references include membership, preferences, review targets, event requesters, job developers, and Slack participants or consent actors. Retain disabled identities when you compare ownership. A disabled identity does not establish erasure.

### Items and reviewers

```
BEGIN READ ONLY;

SELECT id, workspace_id, job_id, artifact_id AS pull_request_id

FROM artifact_signal

WHERE signal_name = 'scm.pull_request.reviewed'

  AND revision !~ '^event~[1-9][0-9]*$';

SELECT o.id, o.workspace_id, j.workspace_id AS job_workspace_id,

       p.workspace_id AS practice_workspace_id

FROM observation o

LEFT JOIN agent_job j ON j.id = o.agent_job_id

LEFT JOIN practice p ON p.id = o.practice_id

WHERE j.id IS NULL OR o.workspace_id <> j.workspace_id

   OR o.workspace_id <> p.workspace_id;

ROLLBACK;
```

For ancestry failures, compare `pull_request_review.pull_request_id` with the exact reviewer event and job routing references. Inspect only the named routing fields in `agent_job.metadata`, not the complete document. For key conflicts, compare `artifact_kind`/`artifact_id` in the job's observations, feedback, and events within their workspaces. For monitor ambiguity, compare `repository_to_monitor` with `repository.provider_id`, the workspace organization, and the connection's provider namespace. For Slack/Outline failures, compare the referenced thread/document workspace with the released result, feedback, or event workspace.

### Proof and feedback

The first query lists groups with multiple historical results. These groups do not stop the migration. See [legacy history reasons](/contributor/practice-review-glossary.md#legacy-history-reasons) for the assignment state and retention rules. Other queries identify proof or feedback conflicts that still need diagnosis.

```
BEGIN READ ONLY;

SELECT agent_job_id, practice_id, artifact_kind, artifact_id, about_user_id,

       array_agg(id ORDER BY id) AS observation_ids

FROM observation

GROUP BY agent_job_id, practice_id, artifact_kind, artifact_id, about_user_id

HAVING count(*) > 1;

SELECT id, workspace_id FROM observation

WHERE outcome NOT IN ('MET', 'NOT_MET', 'NOT_APPLICABLE', 'UNDETERMINED')

   OR (outcome = 'NOT_MET') <> (severity IS NOT NULL)

   OR (evidence IS NOT NULL AND jsonb_typeof(evidence) <> 'object');

SELECT agent_job_id, practice_id FROM observation

GROUP BY agent_job_id, practice_id

HAVING count(DISTINCT practice_revision_id) > 1

    OR (count(practice_revision_id) > 0 AND count(practice_revision_id) < count(*));

SELECT d.id AS dispatch_id, d.feedback_id FROM feedback_dispatch d

JOIN feedback f ON f.id = d.feedback_id

WHERE d.body IS DISTINCT FROM COALESCE(f.body, '');

SELECT b.feedback_id, b.observation_id FROM feedback_observation b

JOIN feedback f ON f.id = b.feedback_id

LEFT JOIN observation o ON o.id = b.observation_id

WHERE o.id IS NULL OR o.workspace_id <> f.workspace_id;

ROLLBACK;
```

For capture/reuse failures, compare pinned `practice_revision` rows with the job's captured practices and readiness decisions. For citation failures, inspect the named keys under authorized source access. For chat-link failures, compare the cited verdict ID, message, and thread workspace. For feedback failures, compare `feedback_observation`, placements, dispatch proposals, contributors, and approval records for the affected review. Compare mapped accounts rather than requiring equal legacy provider-user IDs. Never include bodies, quoted evidence, or credentials in an escalation report.

### Native objects and locks

```
BEGIN READ ONLY;

SELECT parent_table, template_table FROM partman.part_config

WHERE parent_table = 'public.auth_event';

SELECT pid, state, wait_event_type, wait_event, pg_blocking_pids(pid) AS blockers

FROM pg_stat_activity

WHERE datname = current_database() AND pid <> pg_backend_pid();

ROLLBACK;
```

Check that the configured template relation exists. For a named constraint or preservation assertion, retain its exact error and affected relation/IDs. After transaction rollback, temporary cutover tables are unavailable. Reproduce the failure on a restore with the candidate release's exact migration. Do not change migration checksums, skip the assertion, or stop an unknown process to pass a guard.