Work Accommodation Review — Regression Test Plan

An internal engineering reference. It covers the seven work-accommodation-review (WAR) changes that landed on 2026-08-18, what still needs verifying against real data, and what to create, update and delete in Tenzo and the Bullhorn dev sandbox to do it. Background on the mechanism itself lives in ATS Stage Routing — read the review-overlay and regrade-reconcile notes there first. This page is the test plan, not the design.

Scope: what shipped

PR	What it changes
#8776	Regrade outcome reconcile. A regrade that re-derives a knockout verdict now moves the application stage, via a restricted post-commit profile.
#8810	WAR job setting moved from Interview Behavior to Recruiting Results; the feature flag is retired and the job setting is the sole enablement source.
#8822	`never_asked` becomes structural rather than text-matched. The exclusion reaches the knockout gate, weighted average, both binary lifts, the bucket rescale and the completion rate. Completeness is re-checked when a regrade clears a knockout.
#8829	The durable `work_accommodation_review_needed` column is dropped. The `PENDING_ACCOMMODATION_REVIEW` overlay stage is now the only marker. Listing badge and filter derive from stage plus cached ATS stage.
#8842	Shared stage-ownership policy, deduped scoring helpers, CAS invariant stated. Adds the `never_asked` back-fill job.
#8843	Disclosure attribution fix — both speakers now reach the detector. Rejoin-event leak fixed in three places.
#8857	TEN-809. Release presence check widened to the AI-interview pass/fail targets; the stage comment is suppressed unless the move actually wrote a new stage.

Execution status

Group	Coverage	Status
D — detection (#8843)	attribution, rejoin-event leak, unredacted wiring, interviewer-only guard	Verified against the real model and real dev transcripts
Core WAR unit tests	grades, application_stage, review_stage_target, grading	828 passed, 0 failed
A — overlay and release (#8829, #8857)	7 scenarios	Not run — needs Postgres
B — regrade reconcile (#8776, #8822)	6 scenarios	Not run — needs Postgres
C — never_asked (#8822, #8842)	4 scenarios	Not run — needs Postgres
E — settings, listing, ownership (#8810, #8829, #8842)	4 scenarios	Not run — needs Postgres

Group D’s attribution result, 5 runs per arm: production shaping attributed the disclosure correctly 5/5; the pre-fix shaping attributed it to a decoy question 4/5. Re-runnable via dev_scripts/war_detection_attribution_ab.py.

Groups A, B, C and E cannot run from a Claude Code web session. All outbound traffic from an Anthropic-hosted cloud environment passes through an HTTP/HTTPS proxy whose allowlist is domain-based with no port concept, so Postgres on 5432 has no route — verified by observing that port 443 to the same host connects while 5432 is silently dropped. Run these locally, or from a self-hosted environment where egress leaves through your own network boundary. Cosmos and Kombo/Bullhorn are HTTPS and remain reachable from a web session.

Dev sandbox facts

Confirmed live against the dev Bullhorn integration.


Org	`9f49b6d1-b019-468e-a7f6-3faea6e7fc2d`
Integration	`bullhorn:BUH2raYRWJy2ezE72xfyZwkX` — plain Bullhorn, not Innova
`pending_accommodation_review_stage`	not configured — this is exactly the TEN-809 shape
`ai_interview_pass_stage` / `..._fail_stage`	`Candidate Interested` / `Candidate Rejected`
`stage_routing_rules`	not persisted; rules synthesized from flat fields
Thresholds	pass score 70, completion rate 0.8
WAR enablement	off everywhere — no admin group, all 40 jobs `{overrideDefaults: false, enabled: false}`

Sandbox stages available:

New Submission, CV Sent, Shortlisted, Candidate Interested, Candidate Rejected, Client Rejected, Consultant Rejected, Reference Check, Offer Extended, Offer Rejected, Tenzo Test Stage, New Test Stage, 1st–3rd Interview, Final Interview, Manager Review, Hired, Submitted, Placed, New Lead

. Because the sandbox integration is plain Bullhorn, the Innova VRA presence path cannot be exercised here with real data. It needs an Innova org or stays unit-tested only.

Group A — overlay and release

#	Scenario	Expected
A1	Flagged interview completes	CCI on `PENDING_ACCOMMODATION_REVIEW`, flagged non-INFO question holds provisional full credit (INFO questions keep evaluated grade without provisional inflation), ATS stays on `Candidate Interested`, exactly one note posted
A2	Reviewer resolves last item, verdict unchanged	Releases to `CALL_COMPLETED`; ATS move `SKIPPED`; no second note
A3	Reviewer scores the flagged required question 0	Releases to `FAILED_KNOCKOUT`; ATS moves to `Candidate Rejected`; note posted
A4	TEN-809 regression — recruiter moves the submission to `2nd Interview`, then reviewer resolves	ATS untouched, no note, Tenzo hold still resolves
A5	Parking-stage variant — set `pending_accommodation_review_stage` to `Tenzo Test Stage`, rerun A1–A4	Overlay parks in ATS; release unparks and comments
A6	Overlay gate	`FAILED_KNOCKOUT` never overlaid; `INCOMPLETE`/`RESCHEDULED` stays on the retry path and re-fires next completed attempt
A7	Kombo staleness window	Move in Bullhorn, resolve immediately, measure how long the stale snapshot persists (see finding 1)

Group B — regrade reconcile

#	Scenario	Expected
B1	Regrade turns `COMPLETED` into a knockout	Stage `CALL_COMPLETED` → `FAILED_KNOCKOUT`; post-commit ATS move to `Candidate Rejected`
B2	Regrade clears a knockout, outro reached	Back to `CALL_COMPLETED` and `Candidate Interested`
B3	Regrade clears a knockout, outro not reached	`INCOMPLETE`; no stage move; persisted verdict untouched
B4	Bulk-regrade safety	Dispositioned, opted-out and outreach-stage candidates are not resurrected
B5	Application under WAR hold	Left to the release path, not moved by the reconcile
B6	Post-commit ATS failure	Reported as a failed reconcile; `result_edited_since_ats_push` keeps manual re-push available

Group C — never_asked

#	Scenario	Expected
C1	Add a question to the job, regrade an older call	Back-filled row carries `never_asked`; score, completion rate, binary lifts and knockout verdict all unchanged
C2	Reviewer types “Not asked” as free-form answer text	Does not clear a genuine knockout
C3	`GRADE_ONLY` grading failure	Exceptions or returned placeholders that can affect outcomes abort without persistence; purposeful skips and non-required `INFO` placeholders remain harmless
C4	`backfill_never_asked_marker` dry run over legacy rows	Recognises the legacy shape without touching anything else

Group E — settings, listing, ownership

#	Scenario	Expected
E1	WAR toggle under Recruiting Results, org and job override	Works with the feature flag gone
E2	Listing badge vs `needs_work_accommodation_review_only` filter	Agree, including unsynced and JSON-`null` `application_stage` rows
E3	Two calls on one application	Older call’s release or regrade must not overwrite the newer call’s stage
E4	Migration `42dad1876e35`	Dev DB on head; nothing reads the dropped column

What to create, update and delete

Approved scope. Nothing touches production, any other dev org, or any of the 40 existing jobs.

Tenzo (dev)

Action	Object
Create	Job `WAR Regression 2026-08-19`, linked to sandbox Bullhorn req `Data Engineer` (remote id 1634)
Create	Script with ~6 questions: one required knockout, one that invites a scheduling or medical answer, one added mid-run for the `never_asked` case
Create	5 candidates and applications, non-test — `regrade_terminal_outcome` exempts test calls, so test candidates cannot exercise Group B
Create	5 completed AI interviews across the scenarios
Update	WAR setting on the test job only: `overrideDefaults: true, enabled: true`. Not org-wide
Update	Org ATS config `bullhorn:BUH2raYRWJy2ezE72xfyZwkX` — add `pending_accommodation_review_stage: "Tenzo Test Stage"` for A5 only, after snapshotting the doc, then restore byte-for-byte
Update	Grades via the reviewer resolve path; `update_scores` regrades for Groups B and C
Delete	nothing

Bullhorn dev sandbox

Action	Object
Create	5 candidates and job submissions on `Data Engineer`
Update	Submission statuses, driven by Tenzo: `New Submission` → `Candidate Interested` / `Candidate Rejected`. Manually move one to `2nd Interview` for A4 and one to `Tenzo Test Stage` for A5
Update	Tenzo writes interview-result notes on submissions — note count is the assertion for the duplicate-note fix
Delete	nothing. Restore statuses to `New Submission` afterwards. Notes accumulate — Kombo has no clean delete, so the sandbox keeps roughly ten test notes

Open findings & Resolutions

The TEN-809 fix and Kombo sync. review_release_stage_presence reads Kombo’s synced application snapshot. This aligns with all other ATS transitions in Tenzo (AI_INTERVIEW_PASSED, AI_INTERVIEW_FAILED, HUMAN_REVIEW, etc.) which read through the cached application layer. Retained standard cached behavior.
Stuck review hold monitoring. Hardened exception logging in work_accommodation_review_stage_sync.py and work_accommodation_review_stage_release.py to consistently emit error.kind="WorkAccommodationReviewReleaseTransitionFailed".
Innova blank VRA field. Retained safe fail-closed behavior to prevent unexpected downstream ATS writes without customer-specific product alignment.
Workflow step IDs vs stage names target set. Verified non-issue for real ATS setups.
ATS-less orgs release note. Confirmed intended behavior: orgs without ATS integration record resolutions locally within Tenzo.

Running this

Groups A, B, C and E need a working Postgres connection, so run them from a local checkout of claude/war-updates-review-test-5b7cuu — or any branch carrying these changes — rather than a cloud session. Group D needs no database. dev_scripts/war_detection_attribution_ab.py runs the attribution A/B against the live model from anywhere with LLM credentials, and the rest of Group D is already covered by unit tests: tests/unit/caller/test_work_accommodation_review.py, tests/unit/caller/test_flagged_statement_detection.py, tests/unit/post_process_call/test_broadcast_display_transcript.py and tests/unit/post_process_call/test_accommodation_review_unredacted_input.py.

./bin/test reaches a shared Redis on port 6380 via the committed .env.test. In any sandboxed environment without that egress, roughly 32 post_process_call tests fail on a 30-second per-test timeout rather than on anything real. Check the traceback for a blocked sock_connect before treating those failures as regressions.

Internal Documentation

Modules

Fraud Risk

Outreach

ATS Implementations

Work Accommodation Review — Regression Test Plan

Work Accommodation Review — Regression Test Plan

Scope: what shipped

Execution status

Dev sandbox facts

Group A — overlay and release

Group B — regrade reconcile

Group C — never_asked

Group E — settings, listing, ownership

What to create, update and delete

Tenzo (dev)

Bullhorn dev sandbox

Open findings & Resolutions

Running this

​Work Accommodation Review — Regression Test Plan

​Scope: what shipped

​Execution status

​Dev sandbox facts

​Group A — overlay and release

​Group B — regrade reconcile

​Group C — never_asked

​Group E — settings, listing, ownership

​What to create, update and delete

​Tenzo (dev)

​Bullhorn dev sandbox

​Open findings & Resolutions

​Running this

Work Accommodation Review — Regression Test Plan

Scope: what shipped

Execution status

Dev sandbox facts

Group A — overlay and release

Group B — regrade reconcile

Group C — never_asked

Group E — settings, listing, ownership

What to create, update and delete

Tenzo (dev)

Bullhorn dev sandbox

Open findings & Resolutions

Running this