> ## Documentation Index
> Fetch the complete documentation index at: https://f4c7a9e2d8b1-docs.tenzo.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Work Accommodation Review — Regression Test Plan
> Manual and semi-automated verification plan for the 2026-08-18 work-accommodation-review changes, with execution status and open findings.
# Work Accommodation Review — Regression Test Plan
An **internal engineering reference**. It covers the seven work-accommodation-review (WAR) changes
that landed on 2026-08-18, what still needs verifying against real data, and what to create, update
and delete in Tenzo and the Bullhorn dev sandbox to do it.
Background on the mechanism itself lives in
[ATS Stage Routing](/internal/ats/stage-routing) — read the review-overlay and regrade-reconcile
notes there first. This page is the test plan, not the design.
***
## Scope: what shipped
| PR | What it changes |
| ----- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| #8776 | **Regrade outcome reconcile.** A regrade that re-derives a knockout verdict now moves the application stage, via a restricted post-commit profile. |
| #8810 | WAR job setting moved from Interview Behavior to **Recruiting Results**; the feature flag is retired and the job setting is the sole enablement source. |
| #8822 | **`never_asked` becomes structural** rather than text-matched. The exclusion reaches the knockout gate, weighted average, both binary lifts, the bucket rescale and the completion rate. Completeness is re-checked when a regrade clears a knockout. |
| #8829 | **The durable `work_accommodation_review_needed` column is dropped.** The `PENDING_ACCOMMODATION_REVIEW` overlay stage is now the only marker. Listing badge and filter derive from stage plus cached ATS stage. |
| #8842 | Shared stage-ownership policy, deduped scoring helpers, CAS invariant stated. Adds the `never_asked` back-fill job. |
| #8843 | **Disclosure attribution fix** — both speakers now reach the detector. Rejoin-event leak fixed in three places. |
| #8857 | **TEN-809.** Release presence check widened to the AI-interview pass/fail targets; the stage comment is suppressed unless the move actually wrote a new stage. |
***
## Execution status
| Group | Coverage | Status |
| ---------------------------------------------------------- | ------------------------------------------------------------------------- | ------------------------------------------------------------ |
| **D — detection** (#8843) | attribution, rejoin-event leak, unredacted wiring, interviewer-only guard | **Verified** against the real model and real dev transcripts |
| Core WAR unit tests | grades, application\_stage, review\_stage\_target, grading | **828 passed, 0 failed** |
| **A — overlay and release** (#8829, #8857) | 7 scenarios | Not run — needs Postgres |
| **B — regrade reconcile** (#8776, #8822) | 6 scenarios | Not run — needs Postgres |
| **C — never\_asked** (#8822, #8842) | 4 scenarios | Not run — needs Postgres |
| **E — settings, listing, ownership** (#8810, #8829, #8842) | 4 scenarios | Not run — needs Postgres |
Group D's attribution result, 5 runs per arm: production shaping attributed the disclosure
correctly **5/5**; the pre-fix shaping attributed it to a decoy question **4/5**. Re-runnable via
`dev_scripts/war_detection_attribution_ab.py`.
Groups A, B, C and E cannot run from a Claude Code **web** session. All outbound traffic from an
Anthropic-hosted cloud environment passes through an HTTP/HTTPS proxy whose allowlist is
domain-based with no port concept, so Postgres on 5432 has no route — verified by observing that
port 443 to the same host connects while 5432 is silently dropped. Run these locally, or from a
self-hosted environment where egress leaves through your own network boundary. Cosmos and
Kombo/Bullhorn are HTTPS and remain reachable from a web session.
***
## Dev sandbox facts
Confirmed live against the dev Bullhorn integration.
| | |
| -------------------------------------------- | -------------------------------------------------------------------------------------------- |
| Org | `9f49b6d1-b019-468e-a7f6-3faea6e7fc2d` |
| Integration | `bullhorn:BUH2raYRWJy2ezE72xfyZwkX` — **plain Bullhorn, not Innova** |
| `pending_accommodation_review_stage` | **not configured** — this is exactly the TEN-809 shape |
| `ai_interview_pass_stage` / `..._fail_stage` | `Candidate Interested` / `Candidate Rejected` |
| `stage_routing_rules` | not persisted; rules synthesized from flat fields |
| Thresholds | pass score 70, completion rate 0.8 |
| WAR enablement | **off everywhere** — no admin group, all 40 jobs `{overrideDefaults: false, enabled: false}` |
Sandbox stages available: `New Submission, CV Sent, Shortlisted, Candidate Interested,
Candidate Rejected, Client Rejected, Consultant Rejected, Reference Check, Offer Extended,
Offer Rejected, Tenzo Test Stage, New Test Stage, 1st–3rd Interview, Final Interview,
Manager Review, Hired, Submitted, Placed, New Lead`.
Because the sandbox integration is plain Bullhorn, **the Innova VRA presence path cannot be
exercised here with real data.** It needs an Innova org or stays unit-tested only.
***
## Group A — overlay and release
| # | Scenario | Expected |
| -- | --------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| A1 | Flagged interview completes | CCI on `PENDING_ACCOMMODATION_REVIEW`, flagged non-INFO question holds provisional full credit (INFO questions keep evaluated grade without provisional inflation), ATS stays on `Candidate Interested`, exactly **one** note posted |
| A2 | Reviewer resolves last item, verdict unchanged | Releases to `CALL_COMPLETED`; ATS move `SKIPPED`; **no second note** |
| A3 | Reviewer scores the flagged required question 0 | Releases to `FAILED_KNOCKOUT`; ATS moves to `Candidate Rejected`; note posted |
| A4 | **TEN-809 regression** — recruiter moves the submission to `2nd Interview`, then reviewer resolves | ATS untouched, no note, Tenzo hold still resolves |
| A5 | Parking-stage variant — set `pending_accommodation_review_stage` to `Tenzo Test Stage`, rerun A1–A4 | Overlay parks in ATS; release unparks and comments |
| A6 | Overlay gate | `FAILED_KNOCKOUT` never overlaid; `INCOMPLETE`/`RESCHEDULED` stays on the retry path and re-fires next completed attempt |
| A7 | Kombo staleness window | Move in Bullhorn, resolve immediately, measure how long the stale snapshot persists (see finding 1) |
## Group B — regrade reconcile
| # | Scenario | Expected |
| -- | ------------------------------------------------ | --------------------------------------------------------------------------------------------- |
| B1 | Regrade turns `COMPLETED` into a knockout | Stage `CALL_COMPLETED` → `FAILED_KNOCKOUT`; post-commit ATS move to `Candidate Rejected` |
| B2 | Regrade clears a knockout, outro reached | Back to `CALL_COMPLETED` and `Candidate Interested` |
| B3 | Regrade clears a knockout, outro **not** reached | `INCOMPLETE`; no stage move; persisted verdict untouched |
| B4 | Bulk-regrade safety | Dispositioned, opted-out and outreach-stage candidates are **not** resurrected |
| B5 | Application under WAR hold | Left to the release path, not moved by the reconcile |
| B6 | Post-commit ATS failure | Reported as a failed reconcile; `result_edited_since_ats_push` keeps manual re-push available |
## Group C — never\_asked
| # | Scenario | Expected |
| -- | ------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| C1 | Add a question to the job, regrade an older call | Back-filled row carries `never_asked`; score, completion rate, binary lifts and knockout verdict all unchanged |
| C2 | Reviewer types "Not asked" as free-form answer text | Does **not** clear a genuine knockout |
| C3 | `GRADE_ONLY` grading failure | Exceptions or returned placeholders that can affect outcomes abort without persistence; purposeful skips and non-required `INFO` placeholders remain harmless |
| C4 | `backfill_never_asked_marker` dry run over legacy rows | Recognises the legacy shape without touching anything else |
## Group E — settings, listing, ownership
| # | Scenario | Expected |
| -- | -------------------------------------------------------------- | ------------------------------------------------------------------------- |
| E1 | WAR toggle under Recruiting Results, org and job override | Works with the feature flag gone |
| E2 | Listing badge vs `needs_work_accommodation_review_only` filter | Agree, including unsynced and JSON-`null` `application_stage` rows |
| E3 | Two calls on one application | Older call's release or regrade must not overwrite the newer call's stage |
| E4 | Migration `42dad1876e35` | Dev DB on head; nothing reads the dropped column |
***
## What to create, update and delete
Approved scope. Nothing touches production, any other dev org, or any of the 40 existing jobs.
### Tenzo (dev)
| Action | Object |
| ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Create | Job `WAR Regression 2026-08-19`, linked to sandbox Bullhorn req `Data Engineer` (remote id 1634) |
| Create | Script with \~6 questions: one required knockout, one that invites a scheduling or medical answer, one added mid-run for the `never_asked` case |
| Create | 5 candidates and applications, **non-test** — `regrade_terminal_outcome` exempts test calls, so test candidates cannot exercise Group B |
| Create | 5 completed AI interviews across the scenarios |
| Update | WAR setting on the test job only: `overrideDefaults: true, enabled: true`. Not org-wide |
| Update | Org ATS config `bullhorn:BUH2raYRWJy2ezE72xfyZwkX` — add `pending_accommodation_review_stage: "Tenzo Test Stage"` for A5 only, after snapshotting the doc, then restore byte-for-byte |
| Update | Grades via the reviewer resolve path; `update_scores` regrades for Groups B and C |
| Delete | **nothing** |
### Bullhorn dev sandbox
| Action | Object |
| ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Create | 5 candidates and job submissions on `Data Engineer` |
| Update | Submission statuses, driven by Tenzo: `New Submission` → `Candidate Interested` / `Candidate Rejected`. Manually move one to `2nd Interview` for A4 and one to `Tenzo Test Stage` for A5 |
| Update | Tenzo writes interview-result notes on submissions — **note count is the assertion** for the duplicate-note fix |
| Delete | **nothing.** Restore statuses to `New Submission` afterwards. Notes accumulate — Kombo has no clean delete, so the sandbox keeps roughly ten test notes |
***
## Open findings & Resolutions
1. **The TEN-809 fix and Kombo sync.** `review_release_stage_presence` reads
Kombo's synced application snapshot. This aligns with all other ATS transitions in Tenzo
(`AI_INTERVIEW_PASSED`, `AI_INTERVIEW_FAILED`, `HUMAN_REVIEW`, etc.) which read through
the cached application layer. Retained standard cached behavior.
2. **Stuck review hold monitoring.** Hardened exception logging in
`work_accommodation_review_stage_sync.py` and `work_accommodation_review_stage_release.py`
to consistently emit `error.kind="WorkAccommodationReviewReleaseTransitionFailed"`.
3. **Innova blank VRA field.** Retained safe fail-closed behavior to prevent unexpected
downstream ATS writes without customer-specific product alignment.
4. **Workflow step IDs vs stage names target set.** Verified non-issue for real ATS setups.
5. **ATS-less orgs release note.** Confirmed intended behavior: orgs without ATS integration
record resolutions locally within Tenzo.
***
## Running this
Groups A, B, C and E need a working Postgres connection, so run them from a local checkout of
`claude/war-updates-review-test-5b7cuu` — or any branch carrying these changes — rather than a
cloud session.
Group D needs no database. `dev_scripts/war_detection_attribution_ab.py` runs the attribution A/B
against the live model from anywhere with LLM credentials, and the rest of Group D is already
covered by unit tests: `tests/unit/caller/test_work_accommodation_review.py`,
`tests/unit/caller/test_flagged_statement_detection.py`,
`tests/unit/post_process_call/test_broadcast_display_transcript.py` and
`tests/unit/post_process_call/test_accommodation_review_unredacted_input.py`.
`./bin/test` reaches a shared Redis on port 6380 via the committed `.env.test`. In any sandboxed
environment without that egress, roughly 32 `post_process_call` tests fail on a 30-second per-test
timeout rather than on anything real. Check the traceback for a blocked `sock_connect` before
treating those failures as regressions.