> ## Documentation Index
> Fetch the complete documentation index at: https://f4c7a9e2d8b1-docs.tenzo.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# ATS Sync Alerts

> Datadog monitors and ops runbooks for ATS sync and request-queue alerting

<Warning>
  **Restricted Access**: This documentation is only accessible to @tenzo.ai and @salv.ai email addresses.
</Warning>

## Overview

Datadog monitors for ATS sync health and outbound ATS request-queue health. Slack channels:

| Channel                 | Monitors                                                                                           |
| ----------------------- | -------------------------------------------------------------------------------------------------- |
| `#dd-sync-alerts`       | Kombo sync failed, enqueue failed, sync lag, stuck transient stage, hang abandon, handler duration |
| `#dd-ats-request-queue` | Failure rate, HTTP errors, timeouts, exceptions, depth, rate limit                                 |

Related tickets: [TEN-350](https://linear.app/tenzo-ai/issue/TEN-350) (sync alerting), [TEN-502](https://linear.app/tenzo-ai/issue/TEN-502) (stuck CREATING\_APPLICATION), [TEN-576](https://linear.app/tenzo-ai/issue/TEN-576) (reason-tagged abandon / hang alerts).

**Slack format:** alerts are one-line headers with links here. Recoveries do **not** post to Slack (Datadog UI only). Investigate in the alert thread.

### Compact Slack notifications

To keep `#dd-sync-alerts` and `#dd-ats-request-queue` readable, alerts use three layers:

| Layer                 | Setting                                             | Effect                                                 |
| --------------------- | --------------------------------------------------- | ------------------------------------------------------ |
| Monitor message       | One-line `{{#is_alert}}` body + runbook link        | Drops "What to check" from Slack (lives on this page)  |
| Monitor option        | `notification_preset_name: hide_query_and_handles`  | Hides auto-appended query text and footer links        |
| Slack channel display | `snapshot: false`, `tags: false`, `notified: false` | Removes graph previews, tag grid, and "Notified:" line |

Datadog still adds a short attachment title, metric value (metric monitors), and built-in action buttons (Mute, Declare incident, Create case, Investigate with Bits). Mute is enabled via `mute_buttons: true`; the other buttons are native Datadog Slack app features.

Re-apply monitor messages + compact Slack settings (requires `SALV_DD_API_KEY` / `SALV_DD_APP_KEY` with monitor-write and manage-integrations):

```bash theme={null}
python3 scripts/update-ats-alert-monitor-messages.py
```

***

## `#dd-sync-alerts`

### Kombo ATS sync failed (webhook)

**Monitor:** [309657484](https://app.datadoghq.com/monitors/309657484) · **Query:** `KomboSyncNotSucceeded` log count > 0 in 30m, per `ats_integration_id`

**What to check**

* Kombo integration health / sync logs for this `ats_integration_id`
* `sync_state` on matching logs: `failed`, `partially_failed`, `authentication_failed`, etc.
* [Logs](https://app.datadoghq.com/logs?query=env%3Aproduction%20%40error.kind%3AKomboSyncNotSucceeded)

`authentication_failed` webhooks are often recurring Kombo auth drift — treat separately from one-off sync failures.

***

### ATS sync queue enqueue failed

**Canonical metric:** `ats.sync.queue.errors` (publisher-side; every recorded publish failure, including permanent skips)

**Monitor:** [309657487](https://app.datadoghq.com/monitors/309657487) · **Query:** `ats.sync.stream.errors{error_type:publish*}` > 0 in 30m, per `integration_id` + `sync_type`

The monitor still matches the dual-write of `ats.sync.stream.errors` (`error_type:publish*`). Prefer `ats.sync.queue.errors` in new dashboards; the monitor query can switch later without a code change.

**Tags**

* `class:retry` — send failed, cycle timeout, or unexpected publish exception. Cursor is **not** advanced; the next cycle retries.
* `class:permanent` — message cannot fit Service Bus (`reason:message_too_large`). Cursor **is** advanced so the queue cannot wedge on an unsizable payload.
* `reason`: `send_failed` | `message_too_large` | `publish_timeout` | `publish_exception`
* Also tagged with `sync_type`, `integration_id`, `org_id`, `ats_provider`, `topic`

**What to check**

* Service Bus health / publish errors for this integration
* Sync controller logs around job/applicant/candidate/tearsheet/placement queue publish
* Permanent skips: logs with `error.kind:AtsSyncPublishPermanentSkip`
* [Metric](https://app.datadoghq.com/metric/explorer?exp_metric=ats.sync.queue.errors\&exp_scope=env%3Aproduction)
* Companion dual-write: [stream.errors publish\*](https://app.datadoghq.com/metric/explorer?exp_metric=ats.sync.stream.errors\&exp_scope=env%3Aproduction%2Cerror_type%3Apublish*)

***

### Cursor advance rule

Every sync type decides whether to move its cursor with one rule, evaluated top to bottom, first match wins. The rule lives in `server/ats/sync_watermark.py` (`decide_watermark_advance`).

| Outcome of the cycle                                                                                                                          | Cursor                                                                                          |
| --------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- |
| Window not fully enumerated (stream error, mid-stream failure)                                                                                | Held, warning logged                                                                            |
| Any retryable failure (transient item error, failed send)                                                                                     | Held, warning logged; next cycle retries from the same cursor                                   |
| Permanent per-item failure (`message_too_large`) on the batched publishers (jobs, applicants, candidates), where a durable queue owns retries | Advanced, warning logged; item recorded via `class:permanent` and `AtsSyncPublishPermanentSkip` |
| Permanent per-item failure in in-memory mode, where the cursor is the only retry mechanism                                                    | Held, warning logged                                                                            |
| Zero items in the window                                                                                                                      | Advanced                                                                                        |
| Everything succeeded                                                                                                                          | Advanced                                                                                        |

The cursor always moves to the sync **start** time, never the end time, so records modified during the run fall inside the next window.

A publish timeout or exception never reaches the rule: the phase exits before deciding, the cursor is simply not written, and the timeout logs a warning while a non-transient exception logs an error with a traceback. A job publish exception also cancels the applicant and tearsheet phases running alongside it.

When the job publish completes but leaves its cursor held (retryable item failures), the applicant, tearsheet and placement phases still run to completion, and the cycle is then reported as failed for that integration. The tearsheet and placement stream publishers send one message at a time and stop at the first failure, so they never produce permanent skips.

***

### ATS sync queue lag > 1 hour

**Monitor:** [309657489](https://app.datadoghq.com/monitors/309657489) · **Query:** `ats.sync.queue.lag_seconds` > 3600 in 15m, per `integration_id` + `sync_type`

**What to check**

* Service Bus subscription backlog / worker health for this sync type
* Whether this integration is publishing faster than workers can process
* [Metric](https://app.datadoghq.com/metric/explorer?exp_metric=ats.sync.queue.lag_seconds\&exp_scope=env%3Aproduction)
* Companion: `ats.sync.queue.subscription.active_messages`

Chronic lag on a single integration may indicate a disabled consumer or stale metric — verify `active_messages` before deep-diving.

***

### ATS sync hang abandon

**Monitor:** [314970801](https://app.datadoghq.com/monitors/314970801) · **Query:** `ats.sync.queue.abandoned{class:hang}` > 0 in 15m, per `integration_id` + `sync_type` + `reason`

Fires when a sync worker (or the session worker) abandons a message for a **hang-class** reason. Retry-class abandons (`ats_fetch_failed`, transient sync errors, client-init failures, …) emit the same metric but do **not** Slack — they are normal Service Bus redelivery.

`lock_lost` is included at `> 0`: production had **0** `"Lost message/session lock during process_batch"` logs in the 7d before this monitor shipped. Large multi-chunk batches are *allowed* to overrun 900s; if `lock_lost` becomes noisy, raise that reason's threshold rather than dropping the rest of the hang class.

This is the signal for applicant/candidate **600s bulk-fetch timeouts** (`reason:ats_fetch_timeout`). The duration monitor below does **not** catch those — they return ABANDON around 600s, under the 720s duration threshold.

#### Which monitor fires

| Failure                                                           | Signal                                                                                        |
| ----------------------------------------------------------------- | --------------------------------------------------------------------------------------------- |
| Applicant/candidate bulk-fetch `asyncio.wait_for` (600s)          | Hang abandon · `reason:ats_fetch_timeout`                                                     |
| Session lock expired mid-batch                                    | Hang abandon · `reason:lock_lost` **and/or** handler duration > 720s                          |
| Unhandled `process_batch` / `process` / chunk                     | Hang abandon · `unhandled_process_batch`, `unhandled_process`, `unhandled_chunk`              |
| Handler returned the wrong result count / missing `process_batch` | Hang abandon · `invalid_batch_result_count`, `missing_process_batch`                          |
| Missed instrumentation (slot never classified)                    | Hang abandon · `reason:unspecified`                                                           |
| Job/tearsheet/placement fetch errors (no 600s wrapper)            | Hang abandon only if they hang into `lock_lost` / duration; otherwise retry-class metric only |

#### Reason → log grep

| `reason`                     | Class | Log grep                                                                                             |
| ---------------------------- | ----- | ---------------------------------------------------------------------------------------------------- |
| `ats_fetch_timeout`          | hang  | `timed out bulk-fetching`                                                                            |
| `lock_lost`                  | hang  | `Lost message/session lock during process_batch`                                                     |
| `unhandled_process_batch`    | hang  | `Unhandled error in session worker process_batch call`                                               |
| `unhandled_process`          | hang  | `Unhandled error in session worker process call`                                                     |
| `unhandled_chunk`            | hang  | `Unhandled error during applicant chunk processing` / `candidate chunk processing`                   |
| `invalid_batch_result_count` | hang  | `process_batch returned invalid result count`                                                        |
| `missing_process_batch`      | hang  | `did not implement process_batch`                                                                    |
| `unspecified`                | hang  | missed `AbandonReason` on an ABANDON slot — grep `ats.sync.queue.abandoned` for `reason:unspecified` |

**What to check**

* Metric explorer: `ats.sync.queue.abandoned{env:production,class:hang}` grouped by `reason`, `sync_type`, `integration_id`
* Companion: `ats.sync.queue.handler.duration_seconds` and queue lag for the same integration
* Job/tearsheet/placement fetch errors without a 600s wrapper surface as `lock_lost` and/or duration, not `ats_fetch_timeout`

***

### ATS sync handler near session lock

**Monitor:** [314970802](https://app.datadoghq.com/monitors/314970802) · **Query:** `ats.sync.queue.handler.duration_seconds` > 720 in 15m, per `integration_id` + `sync_type`

80% of `lock_renewal_seconds=900`. Catches batches still running toward lock expiry (and slow completes). Does **not** fire for the 600s applicant/candidate fetch-timeout path — use hang abandon `reason:ats_fetch_timeout` for that.

Duration is now emitted on session-worker exception paths (`lock_lost`, unhandled `process_batch`) as well as successful handler returns.

**What to check**

* Whether this integration's receive batch / chunk size is too large for the 900s session lock
* Hang abandon for the same `integration_id` (`lock_lost` vs still-in-progress)
* [Metric](https://app.datadoghq.com/metric/explorer?exp_metric=ats.sync.queue.handler.duration_seconds\&exp_scope=env%3Aproduction)

***

### Candidates stuck in transient Tenzo stage

**Monitor:** [309683123](https://app.datadoghq.com/monitors/309683123) · **Query:** `ats.candidate.stuck_transient_stage.count` > 0 in 15m, per `org_id` + `stage`

#### What we detect

| Stage                  | Extra filter                 | Stuck threshold               |
| ---------------------- | ---------------------------- | ----------------------------- |
| `CREATING_APPLICATION` | `ats_application_id IS NULL` | **6 hours** since stage entry |

Stage entry age is the `created_at` of the latest matching `tenzo_stage_history` row where `new_stage` equals the CCI's current `tenzo_stage` (ordered by `history_seq` desc). If no history row exists, we fall back to `candidate_campaign_info.updated_at`.

Soft-deleted CCIs and campaigns are excluded. Only non-deleted rows are counted.

The policy list in code is extensible (`DEFAULT_TRANSIENT_STAGE_STUCK_POLICIES` in `server/ats/stuck_transient_stage_metrics.py`); only `CREATING_APPLICATION` ships initially. Do **not** add `AWAITING_RESUME_REVIEW` here until that alert is intentionally scoped.

#### Metrics

Emitted every **5 minutes** from the queue server (skipped when `LOCAL` or test).

| Metric                                              | Type  | Meaning                                            |
| --------------------------------------------------- | ----- | -------------------------------------------------- |
| `ats.candidate.stuck_transient_stage.count`         | gauge | Number of stuck CCIs for `(org_id, stage)`         |
| `ats.candidate.stuck_transient_stage.max_age_hours` | gauge | Hours since the oldest stuck row entered the stage |

**Tags:** `org_id`, `stage`. (`integration_id` is omitted — it lives in Cosmos, not on the CCI row used for this query.)

#### What to check

* List stuck rows via `candidate_campaign_ats_sync_dao.list_stuck_transient_stage_candidates` (see Ops inspection below)
* Applicant sync lag: `ats.sync.queue.lag_seconds{sync_type:applicant,env:production}`
* Logs: `Deferred CCI link for native-fallback`, `remains in CREATING_APPLICATION`, `Applicant sync worker timed out bulk-fetching applications`
* [Metric](https://app.datadoghq.com/metric/explorer?exp_metric=ats.candidate.stuck_transient_stage.count\&exp_scope=env%3Aproduction)

***

## `#dd-ats-request-queue`

### ATS request queue failure rate > 5%

**Monitor:** [309673813](https://app.datadoghq.com/monitors/309673813) · **Query:** `(http_error + timeout + exception) / completed > 5%` in 30m, **only if** ≥100 completed requests, per `integration_id`

**What to check**

* ATS/provider health and 4xx/5xx responses for this integration
* Queue depth / capacity: `ats.request_queue.depth`, `ats.request_queue.rate_limiter.capacity_remaining`
* Request-queue service logs for this `integration_id`

***

### ATS request queue HTTP errors spike

**Monitor:** [309673821](https://app.datadoghq.com/monitors/309673821) · **Query:** `ats.request_queue.http_error` > 10 in 30m, per `integration_id`

Absolute spike alert — catches high-volume integrations with many failures even if rate stays under 5%.

**What to check**

* Request-queue logs for HTTP error responses (Kombo passthrough vs direct provider)
* Provider status pages / incident history
* Whether failures cluster on one endpoint (e.g. stage writebacks)

***

### ATS request queue timeouts

**Monitor:** [309673814](https://app.datadoghq.com/monitors/309673814) · **Query:** `ats.request_queue.timeout` > 0 in 15m, per `integration_id`

**What to check**

* `ats.request_queue.depth` and `in_flight` for this integration
* Attachment / long-running requests monopolizing workers
* Request-queue service logs

***

### ATS request queue exceptions

**Monitor:** [309673815](https://app.datadoghq.com/monitors/309673815) · **Query:** `ats.request_queue.exception` > 0 in 15m, per `integration_id`

**What to check**

* Request-queue exception logs for this integration
* Provider connectivity / DNS / TLS issues
* Recent deploys of the ATS request-queue service

***

### ATS request queue depth high

**Monitor:** [309673819](https://app.datadoghq.com/monitors/309673819) · **Query:** `ats.request_queue.depth` warn >15 / critical >50 in 15m, per `integration_id`

**What to check**

* Worker pool / rate-limit capacity for this integration
* Attachment storms / sync storms
* `ats.request_queue.in_flight` and `capacity_remaining`

***

### ATS request queue rate-limit capacity exhausted

**Monitor:** [309673820](https://app.datadoghq.com/monitors/309673820) · **Query:** avg `ats.request_queue.rate_limiter.capacity_remaining` \< 1 in 15m, per `integration_id`

**What to check**

* Whether 429s are appearing in request-queue logs
* Sync storm vs normal traffic for this integration
* Provider RPM budget in rate-limiter config

Often informational during heavy sync — correlate with failure-rate and depth monitors before acting.

***

## Ops inspection (stuck transient stage)

### DAO helper (preferred)

Read-only, bounded list (default `limit=100`), oldest first:

```python theme={null}
from datetime import timedelta

from application_stage.tenzo_stage_enum import TenzoStage
from common_utils.time_utils import utc_now
from dao.candidate_campaign_info.candidate_campaign_ats_sync_dao import (
    candidate_campaign_ats_sync_dao,
)

now = utc_now()
stuck_after = now - timedelta(hours=6)
rows = await candidate_campaign_ats_sync_dao.list_stuck_transient_stage_candidates(
    org_id="<ORG_UUID>",
    stages=[TenzoStage.CREATING_APPLICATION.value],
    stuck_after=stuck_after,
    now=now,
    limit=100,
)
# Each row: candidate_campaign_id, campaign_id, campaign_name, stage,
# stage_entered_at, age_hours
```

Org-wide summary (counts + max age):

```python theme={null}
summaries = await candidate_campaign_ats_sync_dao.summarize_stuck_transient_stages(
    stages=[TenzoStage.CREATING_APPLICATION.value],
    stuck_after=stuck_after,
    now=now,
)
```

### Equivalent SQL sketch

```sql theme={null}
WITH stage_entered AS (
  SELECT
    cci.candidate_campaign_id,
    c.campaign_id,
    c.campaign_name,
    c.org_id,
    cci.tenzo_stage AS stage,
    COALESCE(
      (
        SELECT tsh.created_at
        FROM tenzo_stage_history tsh
        WHERE tsh.candidate_campaign_id = cci.candidate_campaign_id
          AND tsh.new_stage = cci.tenzo_stage
        ORDER BY tsh.history_seq DESC
        LIMIT 1
      ),
      cci.updated_at
    ) AS stage_entered_at
  FROM candidate_campaign_info cci
  JOIN campaigns c ON c.campaign_id = cci.campaign_id
  WHERE cci.deleted IS FALSE
    AND c.deleted IS FALSE
    AND cci.tenzo_stage = 'CREATING_APPLICATION'
    AND cci.ats_application_id IS NULL
)
SELECT *
FROM stage_entered
WHERE stage_entered_at <= NOW() - INTERVAL '6 hours'
ORDER BY stage_entered_at ASC
LIMIT 100;
```

## Implementation pointers

| Piece                              | Location                                                                                                              |
| ---------------------------------- | --------------------------------------------------------------------------------------------------------------------- |
| Policies + emit loop               | `server/ats/stuck_transient_stage_metrics.py`                                                                         |
| DAO summarize / list               | `server/dao/candidate_campaign_info/candidate_campaign_ats_sync_dao.py`                                               |
| Metric name constants              | `server/metrics/datadog_metrics.py` (`MetricNames.ATS_CANDIDATE_STUCK_TRANSIENT_STAGE_*`, `ATS_SYNC_QUEUE_ABANDONED`) |
| Queue wiring                       | `BACKGROUND_TASK_SLOTS` in `server/queue_server/queue_server.py`                                                      |
| Abandon reasons + `ProcessOutcome` | `server/ats/service_bus/process_outcome.py`                                                                           |
| Emit choke point                   | `server/ats/service_bus/service_bus_session_worker.py` (`_emit_settlement_metrics`, `_abandon_decoded_batch`)         |

This path does **not** use incident.io or `operational_alerts/`.