ATS Sync Alerts

Restricted Access: This documentation is only accessible to @tenzo.ai and @salv.ai email addresses.

Overview

Datadog monitors for ATS sync health and outbound ATS request-queue health. Slack channels:

Channel	Monitors
`#dd-sync-alerts`	Kombo sync failed, enqueue failed, sync lag, stuck transient stage, hang abandon, handler duration
`#dd-ats-request-queue`	Failure rate, HTTP errors, timeouts, exceptions, depth, rate limit

Related tickets: TEN-350 (sync alerting), TEN-502 (stuck CREATING_APPLICATION), TEN-576 (reason-tagged abandon / hang alerts). Slack format: alerts are one-line headers with links here. Recoveries do not post to Slack (Datadog UI only). Investigate in the alert thread.

Compact Slack notifications

To keep #dd-sync-alerts and #dd-ats-request-queue readable, alerts use three layers:

Layer	Setting	Effect
Monitor message	One-line `{{#is_alert}}` body + runbook link	Drops “What to check” from Slack (lives on this page)
Monitor option	`notification_preset_name: hide_query_and_handles`	Hides auto-appended query text and footer links
Slack channel display	`snapshot: false`, `tags: false`, `notified: false`	Removes graph previews, tag grid, and “Notified:” line

Datadog still adds a short attachment title, metric value (metric monitors), and built-in action buttons (Mute, Declare incident, Create case, Investigate with Bits). Mute is enabled via mute_buttons: true; the other buttons are native Datadog Slack app features. Re-apply monitor messages + compact Slack settings (requires SALV_DD_API_KEY / SALV_DD_APP_KEY with monitor-write and manage-integrations):

python3 scripts/update-ats-alert-monitor-messages.py

`#dd-sync-alerts`

Kombo ATS sync failed (webhook)

Monitor: 309657484 · Query: KomboSyncNotSucceeded log count > 0 in 30m, per ats_integration_id What to check

Kombo integration health / sync logs for this ats_integration_id
sync_state on matching logs: failed, partially_failed, authentication_failed, etc.
Logs

authentication_failed webhooks are often recurring Kombo auth drift — treat separately from one-off sync failures.

ATS sync queue enqueue failed

Canonical metric: ats.sync.queue.errors (publisher-side; every recorded publish failure, including permanent skips) Monitor: 309657487 · Query: ats.sync.stream.errors{error_type:publish*} > 0 in 30m, per integration_id + sync_type The monitor still matches the dual-write of ats.sync.stream.errors (error_type:publish*). Prefer ats.sync.queue.errors in new dashboards; the monitor query can switch later without a code change. Tags

class:retry — send failed, cycle timeout, or unexpected publish exception. Cursor is not advanced; the next cycle retries.
class:permanent — message cannot fit Service Bus (reason:message_too_large). Cursor is advanced so the queue cannot wedge on an unsizable payload.
reason: send_failed | message_too_large | publish_timeout | publish_exception
Also tagged with sync_type, integration_id, org_id, ats_provider, topic

What to check

Service Bus health / publish errors for this integration
Sync controller logs around job/applicant/candidate/tearsheet/placement queue publish
Permanent skips: logs with error.kind:AtsSyncPublishPermanentSkip
Metric
Companion dual-write: stream.errors publish*

Cursor advance rule

Every sync type decides whether to move its cursor with one rule, evaluated top to bottom, first match wins. The rule lives in server/ats/sync_watermark.py (decide_watermark_advance).

Outcome of the cycle	Cursor
Window not fully enumerated (stream error, mid-stream failure)	Held, warning logged
Any retryable failure (transient item error, failed send)	Held, warning logged; next cycle retries from the same cursor
Permanent per-item failure (`message_too_large`) on the batched publishers (jobs, applicants, candidates), where a durable queue owns retries	Advanced, warning logged; item recorded via `class:permanent` and `AtsSyncPublishPermanentSkip`
Permanent per-item failure in in-memory mode, where the cursor is the only retry mechanism	Held, warning logged
Zero items in the window	Advanced
Everything succeeded	Advanced

The cursor always moves to the sync start time, never the end time, so records modified during the run fall inside the next window. A publish timeout or exception never reaches the rule: the phase exits before deciding, the cursor is simply not written, and the timeout logs a warning while a non-transient exception logs an error with a traceback. A job publish exception also cancels the applicant and tearsheet phases running alongside it. When the job publish completes but leaves its cursor held (retryable item failures), the applicant, tearsheet and placement phases still run to completion, and the cycle is then reported as failed for that integration. The tearsheet and placement stream publishers send one message at a time and stop at the first failure, so they never produce permanent skips.

ATS sync queue lag > 1 hour

Monitor: 309657489 · Query: ats.sync.queue.lag_seconds > 3600 in 15m, per integration_id + sync_type What to check

Service Bus subscription backlog / worker health for this sync type
Whether this integration is publishing faster than workers can process
Metric
Companion: ats.sync.queue.subscription.active_messages

Chronic lag on a single integration may indicate a disabled consumer or stale metric — verify active_messages before deep-diving.

ATS sync hang abandon

Monitor: 314970801 · Query: ats.sync.queue.abandoned{class:hang} > 0 in 15m, per integration_id + sync_type + reason Fires when a sync worker (or the session worker) abandons a message for a hang-class reason. Retry-class abandons (ats_fetch_failed, transient sync errors, client-init failures, …) emit the same metric but do not Slack — they are normal Service Bus redelivery. lock_lost is included at > 0: production had 0 "Lost message/session lock during process_batch" logs in the 7d before this monitor shipped. Large multi-chunk batches are allowed to overrun 900s; if lock_lost becomes noisy, raise that reason’s threshold rather than dropping the rest of the hang class. This is the signal for applicant/candidate 600s bulk-fetch timeouts (reason:ats_fetch_timeout). The duration monitor below does not catch those — they return ABANDON around 600s, under the 720s duration threshold.

Which monitor fires

Failure	Signal
Applicant/candidate bulk-fetch `asyncio.wait_for` (600s)	Hang abandon · `reason:ats_fetch_timeout`
Session lock expired mid-batch	Hang abandon · `reason:lock_lost` and/or handler duration > 720s
Unhandled `process_batch` / `process` / chunk	Hang abandon · `unhandled_process_batch`, `unhandled_process`, `unhandled_chunk`
Handler returned the wrong result count / missing `process_batch`	Hang abandon · `invalid_batch_result_count`, `missing_process_batch`
Missed instrumentation (slot never classified)	Hang abandon · `reason:unspecified`
Job/tearsheet/placement fetch errors (no 600s wrapper)	Hang abandon only if they hang into `lock_lost` / duration; otherwise retry-class metric only

Reason → log grep

`reason`	Class	Log grep
`ats_fetch_timeout`	hang	`timed out bulk-fetching`
`lock_lost`	hang	`Lost message/session lock during process_batch`
`unhandled_process_batch`	hang	`Unhandled error in session worker process_batch call`
`unhandled_process`	hang	`Unhandled error in session worker process call`
`unhandled_chunk`	hang	`Unhandled error during applicant chunk processing` / `candidate chunk processing`
`invalid_batch_result_count`	hang	`process_batch returned invalid result count`
`missing_process_batch`	hang	`did not implement process_batch`
`unspecified`	hang	missed `AbandonReason` on an ABANDON slot — grep `ats.sync.queue.abandoned` for `reason:unspecified`

What to check

Metric explorer: ats.sync.queue.abandoned{env:production,class:hang} grouped by reason, sync_type, integration_id
Companion: ats.sync.queue.handler.duration_seconds and queue lag for the same integration
Job/tearsheet/placement fetch errors without a 600s wrapper surface as lock_lost and/or duration, not ats_fetch_timeout

ATS sync handler near session lock

Monitor: 314970802 · Query: ats.sync.queue.handler.duration_seconds > 720 in 15m, per integration_id + sync_type 80% of lock_renewal_seconds=900. Catches batches still running toward lock expiry (and slow completes). Does not fire for the 600s applicant/candidate fetch-timeout path — use hang abandon reason:ats_fetch_timeout for that. Duration is now emitted on session-worker exception paths (lock_lost, unhandled process_batch) as well as successful handler returns. What to check

Whether this integration’s receive batch / chunk size is too large for the 900s session lock
Hang abandon for the same integration_id (lock_lost vs still-in-progress)
Metric

Candidates stuck in transient Tenzo stage

Monitor: 309683123 · Query: ats.candidate.stuck_transient_stage.count > 0 in 15m, per org_id + stage

What we detect

Stage	Extra filter	Stuck threshold
`CREATING_APPLICATION`	`ats_application_id IS NULL`	6 hours since stage entry

Stage entry age is the created_at of the latest matching tenzo_stage_history row where new_stage equals the CCI’s current tenzo_stage (ordered by history_seq desc). If no history row exists, we fall back to candidate_campaign_info.updated_at. Soft-deleted CCIs and campaigns are excluded. Only non-deleted rows are counted. The policy list in code is extensible (DEFAULT_TRANSIENT_STAGE_STUCK_POLICIES in server/ats/stuck_transient_stage_metrics.py); only CREATING_APPLICATION ships initially. Do not add AWAITING_RESUME_REVIEW here until that alert is intentionally scoped.

Metrics

Emitted every 5 minutes from the queue server (skipped when LOCAL or test).

Metric	Type	Meaning
`ats.candidate.stuck_transient_stage.count`	gauge	Number of stuck CCIs for `(org_id, stage)`
`ats.candidate.stuck_transient_stage.max_age_hours`	gauge	Hours since the oldest stuck row entered the stage

Tags: org_id, stage. (integration_id is omitted — it lives in Cosmos, not on the CCI row used for this query.)

What to check

List stuck rows via candidate_campaign_ats_sync_dao.list_stuck_transient_stage_candidates (see Ops inspection below)
Applicant sync lag: ats.sync.queue.lag_seconds{sync_type:applicant,env:production}
Logs: Deferred CCI link for native-fallback, remains in CREATING_APPLICATION, Applicant sync worker timed out bulk-fetching applications
Metric

`#dd-ats-request-queue`

ATS request queue failure rate > 5%

Monitor: 309673813 · Query: (http_error + timeout + exception) / completed > 5% in 30m, only if ≥100 completed requests, per integration_id What to check

ATS/provider health and 4xx/5xx responses for this integration
Queue depth / capacity: ats.request_queue.depth, ats.request_queue.rate_limiter.capacity_remaining
Request-queue service logs for this integration_id

ATS request queue HTTP errors spike

Monitor: 309673821 · Query: ats.request_queue.http_error > 10 in 30m, per integration_id Absolute spike alert — catches high-volume integrations with many failures even if rate stays under 5%. What to check

Request-queue logs for HTTP error responses (Kombo passthrough vs direct provider)
Provider status pages / incident history
Whether failures cluster on one endpoint (e.g. stage writebacks)

ATS request queue timeouts

Monitor: 309673814 · Query: ats.request_queue.timeout > 0 in 15m, per integration_id What to check

ats.request_queue.depth and in_flight for this integration
Attachment / long-running requests monopolizing workers
Request-queue service logs

ATS request queue exceptions

Monitor: 309673815 · Query: ats.request_queue.exception > 0 in 15m, per integration_id What to check

Request-queue exception logs for this integration
Provider connectivity / DNS / TLS issues
Recent deploys of the ATS request-queue service

ATS request queue depth high

Monitor: 309673819 · Query: ats.request_queue.depth warn >15 / critical >50 in 15m, per integration_id What to check

Worker pool / rate-limit capacity for this integration
Attachment storms / sync storms
ats.request_queue.in_flight and capacity_remaining

ATS request queue rate-limit capacity exhausted

Monitor: 309673820 · Query: avg ats.request_queue.rate_limiter.capacity_remaining < 1 in 15m, per integration_id What to check

Whether 429s are appearing in request-queue logs
Sync storm vs normal traffic for this integration
Provider RPM budget in rate-limiter config

Often informational during heavy sync — correlate with failure-rate and depth monitors before acting.

Ops inspection (stuck transient stage)

DAO helper (preferred)

Read-only, bounded list (default limit=100), oldest first:

from datetime import timedelta

from application_stage.tenzo_stage_enum import TenzoStage
from common_utils.time_utils import utc_now
from dao.candidate_campaign_info.candidate_campaign_ats_sync_dao import (
    candidate_campaign_ats_sync_dao,
)

now = utc_now()
stuck_after = now - timedelta(hours=6)
rows = await candidate_campaign_ats_sync_dao.list_stuck_transient_stage_candidates(
    org_id="<ORG_UUID>",
    stages=[TenzoStage.CREATING_APPLICATION.value],
    stuck_after=stuck_after,
    now=now,
    limit=100,
)
# Each row: candidate_campaign_id, campaign_id, campaign_name, stage,
# stage_entered_at, age_hours

Org-wide summary (counts + max age):

summaries = await candidate_campaign_ats_sync_dao.summarize_stuck_transient_stages(
    stages=[TenzoStage.CREATING_APPLICATION.value],
    stuck_after=stuck_after,
    now=now,
)

Equivalent SQL sketch

WITH stage_entered AS (
  SELECT
    cci.candidate_campaign_id,
    c.campaign_id,
    c.campaign_name,
    c.org_id,
    cci.tenzo_stage AS stage,
    COALESCE(
      (
        SELECT tsh.created_at
        FROM tenzo_stage_history tsh
        WHERE tsh.candidate_campaign_id = cci.candidate_campaign_id
          AND tsh.new_stage = cci.tenzo_stage
        ORDER BY tsh.history_seq DESC
        LIMIT 1
      ),
      cci.updated_at
    ) AS stage_entered_at
  FROM candidate_campaign_info cci
  JOIN campaigns c ON c.campaign_id = cci.campaign_id
  WHERE cci.deleted IS FALSE
    AND c.deleted IS FALSE
    AND cci.tenzo_stage = 'CREATING_APPLICATION'
    AND cci.ats_application_id IS NULL
)
SELECT *
FROM stage_entered
WHERE stage_entered_at <= NOW() - INTERVAL '6 hours'
ORDER BY stage_entered_at ASC
LIMIT 100;

Implementation pointers

Piece	Location
Policies + emit loop	`server/ats/stuck_transient_stage_metrics.py`
DAO summarize / list	`server/dao/candidate_campaign_info/candidate_campaign_ats_sync_dao.py`
Metric name constants	`server/metrics/datadog_metrics.py` (`MetricNames.ATS_CANDIDATE_STUCK_TRANSIENT_STAGE_*`, `ATS_SYNC_QUEUE_ABANDONED`)
Queue wiring	`BACKGROUND_TASK_SLOTS` in `server/queue_server/queue_server.py`
Abandon reasons + `ProcessOutcome`	`server/ats/service_bus/process_outcome.py`
Emit choke point	`server/ats/service_bus/service_bus_session_worker.py` (`_emit_settlement_metrics`, `_abandon_decoded_batch`)

This path does not use incident.io or operational_alerts/.

Internal Documentation

Modules

Fraud Risk

Outreach

ATS Implementations

Overview

Compact Slack notifications

`#dd-sync-alerts`

Kombo ATS sync failed (webhook)

ATS sync queue enqueue failed

Cursor advance rule

ATS sync queue lag > 1 hour

ATS sync hang abandon

Which monitor fires

Reason → log grep

ATS sync handler near session lock

Candidates stuck in transient Tenzo stage

What we detect

Metrics

What to check

`#dd-ats-request-queue`

ATS request queue failure rate > 5%

ATS request queue HTTP errors spike

ATS request queue timeouts

ATS request queue exceptions

ATS request queue depth high

ATS request queue rate-limit capacity exhausted

Ops inspection (stuck transient stage)

DAO helper (preferred)

Equivalent SQL sketch

Implementation pointers

​Overview

​Compact Slack notifications

​#dd-sync-alerts

​Kombo ATS sync failed (webhook)

​ATS sync queue enqueue failed

​Cursor advance rule

​ATS sync queue lag > 1 hour

​ATS sync hang abandon

​Which monitor fires

​Reason → log grep

​ATS sync handler near session lock

​Candidates stuck in transient Tenzo stage

​What we detect

​Metrics

​What to check

​#dd-ats-request-queue

​ATS request queue failure rate > 5%

​ATS request queue HTTP errors spike

​ATS request queue timeouts

​ATS request queue exceptions

​ATS request queue depth high

​ATS request queue rate-limit capacity exhausted

​Ops inspection (stuck transient stage)

​DAO helper (preferred)

​Equivalent SQL sketch

​Implementation pointers

Overview

Compact Slack notifications

`#dd-sync-alerts`

Kombo ATS sync failed (webhook)

ATS sync queue enqueue failed

Cursor advance rule

ATS sync queue lag > 1 hour

ATS sync hang abandon

Which monitor fires

Reason → log grep

ATS sync handler near session lock

Candidates stuck in transient Tenzo stage

What we detect

Metrics

What to check

`#dd-ats-request-queue`

ATS request queue failure rate > 5%

ATS request queue HTTP errors spike

ATS request queue timeouts

ATS request queue exceptions

ATS request queue depth high

ATS request queue rate-limit capacity exhausted

Ops inspection (stuck transient stage)

DAO helper (preferred)

Equivalent SQL sketch

Implementation pointers