Fraud risk operations

Restricted Access: This documentation is only accessible to @tenzo.ai and @salv.ai email addresses.

This page covers rollout, tuning, and health checks. For what the feature does and what each signal checks, see Fraud risk. For the full engineering runbook, including exact function names, SQL, and migration ordering, see server/fraud_risk/OPERATIONS.md.

Turning it on for a customer

Grading and showing are separate steps, and they can be weeks apart. Switching evaluation on starts grading new applicants and storing the findings; a recruiter sees a finding only once the job setting, the org default setting and fraud_signals_tab are all on. Nothing is re-evaluated when that flag flips, so whatever was collected in the meantime appears immediately and costs nothing. Support users in Full Access mode see an org’s findings whether or not that org has opted in, which is how support reads an org exactly as it will look once the flag goes on. Four things are independent, and all four matter:

The job setting. FraudRiskSettings.enabled, set on the job or as the org default, and off until someone turns it on. enable_fraud_risk_evaluation turns it on across an org (dry run first; it skips any job that already auto-closes at a tier, since that job would start closing applications on its first new applicant). Nothing is evaluated or stored while it is off, and it is also one of the three switches that must be on before findings are shown. It lives in Cosmos as part of the job’s settings document, not in Postgres, so no SQL query will find it.
Provider credentials, per environment rather than per org. Each is named in the configuration table in server/fraud_risk/OPERATIONS.md. A missing credential silently disables the signals that need it: they report themselves unevaluated rather than guessing.
The feature flags, per org, set directly in org_feature_flags. There is no admin UI. Allow up to 60 seconds per process for a change to take effect.
The auto-reject threshold, auto_reject_minimum_fraud_risk_level. Leave it unset at first. With no minimum configured nothing is ever closed automatically, so the grades can be reviewed before they start moving anyone’s stage.

Run the contact fingerprint backfill before turning fraud evaluation on for an org. Fingerprints are written as candidates are evaluated, so evaluating first leaves every collision that already exists in the org invisible until each of those candidates happens to be evaluated again.

The feature flags

Flag	What it gates
`fraud_signals_tab`	The reviewer-facing signals panel, the overall risk badge beside the candidate’s name, the Fraud Risk columns in the per-job Candidates and organization-wide Interviews tables, the two candidate filters, the shared-contact drill-down, and the fraud grades in AI chat
`cross_org_fraud_matching`	The three cross-org shared-contact signals, on top of the in-org shared-contact signals

Both default to off with no global default row, so an org without an explicit row inherits off. Cross-org matching is reciprocal. An org matches only against other orgs that have also turned it on, and a global default row does not count. That means turning it on for a single org achieves nothing: it needs at least two participants, and each has to have agreed to it. The seven scoring candidate-scoped signals, the four in-org shared-contact signals, and advisory resume-history comparison are not flag-gated. They run from application creation for jobs with fraud risk evaluation enabled. fraud_signals_tab gates whether a customer sees any of it, not whether it is collected.

Choosing an auto-reject threshold

The scoring cutoffs mean a single signal rarely escalates an application on its own: one High finding scores 40, which buckets to Medium. Reaching High takes 80, so it takes two High findings, or one High plus two Mediums. This has a specific consequence when the shared-contact signals get turned on for an org that already has an auto-reject threshold. A shared-contact finding adds 20 or 40 to an application that previously scored below the cutoff, so evaluating an org that has auto-reject already configured can close applications that would not have been closed before. The effect is largest for orgs whose scored applications cluster just under a cutoff. Check that org’s distribution first:

select cci.fraud_risk_level,
       count(*),
       round(avg(cci.fraud_risk_score)::numeric, 1) as avg_score
from candidate_campaign_info cci
join campaigns c on c.campaign_id = cci.campaign_id
join orgs o on o.org_id = c.org_id
where cci.fraud_risk_level is not null
  and o.name = '<org name>'
group by 1;

An application already scoring 40 or more is one High shared-contact finding (+40) away from the High bucket. One at 60 or more is one Medium finding (+20) away. The current level scores and cutoffs are preliminary and expected to be tuned against live data. All signals weigh the same today; the per-signal weight table exists as the tuning surface but is empty.

Results are cached, and re-deriving them costs money

A candidate-scoped signal is computed once and reused for the life of the candidate. Changing what a signal emits therefore does not reach the rows already stored. Each signal carries a rule version, and bumping it is what forces stored rows to be re-derived. Bump deliberately. Re-derivation costs one real provider call (LLM, Twilio, or Google) per candidate holding that signal. With roughly 24,000 stored rows each for the name-match and VOIP signals, a careless bump is a large bill. Where the new value can be derived from the stored one, the refresh_fraud_signal_copy job rewrites rows in place for free instead. Display names and tooltips are read live off the evaluator, so editing a title or a description needs no bump and reaches everywhere immediately. Cross-application signals are recomputed on every evaluation, so the version gate never withholds them.

The re-evaluation queue

When a candidate joins or changes a shared-contact cluster, every other member of that cluster is written to a durable queue. A worker claims up to 50 candidates every 30 seconds and re-evaluates them. A failed re-evaluation is re-queued rather than lost, up to five attempts, after which the candidate is dead-lettered with error.kind: FraudReevalDeadLetter and nothing retries it later.

Health check: which signals are actually alive

The clearest signal of a misconfiguration is a check that has stopped writing rows. This query shows every signal, how often it fires, and when it was last written:

select signal_id,
       count(*) as rows,
       count(*) filter (where contributions = '[]'::jsonb) as passes,
       count(*) filter (where contributions <> '[]'::jsonb) as flagged,
       max(updated_at)::date as last_written
from candidate_fraud_signals
group by signal_id
order by rows desc;

As of 2026-08-25 in production:

Signal	Rows	Passes	Flagged	Last written
Name match	24,711	24,088	623	2026-08-25
VOIP phone	24,572	20,774	3,798	2026-08-25
Identity match	8,246	6,521	1,725	2026-08-25
Address validation	3,330	3,252	78	2026-08-25
Address consistency	3,216	3,163	53	2026-08-25
GitHub profile	628	596	32	2026-08-25
Shared phone	8	7	1	2026-06-25
Shared IP	8	0	8	2026-06-25
LinkedIn profile	4	0	4	2026-06-25
Shared address	2	1	1	2026-06-25
Shared email	2	0	2	2026-06-25

Two things stand out and are worth reading correctly:

The shared-contact signals are near-zero because only Tenzo has the flag on. That is expected, not a fault.
The LinkedIn signal was unconfigured when this production snapshot was captured. Four rows, none since 2026-06-25, while its resume-sibling GitHub signal has 628 rows and is still being written. The AKS deployment now provisions the Serper API key it needs; after the next worker rollout, confirm new rows and check the provider-auth and provider-quota error kinds below.

When a signal goes quiet

Check the row count above first, then Datadog. The LinkedIn signal binds a distinct error.kind per failure mode so a monitor can route each to the right fix:

`error.kind`	Meaning	Fix
`LinkedInSearchNotConfigured`	The search API key is unset	Provision the environment variable
`LinkedInSearchProviderAuth`	The search provider rejected the credential	Check the key and the plan behind it
`LinkedInSearchProviderQuota`	Rate limit or search credits exhausted	Top up the balance, or slow the caller down
`LinkedInSearchProviderFailed`	Anything else	Investigate; this is the class of breakage that left the signal dead when Bing was retired

Not every failure path reports its bound kind: where an exception is attached, the exception type wins and the log surfaces as an HTTP or JSON error instead. A monitor should match the kind or the message. A response that does not answer the query we sent is discarded rather than read as a profile that does not exist. A not-found grades as High, so a cached or misrouted answer would otherwise become a fraud finding against a real candidate.

One-time jobs

Five jobs exist, all run with python -m one-time-jobs.<name> from server/, all defaulting to a dry run that writes nothing.

Job	What it does	Status
`backfill_contact_fingerprints`	Writes contact fingerprints for existing candidates so shared-contact clusters are visible from the first evaluation	Run before turning fraud evaluation on for an org
`enable_fraud_risk_evaluation`	Turns the per-job fraud risk setting on across an org’s jobs, skipping any job that already carries an auto-close tier	Not yet run against production
`backfill_fraud_risk_evaluation`	Grades the non-terminal applicants who predate fraud evaluation, ignoring the per-job setting and never auto-rejecting	Not yet run. Run `backfill_contact_fingerprints --apply` first, then one org at a time
`clear_vacuous_address_consistency_signals`	Deletes stored address-consistency passes that never actually compared anything	Run against production on 2026-08-25; 109 rows removed, no flagged row touched
`refresh_fraud_signal_copy`	Rewrites stored findings in place when the new copy is derivable from the old, avoiding a paid re-derivation	Do not run. See Known gaps

backfill_fraud_risk_evaluation spends real provider money, per candidate rather than per application: a Twilio line-type lookup, a billable Twilio Identity Match on US and Puerto Rico numbers, a second billable caller-name lookup when that match comes back flagged, two to four Gemini calls, and Google geocoding where the resume carries an address. Its dry run prints those counts per org before anything is spent, so read them before widening past a pilot. Its ordering dependency is not advisory. Shared-contact findings are cached for the life of the candidate and no later run re-derives them, so a candidate graded while candidate_contact_fingerprints is still filling stores “nobody shares this value” permanently. --apply refuses to start until nearly every fingerprint-eligible candidate in scope has a row.

Dev tooling

An internal fraud data page can seed a candidate’s signals to any outcome an evaluator actually produces in production, or run a signal’s real comparison over inputs supplied by hand. It is the way to see the reviewer panel populated without waiting for a real fraudulent application. Be aware that the seeder’s preset copy has drifted from what the evaluators write, so copy verified against a seeded candidate is not necessarily the copy production produces.

Known gaps

The full list, with file and function references, is in the Known gaps sections of server/fraud_risk/OPERATIONS.md and ui/src/components/transcript/FRAUD_SIGNALS.md. The items that matter operationally:

The LinkedIn signal needs post-rollout verification. It has never produced a finding in production: it ran on Bing until that API was retired, then on Google Programmable Search, which refuses the project the key belongs to. Confirm the search key is provisioned and that new LinkedIn signal rows are being written.
refresh_fraud_signal_copy would flatten four distinct address findings into one. It rewrites every stored address-validation finding to a generic “Invalid address” (metadata, including the checked address, is kept), contradicting its own stated contract. It has never been run against production, so nothing has been flattened. Fix the job before running it.
More applications are graded High than Medium, which is not the shape the current cutoffs predict, and 6,409 of the 6,545 High rows carry no stored score at all. The cause is not fully established. A corrective backfill for the part of it that is understood exists and has not been run. Treat the High population as unreliable until it has been.
A stale signal can be dropped from the score but stay on screen. When a stored row cannot be reused and re-derivation returns nothing, the row is left in place. The score no longer counts it, but the panel still draws it, so reviewer and score disagree until the signal re-derives successfully.
A dead-lettered re-evaluation is never retried. The queue model’s documentation describes a nightly sweep for stragglers. There is no such sweep.
One concept has three names. The UI calls it “Auto-close applications”, the setting is auto_reject_minimum_fraud_risk_level, and the stage is FRAUD_REJECTED. Expect to translate between a support conversation, a settings payload, and a stage query.

Internal Documentation

Modules

Fraud Risk

Outreach

ATS Implementations

Turning it on for a customer

The feature flags

Choosing an auto-reject threshold

Results are cached, and re-deriving them costs money

The re-evaluation queue

Health check: which signals are actually alive

When a signal goes quiet

One-time jobs

Dev tooling

Known gaps

​Turning it on for a customer

​The feature flags

​Choosing an auto-reject threshold

​Results are cached, and re-deriving them costs money

​The re-evaluation queue

​Health check: which signals are actually alive

​When a signal goes quiet

​One-time jobs

​Dev tooling

​Known gaps

Turning it on for a customer

The feature flags

Choosing an auto-reject threshold

Results are cached, and re-deriving them costs money

The re-evaluation queue

Health check: which signals are actually alive

When a signal goes quiet

One-time jobs

Dev tooling

Known gaps