How OfferGuard decides
Three independent layers — deterministic rules, retrieval over labeled cases, and a quote-verified LLM — composed by a pure function. Either layer alone can raise the alarm; the top tier needs corroboration.
The pipeline
Validation + rate limit
Zod schema (200–20,000 chars), 64KB body cap, per-IP limits
Normalization + extraction
whitespace fold, homoglyph fold, entities (emails/URLs/IM/phones/crypto), sha256 dedup cache
Deterministic rules
14 weighted red-flag rules with ±40-char evidence excerpts
Embedding retrieval
qwen3-embedding-8b (Fireworks, 1536d) → pgvector cosine kNN, 6 scam + 6 legit ≥ 0.55
LLM verdict (advisory)
Kimi K3 via Fireworks, strict JSON schema, temperature 0, untrusted-data framing
Tier composition
deterministic override logic — not the LLM; verbatim-quote verification drops hallucinated evidence
Rule catalog (14 rules)
| Code | Rule | Severity | Weight | Trigger |
|---|---|---|---|---|
| R01 | FEE_OR_PAYMENT | hard | 40 | Applicant is asked to pay money (processing/training/registration fee, deposit). |
| R02 | CRYPTO_PAYMENT | hard | 35 | Payment requested in cryptocurrency or a wallet address is supplied. |
| R03 | WIRE_OR_GIFT_CARD | hard | 35 | Payment via wire transfer, money order, or gift cards. |
| R10 | CHECK_OVERPAYMENT | hard | 35 | Classic overpayment/check-refund scam structure. |
| R11 | MONEY_MULE_SIGNALS | hard | 35 | Duties involve receiving and forwarding funds or reshipping packages. |
| R04 | IM_ONLY_CONTACT | hard | 25 | Interviews/communication exclusively via consumer IM apps. |
| R05 | SENDER_DOMAIN_MISMATCH | hard | 30 | Sender email domain is freemail while the posting cites a corporate site. |
| R06 | DISPOSABLE_EMAIL | medium | 20 | Contact uses a disposable email domain. |
| R12 | NO_VERIFIABLE_PRESENCE | soft | 10 | Freemail-only contact and no company website mentioned anywhere. |
| R07 | IDENTITY_OVERREACH | medium | 20 | Sensitive personal/financial data requested before hire. |
| R08 | URGENCY_PRESSURE | soft | 10 | Artificial urgency or offer-expiry pressure. |
| R09 | TOO_GOOD_COMP | soft | 15 | Implausible pay for unskilled/entry work. |
| R13 | FORMAT_SPAM | soft | 5 | Spammy formatting: caps/punctuation density. |
| R14 | VAGUE_ROLE | soft | 5 | No concrete responsibilities or skill requirements. |
Evaluation
- E1 — Kaggle EMSCA holdout: 150 fraudulent + 300 legitimate, disjoint from the corpus.
- E2 — manual holdout: 30% of a modern scam/legit set (FTC/IC3-pattern).
- E3 — adversarial: 20 prompt-injection payloads on known-legit postings; verdict must not flip.
- Binary collapse: LIKELY_SCAM = positive, SUSPICIOUS = abstain (reported separately).
First production run — 2026-10-03 (Kimi K3 + qwen3-embedding, 450 held-out cases)
- Precision 98.1% · false-positive rate 0.33% (1 of 300 legitimate postings flagged)
- Recall 36% as LIKELY_SCAM + 20% abstained to SUSPICIOUS — precision-first by design (spec §8)
- Adversarial set: 20/20 injection attempts left the verdict unchanged; 19 were reported as red flags
Full tables and history: docs/eval-results.md in the repository.
Security notes
- Prompt injection: untrusted-data framing, delimiter stripping, temperature 0, strict schema, deterministic tier overrides independent of the LLM, verbatim-quote verification.
- No URL fetching (SSRF surface avoided by design); paste-only input.
- PII: public excerpts are email/phone-redacted; logs never contain posting text.
Educational tool — not legal or financial advice. Analyses are stored (hash-deduplicated) to improve the corpus; request deletion anytime.