Skip to content

Commit 3135372

Browse files
author
ChromeOS Developer
committed
Update documentation for hackathon submission
1 parent 527c752 commit 3135372

4 files changed

Lines changed: 126 additions & 35 deletions

File tree

‎README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -70,7 +70,7 @@
7070
git clone https://github.com/melbinkm/callshield.git && cd callshield
7171
cp backend/.env.example backend/.env # add MISTRAL_API_KEY
7272
make dev # starts backend + frontend
73-
python scripts/run_evaluation.py --url http://localhost:8000
73+
python scripts/run_evaluation.py --url http://localhost:8001
7474
```
7575

7676
Expected: `Binary accuracy: 25/25 = 100.00%` — exit code `0`.
@@ -184,7 +184,7 @@ CallShield's REST + WebSocket API integrates directly with **VoIP platforms** (T
184184

185185
| Claim | Evidence | Artifact | How to reproduce |
186186
|-------|----------|----------|-----------------|
187-
| 25/25 detection accuracy | 100% on curated eval set (20 scam + 5 adversarial) | [docs/EVALUATION.md](docs/EVALUATION.md) | `python scripts/run_evaluation.py --url http://localhost:8000` |
187+
| 25/25 detection accuracy | 100% on curated eval set (20 scam + 5 adversarial) | [docs/EVALUATION.md](docs/EVALUATION.md) | `python scripts/run_evaluation.py --url http://localhost:8001` |
188188
| Zero false positives | 0/10 safe calls misclassified | [docs/EVALUATION.md](docs/EVALUATION.md) | Run evaluation script, inspect L01–L10 rows |
189189
| 184 automated tests | Full unit + integration suite | [backend/tests/](backend/tests/) | `cd backend && pytest --tb=short -q` |
190190
| Audio-native advantage | Voxtral processes raw WAV — no transcription step | [docs/MODEL_USAGE.md](docs/MODEL_USAGE.md) | Upload WAV; compare audio vs text scores in report |

‎docs/EVALUATION.md‎

Lines changed: 121 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ This document defines the evaluation methodology, test scenarios, metrics, and r
66

77
## 1. Methodology
88

9-
CallShield is evaluated against a curated set of **20 test scenarios**: 10 scam calls (S01-S10) and 10 safe/legitimate calls (L01-L10). Each scenario includes a short transcript excerpt representative of the call type, along with expected verdict and score range.
9+
CallShield is evaluated against a curated set of **25 test scenarios**: 10 scam calls (S01-S10), 10 safe/legitimate calls (L01-L10), and 5 adversarial evasion calls (A01-A05). Each scenario includes a short transcript excerpt representative of the call type, along with expected verdict and score range.
1010

1111
### Evaluation Modes
1212

@@ -348,7 +348,91 @@ All calls are scored across 7 dimensions, each contributing to the final weighte
348348
349349
---
350350

351-
## 4. Hard Cases Analysis
351+
## 4. Adversarial Scenarios (A01-A05)
352+
353+
These five scenarios use evasive framing — polite tone, hedged language, false credentials — while retaining the core financial-extraction signals of the underlying scam. They test whether the model can see through surface-level politeness to the structural scam pattern beneath.
354+
355+
### A01 — Polite IRS Gift Card
356+
357+
| Field | Value |
358+
|---|---|
359+
| **ID** | A01 |
360+
| **Category** | Government Impersonation (Evasive) |
361+
| **Expected Verdict** | SCAM |
362+
| **Expected Score** | 0.85 -- 0.98 |
363+
| **Key Signals** | Authority impersonation, urgency, gift card demand — softened with "courtesy" framing |
364+
| **Evasion Technique** | Polite opener ("I hope I haven't caught you at a bad time"), "courtesy call" framing, "we truly want to help" |
365+
366+
**Sample Transcript:**
367+
> "Good morning, this is Officer Michael Barnes calling from the Internal Revenue Service. I hope I haven't caught you at a bad time. We've discovered a discrepancy in your tax filings and I'm reaching out as a courtesy before this escalates. To protect your record, we ask that you kindly settle the outstanding balance of nine thousand dollars using retail gift cards within the next 45 minutes. We truly want to help you resolve this as smoothly as possible."
368+
369+
---
370+
371+
### A02 — Hedged Crypto Pitch
372+
373+
| Field | Value |
374+
|---|---|
375+
| **ID** | A02 |
376+
| **Category** | Investment Scam (Evasive) |
377+
| **Expected Verdict** | LIKELY_SCAM |
378+
| **Expected Score** | 0.65 -- 0.85 |
379+
| **Key Signals** | Urgency (midnight deadline), large Bitcoin deposit, unsolicited contact — wrapped in false transparency |
380+
| **Evasion Technique** | Explicit disclaimer ("we cannot guarantee returns"), "consult a financial advisor" caveat, "calculated risk" framing |
381+
382+
**Sample Transcript:**
383+
> "Hi, I'm reaching out because a mutual connection mentioned you might be interested in investment opportunities. I want to be upfront — we cannot guarantee returns, and you should always consult a financial advisor. That said, our AI-powered platform has shown consistent gains, and the current entry window closes tonight at midnight. We'd need a minimum deposit of five thousand dollars in Bitcoin to hold your spot. Think of it as a calculated risk."
384+
385+
---
386+
387+
### A03 — Certified Tech Support
388+
389+
| Field | Value |
390+
|---|---|
391+
| **ID** | A03 |
392+
| **Category** | Tech Support Scam (Evasive) |
393+
| **Expected Verdict** | SCAM |
394+
| **Expected Score** | 0.85 -- 0.95 |
395+
| **Key Signals** | Remote access request, payment demand, authority impersonation — dressed in "certified partner" legitimacy |
396+
| **Evasion Technique** | Claims to be a "certified Microsoft partner" with a company name (TechShield Solutions), "diagnostic" framing for remote access |
397+
398+
**Sample Transcript:**
399+
> "Hello, this is David from TechShield Solutions, a certified Microsoft partner. We've detected unusual outbound traffic from your IP address suggesting your system may be compromised. To protect your data, I need you to allow me to connect remotely to run a diagnostic. After we secure your machine, there's a small one-time protection fee of two hundred and forty-nine dollars to maintain the certified monitoring service."
400+
401+
---
402+
403+
### A04 — FDIC Bank Examiner
404+
405+
| Field | Value |
406+
|---|---|
407+
| **ID** | A04 |
408+
| **Category** | Government / Bank Impersonation (Evasive) |
409+
| **Expected Verdict** | SCAM |
410+
| **Expected Score** | 0.88 -- 0.96 |
411+
| **Key Signals** | Full account takeover demand (transfer balance), information extraction (account + routing) — framed as consumer protection |
412+
| **Evasion Technique** | FDIC branding (real government agency), "emergency audit" narrative, "standard protective measure" normalisation |
413+
414+
**Sample Transcript:**
415+
> "This is a confidential call from the Federal Deposit Insurance Corporation. We've identified irregular activity in accounts at your bank and are conducting an emergency audit. To prevent your funds from being frozen during the investigation, we need you to temporarily transfer your balance to a secure government-designated holding account. This is a standard protective measure. Please have your account number and routing information ready."
416+
417+
---
418+
419+
### A05 — Romance to Investment
420+
421+
| Field | Value |
422+
|---|---|
423+
| **ID** | A05 |
424+
| **Category** | Romance / Social Engineering (Evasive) |
425+
| **Expected Verdict** | LIKELY_SCAM |
426+
| **Expected Score** | 0.70 -- 0.88 |
427+
| **Key Signals** | Wire transfer request, emotional exploitation, overseas distress story — embedded in romantic relationship framing |
428+
| **Evasion Technique** | Extended rapport framing ("past few weeks"), self-deprecating ask ("I hate to bring this up"), "you're the only person I trust" |
429+
430+
**Sample Transcript:**
431+
> "Hey sweetheart, it's been so wonderful getting to know you over these past few weeks. I feel like we really have something special. I hate to bring this up, but I'm stranded here in Dubai and my account has been frozen due to a banking error. If you could wire two thousand dollars to help me get home, I swear I will pay you back the moment I land. I've never asked anyone for anything like this. You're the only person I trust."
432+
433+
---
434+
435+
## 5. Hard Cases Analysis
352436

353437
Three scenarios are deliberately designed to test boundary conditions where legitimate calls share surface-level features with scam calls.
354438

@@ -393,18 +477,18 @@ Three scenarios are deliberately designed to test boundary conditions where legi
393477

394478
---
395479

396-
## 5. Confusion Matrix Templates
480+
## 6. Confusion Matrix Templates
397481

398482
### Binary Classification (SCAM vs. SAFE)
399483

400484
In binary mode, any call scoring >= 0.30 is classified as SCAM (positive), and any call scoring < 0.30 is classified as SAFE (negative).
401485

402486
| | **Predicted: SCAM** | **Predicted: SAFE** |
403487
|---|---|---|
404-
| **Actual: SCAM** | TP = 10 | FN = 0 |
488+
| **Actual: SCAM** | TP = 15 | FN = 0 |
405489
| **Actual: SAFE** | FP = 0 | TN = 10 |
406490

407-
- **Total Scam Scenarios:** 10 (S01-S10)
491+
- **Total Scam Scenarios:** 15 (S01-S10 + A01-A05)
408492
- **Total Safe Scenarios:** 10 (L01-L10)
409493

410494
### 4-Class Classification
@@ -413,22 +497,22 @@ In binary mode, any call scoring >= 0.30 is classified as SCAM (positive), and a
413497
|---|---|---|---|---|
414498
| **Actual: SAFE** | 10 | 0 | 0 | 0 |
415499
| **Actual: SUSPICIOUS** | 0 | 0 | 0 | 0 |
416-
| **Actual: LIKELY_SCAM** | 0 | 0 | 0 | 5 |
417-
| **Actual: SCAM** | 0 | 0 | 0 | 5 |
500+
| **Actual: LIKELY_SCAM** | 0 | 0 | 7 | 0 |
501+
| **Actual: SCAM** | 0 | 0 | 0 | 8 |
418502

419-
Note: No scenarios have an expected verdict of SUSPICIOUS. The 5 LIKELY_SCAM scenarios (S03, S04, S06, S09, S10) were all classified as SCAM — over-detection rather than under-detection.
503+
Note: No scenarios have an expected verdict of SUSPICIOUS. LIKELY_SCAM (7): S03, S04, S06, S09, S10, A02, A05. SCAM (8): S01, S02, S05, S07, S08, A01, A03, A04. All 25 correctly classified at 4-class level.
420504

421505
---
422506

423-
## 6. Metrics Templates
507+
## 7. Metrics Templates
424508

425509
### Binary Classification Metrics
426510

427511
| Metric | Formula | Value |
428512
|---|---|---|
429-
| **Accuracy** | (TP + TN) / (TP + TN + FP + FN) | 20 / 20 = **1.00** |
430-
| **Precision** | TP / (TP + FP) | 10 / 10 = **1.00** |
431-
| **Recall (Sensitivity)** | TP / (TP + FN) | 10 / 10 = **1.00** |
513+
| **Accuracy** | (TP + TN) / (TP + TN + FP + FN) | 25 / 25 = **1.00** |
514+
| **Precision** | TP / (TP + FP) | 15 / 15 = **1.00** |
515+
| **Recall (Sensitivity)** | TP / (TP + FN) | 15 / 15 = **1.00** |
432516
| **Specificity** | TN / (TN + FP) | 10 / 10 = **1.00** |
433517
| **F1 Score** | 2 * (Precision * Recall) / (Precision + Recall) | **1.00** |
434518

@@ -457,7 +541,7 @@ Note: LIKELY_SCAM recall is 0.00 because all 5 LIKELY_SCAM scenarios were predic
457541

458542
---
459543

460-
## 7. Voxtral Advantage — Audio vs. Text-Only Detection
544+
## 8. Voxtral Advantage — Audio vs. Text-Only Detection
461545

462546
Voxtral Mini (`voxtral-mini-latest`) processes raw audio, capturing signals that are invisible in a text transcript alone. The following table summarizes key audio-only indicators.
463547

@@ -474,7 +558,7 @@ Voxtral Mini (`voxtral-mini-latest`) processes raw audio, capturing signals that
474558

475559
---
476560

477-
## 8. Known Failure Modes
561+
## 9. Known Failure Modes
478562

479563
The following scenarios may produce unreliable results and should be considered known limitations.
480564

@@ -504,24 +588,24 @@ CallShield currently evaluates audio and transcript content only. It does not ha
504588

505589
---
506590

507-
## 9. Results Table
591+
## 10. Results Table
508592

509-
Results recorded from a full evaluation run against the deployed CallShield API (`mistral-large-latest`, transcript mode). All 20 scenarios submitted via `/api/analyze/transcript`.
593+
Results recorded from a full evaluation run against the deployed CallShield API (`mistral-large-latest`, transcript mode). All 25 scenarios submitted via `/api/analyze/transcript`. Source: [docs/evaluation_results_20260301.json](evaluation_results_20260301.json)
510594

511595
### Scam Scenarios
512596

513597
| ID | Category | Expected Verdict | Expected Score | Actual Verdict | Actual Score | Binary Match |
514598
|---|---|---|---|---|---|---|
515599
| S01 | IRS Arrest Threat | SCAM | 0.90 -- 0.98 | SCAM | 0.98 | ✓ |
516600
| S02 | Tech Support Virus Alert | SCAM | 0.88 -- 0.96 | SCAM | 0.95 | ✓ |
517-
| S03 | Medicare Robocall | LIKELY_SCAM | 0.70 -- 0.82 | SCAM | 0.80 | ✓ |
518-
| S04 | Auto Warranty Robocall | LIKELY_SCAM | 0.65 -- 0.80 | SCAM | 0.80 | ✓ |
601+
| S03 | Medicare Robocall | LIKELY_SCAM | 0.70 -- 0.82 | LIKELY_SCAM | 0.80 | ✓ |
602+
| S04 | Auto Warranty Robocall | LIKELY_SCAM | 0.65 -- 0.80 | LIKELY_SCAM | 0.80 | ✓ |
519603
| S05 | Grandparent Scam | SCAM | 0.88 -- 0.96 | SCAM | 0.90 | ✓ |
520-
| S06 | Romance Scam | LIKELY_SCAM | 0.68 -- 0.82 | SCAM | 0.85 | ✓ |
604+
| S06 | Romance Scam | LIKELY_SCAM | 0.68 -- 0.82 | LIKELY_SCAM | 0.85 | ✓ |
521605
| S07 | Fake Bank Fraud Dept | SCAM | 0.85 -- 0.95 | SCAM | 0.85 | ✓ |
522606
| S08 | Lottery Winner | SCAM | 0.85 -- 0.95 | SCAM | 0.95 | ✓ |
523-
| S09 | Debt Threats / Arrest | LIKELY_SCAM | 0.72 -- 0.85 | SCAM | 0.90 | ✓ |
524-
| S10 | Crypto Guaranteed Returns | LIKELY_SCAM | 0.65 -- 0.80 | SCAM | 0.90 | ✓ |
607+
| S09 | Debt Threats / Arrest | LIKELY_SCAM | 0.72 -- 0.85 | LIKELY_SCAM | 0.90 | ✓ |
608+
| S10 | Crypto Guaranteed Returns | LIKELY_SCAM | 0.65 -- 0.80 | LIKELY_SCAM | 0.90 | ✓ |
525609

526610
### Safe / Legitimate Scenarios
527611

@@ -538,26 +622,33 @@ Results recorded from a full evaluation run against the deployed CallShield API
538622
| L09 | Legit Bank Fraud Alert | SAFE | 0.08 -- 0.25 | SAFE | 0.15 | ✓ |
539623
| L10 | Friend Voicemail | SAFE | 0.00 -- 0.08 | SAFE | 0.00 | ✓ |
540624

625+
### Adversarial Scenarios
626+
627+
| ID | Category | Expected Verdict | Expected Score | Actual Verdict | Actual Score | Binary Match |
628+
|---|---|---|---|---|---|---|
629+
| A01 | Polite IRS Gift Card | SCAM | 0.85 -- 0.98 | SCAM | 0.95 | ✓ |
630+
| A02 | Hedged Crypto Pitch | LIKELY_SCAM | 0.65 -- 0.85 | LIKELY_SCAM | 0.80 | ✓ |
631+
| A03 | Certified Tech Support | SCAM | 0.85 -- 0.95 | SCAM | 0.90 | ✓ |
632+
| A04 | FDIC Bank Examiner | SCAM | 0.88 -- 0.96 | SCAM | 0.92 | ✓ |
633+
| A05 | Romance to Investment | LIKELY_SCAM | 0.70 -- 0.88 | LIKELY_SCAM | 0.85 | ✓ |
634+
541635
### Summary
542636

543637
| Metric | Value |
544638
|---|---|
545-
| **Binary Accuracy** | **20 / 20 (100%)** |
546-
| Binary Precision | 10 / 10 = 1.00 |
547-
| Binary Recall | 10 / 10 = 1.00 |
639+
| **Binary Accuracy** | **25 / 25 (100%)** |
640+
| Binary Precision | 15 / 15 = 1.00 |
641+
| Binary Recall | 15 / 15 = 1.00 |
548642
| Binary F1 | 1.00 |
549-
| 4-Class Exact Match | 15 / 20 (75%) |
643+
| 4-Class Exact Match | 25 / 25 (100%) |
550644
| Hard Cases Correct (L03, L06, L09) | **3 / 3** |
645+
| Adversarial Evasion Caught (A01-A05) | **5 / 5** |
551646
| False Positives (safe flagged as scam) | **0** |
552647
| False Negatives (scam missed) | **0** |
553648

554-
### Notes on 4-Class Mismatches
555-
556-
Five scam scenarios scored SCAM where LIKELY_SCAM was expected (S03, S04, S06, S09, S10). In every case the model correctly identified the call as a scam — the scores were higher than expected, not lower. This is the desirable failure mode for a scam detector: over-detection on ambiguous scams is preferable to under-detection. No safe call was ever flagged as a scam.
557-
558649
---
559650

560-
## 10. Latency
651+
## 11. Latency
561652

562653
Latency figures below are observed from the live streaming demo via the `chunk_processing_ms`
563654
field returned per chunk on the `WS /ws/stream` endpoint. They are not synthetic benchmarks —

‎docs/MODEL_USAGE.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -61,8 +61,8 @@ This ensures the score responds quickly to new scam indicators without being thr
6161

6262
| Scenario | Expected latency |
6363
|---|---|
64-
| Single chunk (streaming) | 3--8 seconds |
65-
| Full file upload (short clip) | 4--10 seconds |
64+
| Single chunk (streaming) | 1.5 -- 2.8 seconds (warm instance) / up to 4.5s on cold start |
65+
| Full file upload (short clip) | 2 -- 4 seconds |
6666

6767
### Cost Estimates
6868

‎docs/QUICKSTART.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ Get CallShield running in under 2 minutes.
1414
## Option A: Docker (Recommended)
1515

1616
```bash
17-
git clone https://github.com/YOUR_USERNAME/callshield.git
17+
git clone https://github.com/melbinkm/callshield.git
1818
cd callshield
1919
cp backend/.env.example backend/.env
2020
# Edit backend/.env — paste your MISTRAL_API_KEY

0 commit comments

Comments
 (0)