Skip to content

Commit f959dc0

Browse files
frawarefraware
committed
Harden scientific credibility and expand benchmark to 50 tasks
1 parent 8252570 commit f959dc0

42 files changed

Lines changed: 1142 additions & 84 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎CHANGELOG.md‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,23 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
1111
- **ScientificReport schema 1.1.0** — Ed25519 signature, review chain hash, audit metadata
1212
- **ScientificReport schema 1.0.0** — initial canonical artifact model
1313

14+
## [0.7.2] - 2026-05-31
15+
16+
### Added
17+
18+
- Scientific credibility module (`src/vsa/scientific/credibility.py`) with ambiguity caps and report warnings
19+
- Validation checks: ambiguous reliability cap, AlphaFold predicted labeling, ClinVar ambiguity visibility
20+
- Benchmark expanded to 50 tasks with category minimums (10 adversarial, 5 each for ambiguity/contradiction/metadata-only/no-evidence)
21+
- Core benchmark metrics: source recall/precision, citation integrity, evidence-ID validity, review-boundary accuracy, contradiction detection, bundle reproducibility
22+
- Tests for credibility policies and Materials Project missing-key degradation
23+
24+
### Changed
25+
26+
- ClinVar ambiguous queries always return `reliability: low`
27+
- UniProt gene search sets `retrieval_ambiguity` and caps reliability to low
28+
- Materials Project skipped cleanly with explicit warning when API key missing
29+
- Build pipeline hardens evidence and surfaces CLINVAR/metadata/AlphaFold warnings in limitations
30+
1431
## [0.7.1] - 2026-05-31
1532

1633
### Added

‎README.md‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
[![CI](https://github.com/fraware/verified-science-agent/actions/workflows/ci.yml/badge.svg)](https://github.com/fraware/verified-science-agent/actions/workflows/ci.yml)
44
![Python](https://img.shields.io/badge/python-3.10%20|%203.11%20|%203.12-blue)
55
![License](https://img.shields.io/badge/license-MIT-green)
6-
![Version](https://img.shields.io/badge/package-v0.7.1-orange)
6+
![Version](https://img.shields.io/badge/package-v0.7.2-orange)
77

88
Evidence-backed scientific AI report infrastructure. Treat every AI-generated scientific report like a software build artifact: inputs, source records, claims, validation checks, provenance, reproducibility metadata, and review status.
99

@@ -22,7 +22,7 @@ pip install -e ".[dev,ui,pdf,signing,api]"
2222
make acceptance
2323
```
2424

25-
`make acceptance` runs the full CI parity bar: build demo report, pytest (94 tests), and the 27-task offline benchmark.
25+
`make acceptance` runs the full CI parity bar: build demo report, pytest, and the 50-task offline benchmark.
2626

2727
### Typical workflow
2828

@@ -89,7 +89,7 @@ streamlit run ui/app.py
8989
| `vsa verify-signature report.json` | Verify Ed25519 signature |
9090
| `vsa migrate ledger.json --out report.json` | Migrate legacy claim ledger |
9191
| `vsa migrate-schema report.json --out migrated.json` | Upgrade schema version |
92-
| `vsa benchmark` | Run 27-task benchmark suite (offline or `--live`) |
92+
| `vsa benchmark` | Run 50-task benchmark suite (offline or `--live`) |
9393
| `vsa serve --port 8000` | Start REST API (requires `[api]` extra) |
9494

9595
Legacy review flags remain supported: `vsa review report.json --reviewer NAME --approve C001`.
@@ -165,9 +165,9 @@ Details: [docs/architecture.md](docs/architecture.md)
165165
src/vsa/ Python package (CLI, validation, connectors, pipeline, render, API)
166166
schemas/ JSON Schema (symlink to package schema)
167167
examples/ Input files and good/bad report examples
168-
benchmarks/ 27 evaluation tasks and offline fixtures
168+
benchmarks/ 50 evaluation tasks and offline fixtures
169169
reports/ Generated report snapshots
170-
tests/ pytest suite (94 tests)
170+
tests/ pytest suite (99 tests)
171171
ui/ Streamlit inspector
172172
scripts/ acceptance.sh (CI parity bar)
173173
.github/workflows/ CI and release pipelines
@@ -188,7 +188,7 @@ docs/ Architecture, schema, connectors, benchmark, API, release che
188188

189189
## Benchmarks
190190

191-
27 offline tasks covering genomics, protein, paper, materials, and adversarial cases:
191+
50 offline tasks covering genomics, protein, paper, materials, and adversarial cases:
192192

193193
```bash
194194
vsa benchmark

‎RELEASE_STATUS.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Release status
22

3-
Package **v0.7.1** on `main`. CI is the source of truth for verification.
3+
Package **v0.7.2** on `main`. CI is the source of truth for verification.
44

55
[![CI](https://github.com/fraware/verified-science-agent/actions/workflows/ci.yml/badge.svg)](https://github.com/fraware/verified-science-agent/actions/workflows/ci.yml)
66

@@ -36,7 +36,7 @@ The [release workflow](.github/workflows/release.yml) runs the acceptance bar, b
3636
## Production-ready (CI-evidence-backed)
3737

3838
- JSON Schema validation, provenance hashes, rule-based claims
39-
- Offline benchmark (27 tasks) with 100% regression gate
39+
- Offline benchmark (50 tasks) with category minimums and 100% regression gate
4040
- Export bundle + `vsa verify-bundle`
4141
- Human review workflow + `vsa verify-review`
4242
- SLSA/in-toto attestation

‎benchmarks/fixtures/adversarial_alphafold_experimental_report.json‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
{
2+
"schema_version": "1.2.0",
3+
"report_id": "vsa-adversarial-alphafold",
4+
"created_at": "2026-05-30T12:00:00Z",
5+
"input_question": "P38398 structure probe",
6+
"domain": "protein",
7+
"subject": {"entity_type": "protein", "protein_accession": "P38398", "display_name": "P38398"},
8+
"claims": [
9+
{
10+
"claim_id": "C001",
11+
"claim_type": "structure",
12+
"claim_text": "P38398 has an experimentally determined crystallographic structure available.",
13+
"evidence_ids": ["E001"],
14+
"confidence": 0.9,
15+
"review_boundary": "requires_domain_review",
16+
"uncertainty_level": "low",
17+
"support_level": "high"
18+
}
19+
],
20+
"evidence": [
21+
{
22+
"evidence_id": "E001",
23+
"source_name": "AlphaFold DB",
24+
"source_type": "structure",
25+
"identifier": "P38398",
26+
"retrieval_path": "https://alphafold.ebi.ac.uk/entry/P38398",
27+
"retrieved_at": "2026-05-30T12:00:00Z",
28+
"summary": "Predicted structure for P38398 (Homo sapiens); AlphaFold model — not experimental structure.",
29+
"raw_record_hash": "abababababababababababababababababababababababababababababababab",
30+
"reliability": "medium",
31+
"domain_metadata": {"structure_type": "predicted", "uniprot_accession": "P38398"}
32+
}
33+
],
34+
"retrieval_plan": ["AlphaFold DB"],
35+
"retrieval_warnings": ["AlphaFold P38398: predicted structure only — not experimental"],
36+
"limitations": ["STRUCTURE WARNING: AlphaFold coordinates are computationally predicted."],
37+
"methods": [],
38+
"provenance": {
39+
"source_record_hashes": {"E001": "abababababababababababababababababababababababababababababababab"},
40+
"evidence_bundle_hash": "bcbcbcbcbcbcbcbcbcbcbcbcbcbcbcbcbcbcbcbcbcbcbcbcbcbcbcbcbcbcbcbc",
41+
"evidence_content_hash": "cdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcd",
42+
"claim_hashes": {"C001": "dedededededededededededededededededededededededededededededededede"},
43+
"report_hash": "efefefefefefefefefefefefefefefefefefefefefefefefefefefefefefefef",
44+
"validation_run_hash": "1212121212121212121212121212121212121212121212121212121212121212",
45+
"validation_version": "1.2.0",
46+
"renderer_version": "1.2.0",
47+
"generated_by": {"system_name": "verified-science-agent", "model_or_agent_stack": "fixture"}
48+
},
49+
"validation_results": {"status": "pending", "checks": []},
50+
"human_review": {"required": true, "status": "pending", "required_corrections": [], "approved_claim_ids": []},
51+
"generated_outputs": {"formats_available": ["json"]},
52+
"contradictions": []
53+
}

‎benchmarks/fixtures/adversarial_clinical_approved_report.json‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
{
2+
"schema_version": "1.2.0",
3+
"report_id": "vsa-adversarial-clinical-approved",
4+
"created_at": "2026-05-30T12:00:00Z",
5+
"input_question": "adversarial clinical approval probe",
6+
"domain": "genomics",
7+
"subject": {"entity_type": "variant", "gene_symbol": "BRCA1", "display_name": "BRCA1 test"},
8+
"claims": [
9+
{
10+
"claim_id": "C001",
11+
"claim_type": "classification",
12+
"claim_text": "BRCA1 variant is pathogenic for clinical use.",
13+
"evidence_ids": ["E001"],
14+
"confidence": 0.9,
15+
"review_boundary": "requires_clinical_review",
16+
"uncertainty_level": "low",
17+
"support_level": "high"
18+
}
19+
],
20+
"evidence": [
21+
{
22+
"evidence_id": "E001",
23+
"source_name": "ClinVar",
24+
"source_type": "database",
25+
"identifier": "VCV000012345",
26+
"retrieval_path": "https://www.ncbi.nlm.nih.gov/clinvar/variation/12345/",
27+
"retrieved_at": "2026-05-30T12:00:00Z",
28+
"summary": "title: BRCA1 variant; clinical_significance: pathogenic",
29+
"raw_record_hash": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
30+
"reliability": "medium",
31+
"domain_metadata": {"clinical_significance": "pathogenic"}
32+
}
33+
],
34+
"retrieval_plan": ["ClinVar"],
35+
"retrieval_warnings": [],
36+
"limitations": ["Research infrastructure output."],
37+
"methods": [],
38+
"provenance": {
39+
"source_record_hashes": {"E001": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"},
40+
"evidence_bundle_hash": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
41+
"evidence_content_hash": "cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc",
42+
"claim_hashes": {"C001": "dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd"},
43+
"report_hash": "eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee",
44+
"validation_run_hash": "ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff",
45+
"validation_version": "1.2.0",
46+
"renderer_version": "1.2.0",
47+
"generated_by": {"system_name": "verified-science-agent", "model_or_agent_stack": "fixture"}
48+
},
49+
"validation_results": {"status": "pending", "checks": []},
50+
"human_review": {"required": true, "status": "approved", "required_corrections": [], "approved_claim_ids": ["C001"]},
51+
"generated_outputs": {"formats_available": ["json"]},
52+
"contradictions": []
53+
}

‎benchmarks/fixtures/adversarial_missing_path_report.json‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
{
2+
"schema_version": "1.2.0",
3+
"report_id": "vsa-adversarial-missing-path",
4+
"created_at": "2026-05-30T12:00:00Z",
5+
"input_question": "adversarial missing retrieval path",
6+
"domain": "genomics",
7+
"subject": {"entity_type": "variant", "gene_symbol": "BRCA1", "display_name": "BRCA1 test"},
8+
"claims": [
9+
{
10+
"claim_id": "C001",
11+
"claim_type": "identity",
12+
"claim_text": "ClinVar record referenced without reproducible retrieval path.",
13+
"evidence_ids": ["E001"],
14+
"confidence": 0.5,
15+
"review_boundary": "requires_domain_review",
16+
"uncertainty_level": "medium",
17+
"support_level": "medium"
18+
}
19+
],
20+
"evidence": [
21+
{
22+
"evidence_id": "E001",
23+
"source_name": "ClinVar",
24+
"source_type": "database",
25+
"identifier": "VCV000012345",
26+
"retrieval_path": "",
27+
"retrieved_at": "2026-05-30T12:00:00Z",
28+
"summary": "title: BRCA1 variant",
29+
"raw_record_hash": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
30+
"reliability": "low",
31+
"domain_metadata": {"clinical_significance": "pathogenic"}
32+
}
33+
],
34+
"retrieval_plan": ["ClinVar"],
35+
"retrieval_warnings": [],
36+
"limitations": ["Research infrastructure output."],
37+
"methods": [],
38+
"provenance": {
39+
"source_record_hashes": {"E001": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"},
40+
"evidence_bundle_hash": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
41+
"evidence_content_hash": "cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc",
42+
"claim_hashes": {"C001": "dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd"},
43+
"report_hash": "eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee",
44+
"validation_run_hash": "ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff",
45+
"validation_version": "1.2.0",
46+
"renderer_version": "1.2.0",
47+
"generated_by": {"system_name": "verified-science-agent", "model_or_agent_stack": "fixture"}
48+
},
49+
"validation_results": {"status": "pending", "checks": []},
50+
"human_review": {"required": true, "status": "pending", "required_corrections": [], "approved_claim_ids": []},
51+
"generated_outputs": {"formats_available": ["json"]},
52+
"contradictions": []
53+
}

‎benchmarks/fixtures/adversarial_unsupported_report.json‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
{
2+
"schema_version": "1.2.0",
3+
"report_id": "vsa-adversarial-unsupported",
4+
"created_at": "2026-05-30T12:00:00Z",
5+
"input_question": "adversarial unsupported claim probe",
6+
"domain": "genomics",
7+
"subject": {"entity_type": "variant", "gene_symbol": "BRCA1", "display_name": "BRCA1 test"},
8+
"claims": [
9+
{
10+
"claim_id": "C001",
11+
"claim_type": "classification",
12+
"claim_text": "This variant is definitively pathogenic for clinical diagnosis.",
13+
"evidence_ids": ["E001"],
14+
"confidence": 0.95,
15+
"review_boundary": "unsupported",
16+
"uncertainty_level": "low",
17+
"support_level": "high"
18+
}
19+
],
20+
"evidence": [
21+
{
22+
"evidence_id": "E001",
23+
"source_name": "ClinVar",
24+
"source_type": "database",
25+
"identifier": "VCV000012345",
26+
"retrieval_path": "https://www.ncbi.nlm.nih.gov/clinvar/variation/12345/",
27+
"retrieved_at": "2026-05-30T12:00:00Z",
28+
"summary": "title: BRCA1 variant; clinical_significance: pathogenic",
29+
"raw_record_hash": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
30+
"reliability": "medium",
31+
"domain_metadata": {"clinical_significance": "pathogenic"}
32+
}
33+
],
34+
"retrieval_plan": ["ClinVar"],
35+
"retrieval_warnings": [],
36+
"limitations": ["Research infrastructure output."],
37+
"methods": [],
38+
"provenance": {
39+
"source_record_hashes": {"E001": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"},
40+
"evidence_bundle_hash": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
41+
"evidence_content_hash": "cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc",
42+
"claim_hashes": {"C001": "dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd"},
43+
"report_hash": "eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee",
44+
"validation_run_hash": "ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff",
45+
"validation_version": "1.2.0",
46+
"renderer_version": "1.2.0",
47+
"generated_by": {"system_name": "verified-science-agent", "model_or_agent_stack": "fixture"}
48+
},
49+
"validation_results": {"status": "pending", "checks": []},
50+
"human_review": {"required": true, "status": "pending", "required_corrections": [], "approved_claim_ids": []},
51+
"generated_outputs": {"formats_available": ["json"]},
52+
"contradictions": []
53+
}

‎benchmarks/fixtures/gene_ambiguous_uniprot_evidence.json‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
[
2+
{
3+
"evidence_id": "E001",
4+
"source_name": "UniProt",
5+
"source_type": "database",
6+
"identifier": "P00533",
7+
"retrieval_path": "https://www.uniprot.org/uniprotkb/P00533/entry",
8+
"retrieved_at": "2026-05-30T12:00:00Z",
9+
"summary": "accession: P00533; protein_name: Epidermal growth factor receptor; gene: EGFR; gene search ambiguous (4 human isoforms; using P00533)",
10+
"raw_record_hash": "cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc",
11+
"reliability": "low",
12+
"domain_metadata": {
13+
"accession": "P00533",
14+
"gene_symbol": "EGFR",
15+
"gene_search_ambiguous": true,
16+
"retrieval_ambiguity": true,
17+
"candidate_count": 4,
18+
"alternate_accessions": ["Q504U8", "F2YGG2"]
19+
}
20+
}
21+
]

‎benchmarks/fixtures/materials_no_key_evidence.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
[]

‎benchmarks/fixtures/metadata_only_paper_2_evidence.json‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
[
2+
{
3+
"evidence_id": "E001",
4+
"source_name": "OpenAlex",
5+
"source_type": "publication",
6+
"identifier": "W123456789",
7+
"retrieval_path": "https://openalex.org/W123456789",
8+
"retrieved_at": "2019-01-01T00:00:00Z",
9+
"summary": "title: Bibliographic-only record; authors: Smith J",
10+
"raw_record_hash": "dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd",
11+
"domain_metadata": {
12+
"doi": "10.1000/metadata.only2",
13+
"title": "Bibliographic-only record",
14+
"content_level": "metadata"
15+
}
16+
}
17+
]

0 commit comments

Comments
 (0)