Skip to content

CHANGELOG.md

Latest commit

 

History

History
116 lines (102 loc) · 6.33 KB

CHANGELOG.md

File metadata and controls

116 lines (102 loc) · 6.33 KB
Raw

Changelog

All notable changes to this project are documented here. Format follows Keep a Changelog.

[Unreleased]

Changed

  • Step 2 "Run Agent 2" now shows an in-flight indicator ("Analyzing duplicates — this takes 1–2 minutes…") and the button is disabled while the synchronous run is in progress (IST_CD/templates/_step2_review.html). Added the missing standard .htmx-indicator CSS rules (IST_CD/static/style.css), which also fixes the previously non-functional upload/search/generate indicators in ingest.html / outreach.html.
  • Step 2 dedup scoring policy (POC/dedup_agent/scoring.py): a conflicting email/phone no longer forces a pair to "likely distinct" (confidence 0.2). People change jobs, numbers, and mailboxes, so conflict now caps the pair below the auto-duplicate threshold and routes it to the judge/review queue instead of dismissing it. scorer_version bumped to deterministic-v2, which invalidates previously cached pair assessments.

Removed

  • Owner-only building materials untracked from git (kept on the owner's local disk; history not rewritten): data-samples/, docs/, archive/, reference/, .agents/, .superpowers/, ENGINEER/, tools/, and root meta files AGENTS.md, AI_CONTEXT.md, CONTEXT.md, MISSION.md, NOTES.md, RESOURCES.md — all now in .gitignore and excluded from future release packages. The repo and IT package now contain only the functional codebase plus deployment-necessary files. README.md updated to match (directory table, quick start, run instructions).
  • Gemini Step 1 public-research provider (gemini_public_research.py deleted; git history preserves it): the Gemini API is unreachable from the HK office LAN. Perplexity Sonar is now the default Step 1 provider (STEP1_RESEARCH_PROVIDER and CLI/UI defaults); GEMINI_API_KEY_IST / GEMINI_STEP1_MODEL are replaced by PERPLEXITY_API_KEY / PERPLEXITY_MODEL in .env.template and DEPLOYMENT docs.

Added

  • Bulk database save: new save_contacts() in IST_CD/db.py appends many contacts with a single workbook read/write instead of one full rewrite per contact (IST_CD/db.py:475). Single-contact save_contact() behavior is unchanged.
  • Duplicate protection on legacy Excel import: rows whose name+company already exist in the master database, or repeat within the uploaded file, are skipped and counted. Import response now reports "Imported X contacts. Skipped Y duplicates. Total: Z" (IST_CD/app.py:450).
  • Explicit user-facing 409 warning on import when master_database.xlsx is locked (e.g. open in Excel), matching the existing lock-handling convention.
  • POC/tests/test_bulk_import.py: coverage for bulk save (sequential IDs, append-without-touching-existing) and import duplicate skipping.
  • POC/tests/robustness_ist50.py: offline robustness runner used to test the pipeline against data-samples/ist/IST_test_sample_50.xlsx (49 contacts).
  • SiliconFlow Step 1 public-research provider (POC/sprint_2/siliconflow_public_research.py): a deterministic search-fetch-evaluate pipeline as an alternative to the Gemini Google-Search-grounding path. Code builds queries, searches the web (Tavily when configured, else DuckDuckGo via the optional ddgs package), fetches and extracts public pages, and calls the SiliconFlow chat API only to judge the fetched evidence; verbatim-citation checks, two-domain corroboration, and confidence tiers (>=85 strong, 60-84 review, <60 quarantine) are enforced in code, with one bounded retry round on insufficient evidence. Enable with STEP1_RESEARCH_PROVIDER=siliconflow; requires SILICONFLOW_API_KEY (plus optional SILICONFLOW_MODEL), and TAVILY_API_KEY is the optional preferred search backend. No new pinned dependencies (ddgs/trafilatura/beautifulsoup4 are optional imports).
  • Perplexity Sonar Step 1 public-research provider (POC/sprint_2/perplexity_public_research.py): a minimal one-call alternative to the Gemini path — one grounded chat completion (Perplexity runs the web search itself), then citation extraction, verdict mapping, and the shared identity guards (matched_identity gate plus the name-token-overlap veto reused from the SiliconFlow provider). No fetch stage, no corroboration rule. Enable with STEP1_RESEARCH_PROVIDER=perplexity; requires PERPLEXITY_API_KEY (plus optional PERPLEXITY_MODEL, default sonar-pro). No new dependencies.

Fixed

  • DEPLOYMENT.md / DEPLOYMENT.html: corrected the OCR setup to match actual behavior (POC/sprint_1/ocr_enhanced.py). TESSDATA_PREFIX no longer claims a nonexistent "bundled tessdata" default — the doc now instructs setting it to the bundled tessdata\ folder (ships eng, chi_sim, chi_tra), documents that OCR always runs chi_sim+chi_tra+eng, and adds Tesseract/tessdata checks to the §7 acceptance checklist.
  • Removed hardcoded owner-machine paths (C:\Users\fengy\...) from the Step 1 corrections-cache lookup in POC/sprint_2/verify_contacts.py. The cache now defaults to the repo-relative data\pre_check_results.json and data\final_corrections.json copies, with optional STEP1_PRE_CHECK_CACHE_PATH / STEP1_FINAL_CORRECTIONS_PATH env overrides — so the lookup behaves identically on any host.
  • SiliconFlow Step 1 provider: wrong-person evidence can no longer drive corrections. The LLM evaluation must now gate every citation with matched_identity (same_person/different_person/uncertain; name+company+title judged jointly, company match required), and a deterministic veto drops any proposed name correction that shares no name token with the stored name — fixing a live A/B failure where "Luke Li" was "corrected" to a different person ("Lei Zhang").

Performance

  • Legacy Excel import is ~51x faster at 50 contacts (3.65s -> 0.07s in local benchmark); the improvement grows with batch size, making ~2000-row imports practical (previously tens of minutes with browser-timeout and partial-import risk).

Notes for IT

  • No new dependencies (requirements.txt unchanged), no new required env vars (the STEP1_PRE_CHECK_CACHE_PATH / STEP1_FINAL_CORRECTIONS_PATH overrides are optional), no database schema change. Update per DEPLOYMENT.md section 5: replace code, keep the existing data\ folder, run the smoke test.