Skip to content

Improve collection evidence support#49

Draft
giaphutran12 wants to merge 1 commit into
codex/collection-source-coherencefrom
codex/collection-evidence-support
Draft

Improve collection evidence support#49
giaphutran12 wants to merge 1 commit into
codex/collection-source-coherencefrom
codex/collection-evidence-support

Conversation

@giaphutran12
Copy link
Copy Markdown
Collaborator

Summary

  • add evidence for URL-valued cells when the extractor returns a URL field without a separate evidence quote
  • merge duplicate entity rows that differ only by corporate suffixes like Corp. or Corporation
  • prefer later, stronger canonical source pages when duplicate rows disagree
  • update stale focused benchmark answer keys for Paddle's current pricing page and NVIDIA's May 20, 2026 Q1 FY2027 earnings release

Verification

  • npm --prefix backend test (76/76)
  • npm --prefix backend run build
  • make verify-self-healing
  • node --check benchmarks/dataset-agent/run-benchmark.mjs
  • git diff --check

Real Agent/self-healing evidence:

  • benchmark-results/collection-evidence-support-mcp-20260523-001: mcp-docs-pages passed 1/1, factual 1.0, entity 1.0, domain 1.0, evidence 1.0, claim 1.0, cost $0.022256
  • benchmark-results/collection-evidence-support-earnings-20260523-003: earnings-release-pages passed 1/1 with 3 rows, NVIDIA Q1 FY2027 source only, cost $0.067237
  • benchmark-results/collection-evidence-support-4prompt-20260523-002: focused 4-prompt collection-self-heal run passed 4/4, factual 1.0, entity 1.0, domain 1.0, evidence 1.0, claim 1.0, cost $0.193776

Notes

No auto-merge. This stacks on PR #48 and keeps the collection runtime behind explicit Agent opt-in.

@giaphutran12 giaphutran12 self-assigned this May 22, 2026
@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented May 22, 2026

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 9d29c2d1-b51a-4aa8-8218-1e8dbef9cb30

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/collection-evidence-support

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant