Skip to content

expermimental build#15607

Draft
ychhabria wants to merge 2 commits intomainfrom
ychhabria/expermimental-build
Draft

expermimental build#15607
ychhabria wants to merge 2 commits intomainfrom
ychhabria/expermimental-build

Conversation

@ychhabria
Copy link

@ychhabria ychhabria commented Mar 24, 2026

Summary

  • tune Codex file-search defaults from a realistic quality/performance eval loop
  • raise the default result budget to 100 across the shared file-search crate and the TUI/app-server entrypoints
  • set the shared file-search default worker count to 8

Evaluation

  • built a realistic search scenario set centered on Codex code navigation and MCP-heavy use cases
  • corpus roots: codex, openai, and docs
  • exact-target set: 44 search cases across MCP/tool lookup, approvals/sandbox prompts, protocol/schema lookup, OAuth/MCP client files, connector catalog/docs, and large-repo application files
  • broad probes: mcp, approval, sandbox, oauth, connector, github, and slack
  • tried engine-ranking changes and rerank formulas in parallel; they either regressed quality or failed to reproduce end-to-end, so they are not included in this PR
  • ran a ten-way config sweep: l50_t2, l50_t4, l50_t8, l75_t2, l75_t4, l75_t8, l100_t2, l100_t4, l100_t8, l150_t4

Among the top-quality tier, l100_t8 was the fastest setting:

  • MRR = 0.4445
  • hit@1 = 0.3864
  • hit@5 = 0.5000
  • hit@10 = 0.5682
  • mean latency = 3642.54 ms
  • median latency = 745.91 ms

Other tied top-quality configs were slower:

  • l75_t4 mean 3999.77 ms
  • l100_t4 mean 4014.52 ms
  • l50_t8 mean 4414.51 ms
  • l100_t2 mean 4688.22 ms

Changes kept

  • codex-file-search default limit = 100
  • codex-file-search default threads = 8
  • app-server fuzzy search MATCH_LIMIT = 100
  • app-server fuzzy search MAX_THREADS = 8
  • TUI and TUI app-server FILE_SEARCH_LIMIT = 100

Verification

  • git diff --check
  • cargo test -p codex-file-search (15 passed; 0 failed)
  • GitHub CI started successfully; early code-related checks are green and the remaining matrix is pending

Notes

  • this is performance-only work; no UX/search-presentation changes are included
  • raw eval artifacts were saved locally under ~/repos/docs/codex

@github-actions
Copy link
Contributor


Thank you for your submission, we really appreciate it. Like many open-source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution. You can sign the CLA by just posting a Pull Request Comment same as the below format.


I have read the CLA Document and I hereby sign the CLA


You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot.

@etraut-openai etraut-openai added the oai PRs contributed by OpenAI employees label Mar 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

oai PRs contributed by OpenAI employees

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants