fix: improve large dictionary matching performance - #78
Conversation
jan-kubica
changed the title
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates the stella-aho-corasick library to version 1.0.2, optimizing the wholeWords option by utilizing a Standard automaton and selecting leftmost-longest matches after boundary filtering. It also optimizes case-insensitive building for ASCII patterns and removes several unused search methods. The review feedback suggests two key improvements: simplifying the candidate selection loop in src/lib.rs since the candidates are already sorted in the desired order, and avoiding an unnecessary clone of effective_patterns in src/case_folding.rs by moving the vector directly.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| let mut best = candidates[i]; | ||
| i += 1; | ||
| while i < candidates.len() | ||
| && candidates[i].start == start | ||
| { | ||
| let candidate = candidates[i]; | ||
| if candidate.end > best.end | ||
| || (candidate.end == best.end | ||
| && candidate.pattern < best.pattern) | ||
| { | ||
| best = candidate; | ||
| } | ||
| i += 1; | ||
| } |
There was a problem hiding this comment.
Since candidates is already sorted by start (ascending), then end (descending), and then pattern (ascending), the first candidate for any given start is guaranteed to be the leftmost-longest match with the lowest pattern index.\n\nThe subsequent loop that compares candidate.end and candidate.pattern is redundant because no subsequent candidate with the same start can have a larger end, or the same end with a smaller pattern index. We can simplify this loop to just skip the remaining candidates with the same start without performing any comparisons.
let best = candidates[i];\n i += 1;\n while i < candidates.len()\n && candidates[i].start == start\n {\n i += 1;\n }| fallback_patterns: if raw_supports_overlapping { | ||
| None | ||
| } else { | ||
| Some(effective_patterns.clone()) | ||
| }, |
There was a problem hiding this comment.
The effective_patterns vector is passed by value to CaseFoldingAC::build and is not used after the fallback_patterns assignment. We can avoid cloning it by moving effective_patterns directly into Some(effective_patterns).
fallback_patterns: if raw_supports_overlapping {\n None\n } else {\n Some(effective_patterns)\n },
jan-kubica
changed the title
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ac249f6fa6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| let effective_kind = if whole_words { | ||
| RawMatchKind::LeftmostLongest | ||
| RawMatchKind::Standard |
There was a problem hiding this comment.
When wholeWords is true this discards the requested/default matchKind, and the new whole-word path later selects the longest candidate at each start. That changes documented leftmost-first behavior for inputs where multiple boundary-valid patterns start together, e.g. new AhoCorasick(["a", "a-"], { wholeWords: true, matchKind: "leftmost-first" }).findIter("a-") now returns/replaces "a-" instead of pattern 0 "a".
Useful? React with 👍 / 👎.
1 participant
This improves matching performance for large literal dictionaries and whole-word searches. It avoids building a fallback automaton on the first scan when the standard overlapping search can provide the needed candidates. Large case-insensitive dictionaries use a folded-text path instead of forcing the builder's ASCII case-insensitive mode, keeping offsets and match semantics unchanged while reducing setup and search overhead.