Skip to content

[DAT-63] fix: Inconsistent sort due to per-field limit #7

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Oct 29, 2024
Merged

[DAT-63] fix: Inconsistent sort due to per-field limit #7

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 17 additions & 5 deletions src/search/SearchEngine.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -264,11 +264,12 @@ class SearchEngine {
}

const allResults = this.searchOnIndexes(query)
const results = this.deduplicateAndFlatten(allResults)
const sortedResults = this.sortSearchResults(results)
const dedupResults = this.deduplicateAndFlatten(allResults)
const sortedResults = this.sortSearchResults(dedupResults)
const results = this.limitSearchResults(sortedResults)

const normResults: SearchResult[] = []
for (const res of sortedResults) {
for (const res of results) {
const normalizedRes = await normalizeSearchResult(this.client, res, query)
normResults.push(normalizedRes)
}
Expand All @@ -286,11 +287,18 @@ class SearchEngine {
}
// TODO: do not use flexsearch store and rely on pouch storage?
// It's better for memory, but might slow down search queries
//
// XXX - The limit is specified twice because of a flexsearch inconstency
// that does not enforce the limit if only given in second argument, and
// does not return the correct type is only given in third options
const indexResults = index.index.search(query, LIMIT_DOCTYPE_SEARCH, {
limit: LIMIT_DOCTYPE_SEARCH,
//
// XXX - The given limit here is arbitrary because flexsearch enforce it on matching
// field, which can cause issue related to the sort: if we search on name+path for files,
// and limit on 100, the 101th result on name will be skipped, but might appear on path,
// which will make it appear in the search results, but in incorrect order.
const FLEXSEARCH_LIMIT = 10000
const indexResults = index.index.search(query, FLEXSEARCH_LIMIT, {
limit: FLEXSEARCH_LIMIT,
enrich: true
})
const newResults = indexResults.map(res => ({
Expand Down Expand Up @@ -373,6 +381,10 @@ class SearchEngine {
// Then name
return this.compareStrings(aRes.doc.name, bRes.doc.name)
}

limitSearchResults(searchResults: RawSearchResult[]): RawSearchResult[] {
return searchResults.slice(0, LIMIT_DOCTYPE_SEARCH)
}
}

export default SearchEngine