How to improve message search

[shining]

Hello,

We ran into many problems with message search in Synapse and in fact most of them are already reported here and regrouped with the label A-Message-Search

The recent ticket #20092 also lists some of the biggest issues but I am not a big fan of the substring search solution.
Prefix search would be interesting but infix search, sorted by timestamp, would lead to too many false positives in my opinion.

And also it does not fix all the problems.
As a french speaker I would also like to fix #1523 and properly deal with accents, case sensitivity (eg réunion RÉUNION) and non-ASCII punctuation (eg l’appel)

We also found several problems related to sender search returning partial results.

I spent a few days running Claude on all these problems, designing and testing, and I am now happy with the results and would like to share them if you are interested.

1. Add a reindex_search administrative job to rebuild the search index

We need a way to rewrite the event_search table, several of the following patches need it.

2. Fold non-ASCII punctuation before indexing and searching

Example: l’appel, written with the apostrophe, is currently one token, and neither appel nor l'appel finds it.

We need to split the tokens here, before filling the index and before running the query.

3. Fold case outside ASCII before indexing and searching

Synapse currently uses LC_CTYPE=C so PostgreSQL is only able to lowercase ASCII and nothing
else.

Example: réunion does not find RÉUNION.

Solution: use Python casefold

This should also fix #3116

4. Better support French inclusive writing

Example: salariées does not find salarié·es.

Solution: index the joined form as well

5. Allow configuring the text search configuration used for message search

Example: reunion finds réunion.

This is a long asked feature #1523 .

Solution: add a new option search.text_config which names the configuration used to index, query and highlight

This allows to replace the default english, which was hardcoded everywhere, with a configuration that runs unaccent ahead of the stemmer.

6. Refuse to start on a text search configuration that does not exist

Protection for the previous patch, an invalid text search configuration would prevent sending messages.
Synapse now refuses to start instead.

7. Omit the search count when a filter has dropped matched events

Example: #3077 -> a count of 201 sitting next to an empty results.

The count is computed in SQL and the request's filter is applied afterwards in Python. A search filtered by sender answered with the total count from everyone in those rooms.
count is described as approximate in the specification and is not required, so report it when it is right and leave the key out when it is not.

8. Index events by sender so a from: search stops walking rooms

Searching by sender alone cannot go through /search, which requires a search term, so a client could use /messages with a senders filter over every room.

Synapse pushes the criterion down into the SQL, but no index on events carries sender, so each room costs a backwards scan.
The pathological case is the common one: a room the person has never written in is scanned in full before the page comes back empty.

9. Apply the parts of a search filter that SQL can express

Problem: /search ranks its matches, keeps the best 500, and only then applies the filter in Python.

Solution: Push senders, not_senders, types, not_types and contains_url into the search queries, before the limit.
No schema change, no reindex.

10. Make the components of composite tokens searchable

PostgreSQL's parser is built for documents, not chat: it splits a word written with hyphens or underscores, but stops at a dot, and indexes a URL as its host and its path in one piece.

search.split_composite_tokens (enabled by default) allows to index all components of an URL for instance.

Example: png finds foo.png, and project finds it inside https://gitlab.example/team/project/-/issues/87.

Fixes long standing issue #1426 and #3024

11. Restore prefix matching in message search, behind an option

Synapse searched by prefix until 1.71, where matrix-org/synapse#11635 replaced the hand-built query with websearch_to_tsquery, which has no way to express :*.

search.prefix_matching, off by default, gives a prefix branch to the last term only: the word the user is still typing.
The old behaviour prefixed every term, which is probably not what we want.

Example: serv finds serveurs, qwen finds qwen3.6

This is linked to #20092