Is there a way to index terms with special characters, like html tags.

[argl]

When evaluating pagefind for indexing/searching MDN content (nice demo!), I noticed there are a lot of pages on MDN with the literal HTML tag as the title, for example the <table> documentation page.

Ideally, I would search for <table> and get to the main HTML table documentation page, but unfortunately, this is not the case. I understand that this is a wacky request very specific to the MDN content, but I am wondering if this can be done, and how? I do have full control over the HTML that is passed to the indexer.

Thanks for this great library!

[bglw]

👋 @argl

There isn't currently a way to do this for Pagefind, as it always normalizes terms both when searching and indexing. So <table> is normalized to tabl (stemmed) at both index time and search time.

Supporting special characters is something that could likely be added as an option, though. One of the later additions to Pagefind's internals is the ability to index multiple words at a single "location" — e.g. html_tag is indexed as ['html_tag', 'html', 'tag'], so I can imagine it being possible for <table> to be indexed as ['<table>', 'tabl'].

It would need to be an opt-in feature so as to not cause too much chaos and index bloat, and it would need more thought into how the frontend searches. Currently if you type <table> into search, it only runs a search for tabl, but that will also be solvable.

Something to ponder at least!

[argl]

Thanks! This is more or less in line with what I thought. I did hack literal html tag searches into pagefind to check the effort needed. As you said, this touches the indexer and the search API side. There are some magic regular expressions along the way which I would find difficult to quickly marry to a configurable system without some major code additions.

Thanks again for this amazing piece of work, even if it does not fit our requirements fully today, I hope I will find a use for it in the future!

[bglw]

I could imagine a smaller lift could be to configure certain characters as exempt from stripping. E.g. in a config file

include_characters: "<>!-"

The regexes would stay mostly as-is, but have configured characters compiled in and retained.

The main hard part would be ensuring <table> could be searched for with "<table>" and with "table", but I think the current term normalization would take care of that anyway. So this might be more possible than it seems on the surface :) I'll see about taking a look soon

[bglw]

👋 @argl this is now on the current Pagefind alpha release (v1.4.0-alpha.1) if you'd like to take it for a spin :)

Docs here: https://unreleased.pagefind.app/docs/indexing/#indexing-special-characters

[argl]

This is great news, I'll try to give it another go this week. Thanks!