Open
Description
Summary
We need to build support for Elastic's LLM indexer. After an initial indexing, the LLM indexer indexes incrementally. To support this incremental indexing, we need to support a "content last modified" date. This date will tell the indexer when the content (and only the content) of a file has changed so that it can reindex accordingly.
Changes that don't matter
Examples of changes that should not update the "content last modified" date:
docs-builder
configuration changes, like updating CSS, or changing the page layout requires a new docs/assembler build and changes the underlying HTML file, but does not require a new indexing.- Image additions/deletions/changes impact docs-builder and changes the underlying HTML file but does not require a new indexing.
- Header and footer changes impact docs-builder and change the underlying HTML file but do not require new indexing.
- Navigation changes/reordering impact docs-builder and change the underlying HTML file but do not require new indexing.
Implementation
- As a part of each assembler build, determine which files have content changes.
- Figure out how the changed files affect other content. For example:
- If a partial file is changed and that file is included in multiple other files, we need to know every file that includes the partial.
- If a variable changes, that change will impact multiple pages. We need to know every file that includes that variable.
- Resolve links to a path
- Update metadata for the page (only metadata needed currently is the URL path)
- Update the content last modified date for changed files. NOTE: Partials don't need to be indexed and should be marked as no-index for the indexer as the content is indexed as a part of the page that they are built to.