Skip to content

Support new Elastic LLM indexer #1146

Open
@bmorelli25

Description

@bmorelli25

Summary

We need to build support for Elastic's LLM indexer. After an initial indexing, the LLM indexer indexes incrementally. To support this incremental indexing, we need to support a "content last modified" date. This date will tell the indexer when the content (and only the content) of a file has changed so that it can reindex accordingly.

Changes that don't matter

Examples of changes that should not update the "content last modified" date:

  • docs-builder configuration changes, like updating CSS, or changing the page layout requires a new docs/assembler build and changes the underlying HTML file, but does not require a new indexing.
  • Image additions/deletions/changes impact docs-builder and changes the underlying HTML file but does not require a new indexing.
  • Header and footer changes impact docs-builder and change the underlying HTML file but do not require new indexing.
  • Navigation changes/reordering impact docs-builder and change the underlying HTML file but do not require new indexing.

Implementation

  1. As a part of each assembler build, determine which files have content changes.
  2. Figure out how the changed files affect other content. For example:
    • If a partial file is changed and that file is included in multiple other files, we need to know every file that includes the partial.
    • If a variable changes, that change will impact multiple pages. We need to know every file that includes that variable.
  3. Resolve links to a path
  4. Update metadata for the page (only metadata needed currently is the URL path)
  5. Update the content last modified date for changed files. NOTE: Partials don't need to be indexed and should be marked as no-index for the indexer as the content is indexed as a part of the page that they are built to.

Contact

@siamakp-elastic

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions