[6.x] Improve search indexing performance

[duncanmcclean]

This pull request aims to improve the performance of updating search indexes in Statamic.

TL;DR We're now passing around references, chunking documents and inserting them via the queue.

The Issue

On larger sites, with lots of entries, clicking the "Update" button in the search utility or running the search:update command would cause page timeouts and significant memory issues.

Even though we're returning LazyCollection instances from our searchables, some collection methods, like ->each() evaluate the underlying array, even when the closure is empty. We believe this was happening due the collection containing lots of Entry objects.

Changes

• Searchables now return collections of references (eg. entry::the-entries-id), instead of collections of Entry objects.
• You can now define a query_scope on search indexes, which is much more performant than the filter option.
• Documents are now inserted via a queued job, in chunks of 100.
  • In the job, we group the documents by their prefix, query to get the objects, and send them to the index's insertDocuments method.
  • The queue, queue_connection and chunk_size can be configured in the search.php config file.

As a result of these changes, I was able to re-index a site with 100k entries in ~25 seconds (using the Rad Pack Typesense driver).

Breaking Changes

Custom Searchables

Searchables should now return collections of references (eg. entry::the-entry-id) instead of objects.

- return Thing::query()->lazy();
+ return Thing::query()->lazy()->pluck('reference');

If your searchable allows for it, you may want to consider adding support for query scopes, which we now recommend over filtering.

$query = Thing::query();

$this->applyQueryScope($query);

return $query->pluck('reference');

If you support filtering, you may want to split the "with filter" & "without filter" cases into separate return statements.

The ->filter() method evaluates every item in the collection, which is an unnecessary performance hit when no filter is configured:

$query = Thing::query();

if ($this->hasFilter()) {
	return $query
		->lazy(config('statamic.search.chunk_size'))  
		->filter($this->filter())  
		->values()  
		->map->reference();
}

return $query->pluck('reference');

Example PR: statamic-rad-pack/runway#748

Custom Search Drivers

The insertDocument method is now public:

- protected function insertDocuments(Documents $documents)
+ public function insertDocuments(Documents $documents)

If you were previously overriding the insertMultiple method to chunk documents, you don't need to do that anymore (chunking is now handled by the base method).

If you need to manipulate the fields array before it gets sent to your index, you may define a fields method:

public function fields(Searchable $searchable)
{
    return array_merge(
        $this->searchables()->fields($searchable),
        [
            '_some_special_field_' => $searchable->id(),
        ]
    );
}

Example PRs:

• Statamic 6: Search indexing improvements statamic-rad-pack/typesense#24
• Statamic 6: Search indexing improvements statamic-rad-pack/meilisearch#48

---

Fixes #13122