Skip search shards with INDEX_REFRESH_BLOCK #129132

benchaplin · 2025-06-09T05:56:43Z

#117543 introduced a ClusterBlock which is applied to new indices in Serverless which do not yet have search shards up. We should skip searches for indices with this block in order to avoid meaningless 503s.

elasticsearchmachine · 2025-06-09T05:57:12Z

Pinging @elastic/es-search-foundations (Team:Search Foundations)

tlrx

Left some comments. The search part should be reviewed by the ES Search team.

tlrx · 2025-06-12T09:08:50Z

server/src/main/java/org/elasticsearch/cluster/metadata/IndexMetadata.java

@@ -283,7 +283,8 @@ public enum APIBlock implements Writeable {
        READ("read", INDEX_READ_BLOCK, Property.ServerlessPublic),
        WRITE("write", INDEX_WRITE_BLOCK, Property.ServerlessPublic),
        METADATA("metadata", INDEX_METADATA_BLOCK, Property.ServerlessPublic),
-        READ_ONLY_ALLOW_DELETE("read_only_allow_delete", INDEX_READ_ONLY_ALLOW_DELETE_BLOCK);
+        READ_ONLY_ALLOW_DELETE("read_only_allow_delete", INDEX_READ_ONLY_ALLOW_DELETE_BLOCK),
+        REFRESH("refresh", INDEX_REFRESH_BLOCK);


I don't think we should allow blocking refreshes on indices after they have been created. Can we revert this please?

tlrx · 2025-06-12T09:13:29Z

server/src/internalClusterTest/java/org/elasticsearch/search/SearchWithIndexBlocksIT.java

+        var addIndexBlockRequest = new AddIndexBlockRequest(IndexMetadata.APIBlock.REFRESH, "test");
+        client().execute(TransportAddIndexBlockAction.TYPE, addIndexBlockRequest).actionGet();


The refresh block should be added automatically to newly created indices as long as they have replicas and the "use refresh block" setting is enabled in the node setting. We should remove the ability to add the refresh block through the Add Index Block API.

tlrx · 2025-06-12T09:16:17Z

server/src/internalClusterTest/java/org/elasticsearch/search/SearchWithIndexBlocksIT.java

+        assertHitCount(prepareSearch().setQuery(QueryBuilders.matchAllQuery()), 0);
+    }
+
+    public void testSearchMultipleIndicesEachWithAnIndexRefreshBlock() {


I think this could be folded into a single test, where one or more indices are randomly created, most of some with replicas but other without replicas, and then allocate zero or more search shards and check the expected results, finally assigning all search shards and check the results again.

cbuescher

I did a first pass on the search related side of things and left a few questions and comments.

cbuescher · 2025-06-18T17:13:32Z

server/src/main/java/org/elasticsearch/action/search/TransportSearchAction.java

+                shardId,
+                shardRouting.getShardRoutings(),
+                finalIndices,
+                finalIndices == SKIPPED_INDICES


This change is in the "getLocalShardsIterator" code branch, however there is another one for PIT ("getLocalShardsIteratorFromPointInTime"), has this been considered as getting a similar "skip" treatment or doesn't this apply because we don't expect any recently started shards there?

Same question for "getRemoteShardsIteratorFromPointInTime"

cbuescher · 2025-06-18T17:19:57Z

server/src/main/java/org/elasticsearch/action/search/TransportSearchAction.java

@@ -148,6 +148,8 @@ public class TransportSearchAction extends HandledTransportAction<SearchRequest,
        Property.NodeScope
    );

+    private static final OriginalIndices SKIPPED_INDICES = new OriginalIndices(Strings.EMPTY_ARRAY, IndicesOptions.strictExpandOpen());


As far as I can understand, this is some unique "Marker" instance for the result map. Can you add comments along that line?

cbuescher · 2025-06-18T17:44:24Z

server/src/main/java/org/elasticsearch/action/search/TransportSearchAction.java

@@ -1304,7 +1310,8 @@ private void executeSearch(

        Map<String, Float> concreteIndexBoosts = resolveIndexBoosts(searchRequest, projectState.cluster());

-        adjustSearchType(searchRequest, shardIterators.size() == 1);
+        boolean oneOrZeroValidShards = shardIterators.size() == 1 || allOrAllButOneSkipped(shardIterators);


Can we simplify this logic or at least rename it?
We originally seem to want to adjust the search type if we only have one "valid" shard iterator. What does that mean if we have shardIterators.size() but that shard is marked as "skipped" here? Would it even make sense to adjust the type then?

I would prefer to rewrite "allOrAllButOneSkipped" into something like "onlyOneValid" or similar that would include the "shardIterators.size() == 1" condition and in addition return "false" as soon as we found >1 non-skipped iterator.
I'm not entirely sure what we should for:

shardIterators.size() == 0 (that would not lead to search type adjustment currently)

shardIterators.size() => 1 but all are skipped

Reading "adjustSearchType" more closely, I think we don't need that adjustment unless we have exactly one non-skipped shard iterator.

cbuescher · 2025-06-18T18:23:08Z

server/src/main/java/org/elasticsearch/action/search/CanMatchPreFilterSearchPhase.java

@@ -186,6 +186,12 @@ private void runCoordinatorRewritePhase() {
        assert assertSearchCoordinationThread();
        final List<SearchShardIterator> matchedShardLevelRequests = new ArrayList<>();
        for (SearchShardIterator searchShardIterator : shardsIts) {
+            if (searchShardIterator.prefiltered() == false && searchShardIterator.skip()) {


As far as I understand this is what actually skips the shards being searched. Why is this done here in the CanMatchPreFilterSearchPhase? My understanding is that we don't always use this phase, e.g. "shouldPreFilterSearchShards" returns false for all searches that are not QUERY_THEN_FETCH (and other cases). Wouldn't we still run into 503s for those cases?

benchaplin added 5 commits June 4, 2025 19:00

Skip indices that have an index refresh block

2aa74e3

Merge branch 'main' into skip_search_shards_with_index_block

12b6b81

Construct the iterator skipped

9c705cd

Fix javadocs

1c75721

Add unit test

1ecc447

benchaplin requested a review from tlrx June 9, 2025 05:56

benchaplin added >non-issue Team:Search Foundations Meta label for the Search Foundations team in Elasticsearch :Search Foundations/Search Catch all for Search Foundations v9.1.0 labels Jun 9, 2025

elasticsearchmachine added the serverless-linked Added by automation, don't add manually label Jun 9, 2025

benchaplin requested a review from a team as a code owner June 9, 2025 14:39

github-actions bot deployed to docs-preview June 9, 2025 14:40 View deployment

benchaplin force-pushed the skip_search_shards_with_index_block branch from 998cdd5 to 1ecc447 Compare June 9, 2025 17:05

benchaplin removed the request for review from a team June 9, 2025 17:06

elasticsearchmachine and others added 9 commits June 9, 2025 17:13

[CI] Auto commit changes from spotless

cdb4bc1

Merge branch 'main' into skip_search_shards_with_index_block

b7ade2d

Merge branch 'main' into skip_search_shards_with_index_block

cd991c2

Merge branch 'main' into skip_search_shards_with_index_block

5f50d5c

Rewrite DFS if processing one or zero unskipped shard iterators

3f86fb8

Make can-match support already skipped shard iterators

0edc27c

Add IT for executing search and PIT against refresh blocked indices

9de6f06

Fix resource leak by using decRef assertion

be37bf6

[CI] Auto commit changes from spotless

17706e2

benchaplin requested review from a team as code owners June 11, 2025 21:03

benchaplin force-pushed the skip_search_shards_with_index_block branch from e233cc7 to 17706e2 Compare June 11, 2025 21:04

Merge branch 'main' into skip_search_shards_with_index_block

8759a07

tlrx reviewed Jun 12, 2025

View reviewed changes

cbuescher reviewed Jun 18, 2025

View reviewed changes

elasticsearchmachine added v9.2.0 and removed v9.1.0 labels Jun 26, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Skip search shards with INDEX_REFRESH_BLOCK #129132

Skip search shards with INDEX_REFRESH_BLOCK #129132

benchaplin commented Jun 9, 2025

Uh oh!

elasticsearchmachine commented Jun 9, 2025

Uh oh!

tlrx left a comment

Uh oh!

tlrx Jun 12, 2025

Uh oh!

tlrx Jun 12, 2025

Uh oh!

tlrx Jun 12, 2025

Uh oh!

cbuescher left a comment

Uh oh!

cbuescher Jun 18, 2025

Uh oh!

cbuescher Jun 18, 2025

Uh oh!

cbuescher Jun 18, 2025

Uh oh!

cbuescher Jun 18, 2025

Uh oh!

cbuescher Jun 18, 2025

Uh oh!

cbuescher Jun 18, 2025

Uh oh!

Uh oh!

		var addIndexBlockRequest = new AddIndexBlockRequest(IndexMetadata.APIBlock.REFRESH, "test");
		client().execute(TransportAddIndexBlockAction.TYPE, addIndexBlockRequest).actionGet();

Skip search shards with INDEX_REFRESH_BLOCK #129132

Are you sure you want to change the base?

Skip search shards with INDEX_REFRESH_BLOCK #129132

Conversation

benchaplin commented Jun 9, 2025

Uh oh!

elasticsearchmachine commented Jun 9, 2025

Uh oh!

tlrx left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cbuescher left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!