Snapshots support multi-project #130000

ywangd · 2025-06-25T11:43:50Z

This PR makes snapshot service code and APIs multi-project compatible.

Resolves: ES-10225
Resolves: ES-10226

…rvice-multi-project

elasticsearchmachine · 2025-06-26T04:41:18Z

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

ywangd · 2025-06-26T08:09:52Z

Most of the changes are for SnapshotsService and BlobStoreRepository. The rest is mostly cascading. Initially I wanted to have separate PRs for clone, cleanup and maybe get-status APIs. But it does not seem to make much sense since there is a large code overlap and reuse among all APIs, e.g. it is actually better and easier to reason with to make the entirety of SnapshotsService project-aware. Restore is still left out. It is handled by different classes and will be addressed seperately.

ywangd · 2025-06-26T08:10:16Z

server/src/main/java/org/elasticsearch/repositories/ProjectRepo.java

+ * @param projectId The project that the repository belongs to
+ * @param name      Name of the repository
+ */
+public record ProjectRepo(ProjectId projectId, String name) implements Writeable {


This is an existing class extracted from RepositoryOperation.

ywangd · 2025-06-27T02:01:20Z

server/src/main/java/org/elasticsearch/repositories/blobstore/BlobStoreRepository.java

+                    final var projectMetadata = clusterMetadata.getProject(getProjectId());
+                    executor.execute(ActionRunnable.run(allMetaListeners.acquire(), () -> {
+                        if (finalizeSnapshotContext.serializeProjectMetadata()) {
+                            PROJECT_METADATA_FORMAT.write(projectMetadata, blobContainer(), snapshotId.getUUID(), compress);
+                        } else {
+                            GLOBAL_METADATA_FORMAT.write(clusterMetadata, blobContainer(), snapshotId.getUUID(), compress);
+                        }
+                    }));


This is where we conditionally write ProjectMetadata for multi-project snapshots. The Metadata in this case is a thin wrapper around ProjectMetadata to reuse existing finalization related classes.

ywangd · 2025-06-27T02:44:16Z

server/src/main/java/org/elasticsearch/snapshots/SnapshotsService.java

+    private void startExecutableClones(SnapshotsInProgress snapshotsInProgress) {
+        for (List<SnapshotsInProgress.Entry> entries : snapshotsInProgress.entriesByRepo()) {
+            startExecutableClones(entries);
+        }
+    }
+
+    /**
+     * Maybe kick off new shard clone operations for all repositories of the specified project
+     */
+    private void startExecutableClones(SnapshotsInProgress snapshotsInProgress, ProjectId projectId) {
+        for (List<SnapshotsInProgress.Entry> entries : snapshotsInProgress.entriesByRepo(projectId)) {
+            startExecutableClones(entries);
        }
    }

+    /**
+     * Maybe kick off new shard clone operations for the single specified project repository
+     */
+    private void startExecutableClones(SnapshotsInProgress snapshotsInProgress, ProjectRepo projectRepo) {
+        startExecutableClones(snapshotsInProgress.forRepo(Objects.requireNonNull(projectRepo)));
+    }
+


Snapshotting is state machine that triggers next operation when the current operation finishes. In most cases, the triggering is confined in the same repository. This is the simplest case and gets migrated as is. The other case is triggering across all repositories. In a MP setup, this could mean either across all repositories of all projects or across all repositories of a single project. This is the reason for the 3 variants of the same named method here. The principles that I have applied are:

If the scope was a single repository, keep it as is.

If the scope was all repositories and reacting to cluster state changes, i.e. applyClusterState, it applies to all repositories across all projects.

If the scope was all repository and happening after completing a particular snapshot operation, e.g. deleting a snapshot entry, it applies to all repositories of a single project that the operation is associated with.

ywangd · 2025-06-27T02:45:58Z

server/src/main/java/org/elasticsearch/snapshots/SnapshotsService.java

+    private static Tuple<ClusterState, List<SnapshotDeletionsInProgress.Entry>> readyDeletions(
+        ClusterState currentState,
+        @Nullable ProjectId projectId
+    ) {


This is another example of different scopes for triggering next operation. In this case, it does not have the single repository scope, but either cluster wide or single project (when projectId == null).

ywangd · 2025-06-27T03:00:24Z

Depending on how we handle project soft-deletion and/or clean-up, snapshots may see a project getting concurrently deleted and thereore fail. This PR does not attempt to handle it more gracefully since it would become noise and get burried in the large amount of namespacing changes. I can raise a separate ticket to track this work.

…rvice-multi-project

pxsalehi · 2025-06-27T11:19:37Z

Depending on how we handle project soft-deletion and/or clean-up, snapshots may see a project getting concurrently deleted and thereore fail. This PR does not attempt to handle it more gracefully since it would become noise and get burried in the large amount of namespacing changes. I can raise a separate ticket to track this work.

yeah, we'd need a new ticket for this under the soft-deletion epic. We briefly mentioned in the design doc that once the project is marked for deletion, we should 1) prevent any new snapshots being scheduled/requested. This partially goes back to making those internal actions aware of checking for the deletion project block. 2) any ongoing snapshotting should be cancelled for that project (I guess not that simple but somehow at least fail graciously and not blow up). For 1, we have ES-12121. But I don't think 2 has a ticket yet.

pxsalehi

LGTM. To my eyes these all look like straight-forward changes to trickle down project ID everywhere necessary. I don't have a strong opinions about the details. Although considering snapshotting code is convoluted and in a delicate state, I'm gonna defer the final approval to David (or anyone else with more snapshotting experience).

pxsalehi · 2025-06-27T11:29:43Z

server/src/main/java/org/elasticsearch/cluster/metadata/ProjectMetadata.java

+            if (token == null) {
+                token = parser.nextToken();
+            }


what's this all about?

pxsalehi · 2025-06-27T11:37:57Z

server/src/main/java/org/elasticsearch/repositories/blobstore/BlobStoreRepository.java

@@ -2014,7 +2028,7 @@ private void getOneSnapshotInfo(BlockingQueue<SnapshotId> queue, GetSnapshotInfo
            Exception failure = null;
            SnapshotInfo snapshotInfo = null;
            try {
-                snapshotInfo = SNAPSHOT_FORMAT.read(metadata.name(), blobContainer(), snapshotId.getUUID(), namedXContentRegistry);
+                snapshotInfo = SNAPSHOT_FORMAT.read(getProjectRepo(), blobContainer(), snapshotId.getUUID(), namedXContentRegistry);


is getProjectRepo() now for non-MP always uding DEFAULT as project name here?

ywangd added 3 commits June 25, 2025 13:52

Make SnapshotsService and related APIs support multi-project

d4c3e22

Fix more issues and write the correct metadata

f3c531f

unmute

7655d7c

elasticsearchmachine added the v9.2.0 label Jun 25, 2025

tweak

f669e03

elasticsearchmachine added the serverless-linked Added by automation, don't add manually label Jun 25, 2025

ywangd added 9 commits June 26, 2025 00:14

fix license header

a7454d6

Merge remote-tracking branch 'origin/main' into ES-10225-snapshots-se…

c7cb973

…rvice-multi-project

fix test and improve logging

d9248b4

fix tests

132401f

Merge remote-tracking branch 'origin/main' into ES-10225-snapshots-se…

e4f7a8d

…rvice-multi-project

fix more tests

0981ad4

Merge remote-tracking branch 'origin/main' into ES-10225-snapshots-se…

1f05ee1

…rvice-multi-project

tweak

6109e9d

tighten assertions

b846483

ywangd added >non-issue :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs labels Jun 26, 2025

ywangd requested review from pxsalehi and DaveCTurner June 26, 2025 04:39

ywangd marked this pull request as ready for review June 26, 2025 04:40

ywangd requested a review from a team as a code owner June 26, 2025 04:40

elasticsearchmachine added the Team:Distributed Coordination Meta label for Distributed Coordination team label Jun 26, 2025

ywangd commented Jun 26, 2025

View reviewed changes

ywangd commented Jun 27, 2025

View reviewed changes

tweak

89b4c00

Merge remote-tracking branch 'origin/main' into ES-10225-snapshots-se…

a04e77e

…rvice-multi-project

pxsalehi reviewed Jun 27, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Snapshots support multi-project #130000

Snapshots support multi-project #130000

ywangd commented Jun 25, 2025

Uh oh!

elasticsearchmachine commented Jun 26, 2025

Uh oh!

ywangd commented Jun 26, 2025

Uh oh!

ywangd Jun 26, 2025

Uh oh!

ywangd Jun 27, 2025

Uh oh!

ywangd Jun 27, 2025

Uh oh!

ywangd Jun 27, 2025

Uh oh!

ywangd commented Jun 27, 2025

Uh oh!

pxsalehi commented Jun 27, 2025

Uh oh!

pxsalehi left a comment

Uh oh!

pxsalehi Jun 27, 2025

Uh oh!

pxsalehi Jun 27, 2025

Uh oh!

Uh oh!

Snapshots support multi-project #130000

Are you sure you want to change the base?

Snapshots support multi-project #130000

Conversation

ywangd commented Jun 25, 2025

Uh oh!

elasticsearchmachine commented Jun 26, 2025

Uh oh!

ywangd commented Jun 26, 2025

Uh oh!

ywangd Jun 26, 2025

Choose a reason for hiding this comment

Uh oh!

ywangd Jun 27, 2025

Choose a reason for hiding this comment

Uh oh!

ywangd Jun 27, 2025

Choose a reason for hiding this comment

Uh oh!

ywangd Jun 27, 2025

Choose a reason for hiding this comment

Uh oh!

ywangd commented Jun 27, 2025

Uh oh!

pxsalehi commented Jun 27, 2025

Uh oh!

pxsalehi left a comment

Choose a reason for hiding this comment

Uh oh!

pxsalehi Jun 27, 2025

Choose a reason for hiding this comment

Uh oh!

pxsalehi Jun 27, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!