[BUG] S3 snapshots get uploaded outside of the configured `base_path`

[atrendler-tc]

Describe the bug

After migrating our Opensearch container from 2.6.0 to 3.2.0 we discovered that Opensearch creates data backups outside of the configured base_path. It seems that only metadata actually gets uploaded properly in the base_path.

Related component

Storage:Snapshots

To Reproduce

1. Create an S3 bucket
2. Configure S3 repository:

curl -sku ${OPENSEARCH_USERNAME}:${OPENSEARCH_PASSWORD} -X PUT "https://localhost:9200/_snapshot/opensearch_snapshots?pretty" -H 'Content-Type: application/json' -d'
{
  "type": "s3",
  "settings": {
    "bucket": "<bucket>",
    "region": "<region>",
    "base_path": "opensearch_snapshots"
  }
}
'

3. Verify that the base_path only contains metadata.
4. Verify that the root of the S3 bucket gets filled with separate directories that contain the actual data of the indices.

Expected behavior

Both data and metadata should be uploaded to the base_path.

Additional Details

No response

[RamHaridas]

The same issue is being observed with S3 compatible storage (MinIO) as well.

[atrendler-tc]

Hopefully we can get some eyes on this!

[RamHaridas]

@atrendler-tc try setting parameter shard_path_type to fixed during repository creation.

curl -sku ${OPENSEARCH_USERNAME}:${OPENSEARCH_PASSWORD} -X PUT "https://localhost:9200/_snapshot/opensearch_snapshots?pretty" -H 'Content-Type: application/json' -d'
{
  "type": "s3",
  "settings": {
    "bucket": "<bucket>",
    "region": "<region>",
    "base_path": "opensearch_snapshots",
   "shard_path_type" : "FIXED"
  }
}
'