Retry on dynamic index creation when an OpenSearchException is thrown

[engechas]

Description

Adds infinite retry when an OpenSearchException is thrown during dynamic index creation. This aligns with the behavior of static index creation.

Issues Resolved

#3393

Check List

• New functionality includes testing.
• New functionality has a documentation issue. Please link to it in this PR.
  • New functionality has javadoc added
• Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[graytaylor0]

What if it’s not a retryable error? For example, what if the dynamic index name contained capitalization. Would we infinitely retry?

[engechas]

What if it’s not a retryable error? For example, what if the dynamic index name contained capitalization. Would we infinitely retry?

Is that exposed as an OpenSearch exception? If so then yes it would spin and retry. I am not sure there is a simple way to get more granular. We could try to match the text of the exception maybe but I think we'd end up with more crashes for truly retryable issues. Open to ideas.

My thought is that any exception from opensearch is an external dependency failure that requires rectification from the user. I think we should avoid crashes from external dependency failures in general and instead try to provide wording on how it can be fixed inline so the customer can take action.

Let me know if you feel otherwise and I can work on granularity here