Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Support custom index type for private opensearch serverless collection as sink #4188

Closed
oeyh opened this issue Feb 27, 2024 · 0 comments · Fixed by #4250
Closed

[BUG] Support custom index type for private opensearch serverless collection as sink #4188

oeyh opened this issue Feb 27, 2024 · 0 comments · Fixed by #4250
Assignees
Labels
bug Something isn't working

Comments

@oeyh
Copy link
Collaborator

oeyh commented Feb 27, 2024

Is your feature request related to a problem? Please describe.
As a Data Prepper user, I'd like to use custom index type in opensearch sink to set up index template when using vpc collection as sink.

Current issue with this setup is when opensearch sink initializes, it tries to set up index and index template before creating or updating serverless network policy, which results in 401 errors.

Trackback looks like this:

2024-03-07T21:04:55,273 [log-pipeline-sink-worker-2-thread-1] INFO  org.opensearch.dataprepper.plugins.sink.opensearch.index.AbstractIndexManager - Index template vpc-aoss-test-000003-index-template does not exist and should be created
2024-03-07T21:04:55,285 [log-pipeline-sink-worker-2-thread-1] WARN  org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Failed to initialize OpenSearch sink with a retryable exception.
org.opensearch.client.opensearch._types.OpenSearchException: Request failed: [http_exception] server returned 401
	at org.opensearch.client.transport.aws.AwsSdk2Transport.parseResponse(AwsSdk2Transport.java:494) ~[opensearch-java-2.8.1.jar:?]
	at org.opensearch.client.transport.aws.AwsSdk2Transport.executeSync(AwsSdk2Transport.java:392) ~[opensearch-java-2.8.1.jar:?]
	at org.opensearch.client.transport.aws.AwsSdk2Transport.performRequest(AwsSdk2Transport.java:192) ~[opensearch-java-2.8.1.jar:?]
	at org.opensearch.dataprepper.plugins.sink.opensearch.index.ComposableTemplateAPIWrapper.putTemplate(ComposableTemplateAPIWrapper.java:34) ~[opensearch-2.7.0-SNAPSHOT.jar:?]
	at org.opensearch.dataprepper.plugins.sink.opensearch.index.ComposableIndexTemplateStrategy.createTemplate(ComposableIndexTemplateStrategy.java:47) ~[opensearch-2.7.0-SNAPSHOT.jar:?]
	at org.opensearch.dataprepper.plugins.sink.opensearch.index.AbstractIndexManager.checkAndCreateIndexTemplate(AbstractIndexManager.java:258) ~[opensearch-2.7.0-SNAPSHOT.jar:?]
	at org.opensearch.dataprepper.plugins.sink.opensearch.index.AbstractIndexManager.checkAndCreateIndexTemplate(AbstractIndexManager.java:234) ~[opensearch-2.7.0-SNAPSHOT.jar:?]
	at org.opensearch.dataprepper.plugins.sink.opensearch.index.AbstractIndexManager.setupIndex(AbstractIndexManager.java:224) ~[opensearch-2.7.0-SNAPSHOT.jar:?]
	at org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink.doInitializeInternal(OpenSearchSink.java:231) ~[opensearch-2.7.0-SNAPSHOT.jar:?]
	at org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink.doInitialize(OpenSearchSink.java:193) ~[opensearch-2.7.0-SNAPSHOT.jar:?]
	at org.opensearch.dataprepper.model.sink.AbstractSink.initialize(AbstractSink.java:52) ~[data-prepper-api-2.7.0-SNAPSHOT.jar:?]
	at org.opensearch.dataprepper.pipeline.Pipeline.isReady(Pipeline.java:200) ~[data-prepper-core-2.7.0-SNAPSHOT.jar:?]
	at org.opensearch.dataprepper.pipeline.Pipeline.lambda$execute$2(Pipeline.java:252) ~[data-prepper-core-2.7.0-SNAPSHOT.jar:?]

Describe the solution you'd like
Support custom index type for private opensearch serverless collection as sink. This can probably be resolved by just moving network policy update code ahead of setting up index.

Describe alternatives you've considered (Optional)
N/A

Additional context
N/A

@oeyh oeyh added the untriaged label Feb 27, 2024
@kkondaka kkondaka changed the title Support custom index type for private opensearch serverless collection as sink [BUG] Support custom index type for private opensearch serverless collection as sink Feb 27, 2024
@kkondaka kkondaka added bug Something isn't working and removed untriaged labels Feb 27, 2024
@oeyh oeyh self-assigned this Mar 7, 2024
@oeyh oeyh closed this as completed in #4250 Mar 8, 2024
@github-project-automation github-project-automation bot moved this from Unplanned to Done in Data Prepper Tracking Board Mar 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Development

Successfully merging a pull request may close this issue.

2 participants