Kafka local aws glue registry support

[katelfox]

Is your feature request related to a problem? Please describe.
Unable to run data prepper locally in a docker container pointing to a glue schema registry in another docker container (motoserver)

Describe the solution you'd like
All that is necessary for the support is to allow specifying the AWS registry url and populating AWSSchemaRegistryConstants.AWS_ENDPOINT in the configuration if it is specified

Describe alternatives you've considered (Optional)
At this point I don't know of any way to be able to consume messages from a local kafka with a local aws glue schema registry

Additional context
Add any other context or screenshots about the feature request here.

[dlvenable]

It sounds like this could be a new configuration that we could add and use. @katelfox , Would you be able to add this feature?

[katelfox]

Maybe -- my first attempt wasn't successful. I will hopefully have time to go back again and make it work. Would someone be able to help me/provide guidance? So far the other thing I've found that needs to happen is dummy user name and password to be passed in -- not sure of the best way to do that. For now I was adding it as environment variables when running the docker container.

[dlvenable]

Hello @katelfox ,

I think we would need to set the endpoint by adding a new line into this section.

data-prepper/data-prepper-plugins/kafka-plugins/src/main/java/org/opensearch/dataprepper/plugins/kafka/util/KafkaSecurityConfigurer.java

Lines 398 to 402 in 137e1e7

	configs.put(AWSSchemaRegistryConstants.AVRO_RECORD_TYPE, AvroRecordType.GENERIC_RECORD.getName());
	configs.put(AWSSchemaRegistryConstants.CACHE_TIME_TO_LIVE_MILLIS, "86400000");
	configs.put(AWSSchemaRegistryConstants.CACHE_SIZE, "10");
	configs.put(AWSSchemaRegistryConstants.COMPATIBILITY_SETTING, Compatibility.FULL);
	glueDeserializer = new GlueSchemaRegistryKafkaDeserializer(awsGlueCredentialsProvider, configs);

We also need to make this configurable. So that means either using an existing configuration or using one that exists. I think we can use the existing registry_url configuration. I believe it is not used for Glue so we could make use of it.

data-prepper/data-prepper-plugins/kafka-plugins/src/main/java/org/opensearch/dataprepper/plugins/kafka/configuration/SchemaConfig.java

Lines 29 to 30 in 137e1e7

	@JsonProperty("registry_url")
	private String registryURL;

You should be able to get it using kafkaConsumerConfig.getSchemaConfig().getRegistryURL() or something similar to that.