AWS SQS Message Queue Logic - Queue Name / ARN Construction

[techdragon]

While setting up a demo project using the SQS Message Queue functionality, I've come across some behaviour I'm finding confusing with how ServiceStack is trying to connect to AWS SQS.

Having read through the ServiceStack code I had expected it to try and use existing SQS queue names based on the normal prefix + Class/Type name + suffix logic covered in the documentation... but run through a SQS specific transform to make sure that the SQS Queue name is valid for AWS's requirements. ( The code here https://github.com/ServiceStack/ServiceStack/blob/main/ServiceStack.Aws/src/ServiceStack.Aws/Sqs/SqsExtensions.cs#L26-L33 ) However the logs coming from ServiceStack make it look like the rename to make a valid AWS SQS name is not taking place.

dbug: ServiceStack.Aws.Sqs.SqsQueueManager[0]
      SQS Queue named [mq:Hello.inq] does not exist

This just due to some custom class stringification helper I haven't spotted yet and the proper queue name is constructed per the linked extension class method?

Also on the topic of the Queue Name / ARN construction, I haven't been able to find where I can define the destination AWS Account ID to use when building the SQS ARN. This is necessary for some scenarios, like messaging between isolated AWS accounts that cant share direct network access for HTTP API use. It's easy enough to ensure the caller code is run with the correct AWS credentials via environment variables, but without a way to specify the SQS queue is in a different AWS account, it will never go to the right place for this scenario.

[mythz]

All MQs providers using the conventional MQ names documented in the Message Workflow.

The MQ names can be customized with the QueueNames.cs class to adopt a different prefix or naming strategy, but if you do change the prefix or ResolveQueueNameFn, it also needs to be configured in client Apps so they use the same custom Queue Names.

We likely don't support the other ARN construction requirements you're referring to.

Note that ServiceStack MQ is a restricted solution designed to enable MQ endpoints for invoking your Services, if you need more than this or AWS-specific features you'll want to use AWS SQS clients + classes directly.

[techdragon]

I think I've got the issue worked out, but in order to pass the account id into the API calls, I need to subclass/override the ServiceStack call to the AWS SDK to add in the other parameter to the GetQueueUrlRequest object constructor... But all the AWS SDK calls are happening via async wrappers in https://github.com/ServiceStack/ServiceStack/blob/main/ServiceStack.Aws/src/ServiceStack.Aws/Support/NetCoreAwsExtensions.cs ... and I cant work out how this is being combined with the AmazonSQSClient from the AWS .Net SDK. Which I'm pretty sure is being setup here https://github.com/ServiceStack/ServiceStack/blob/main/ServiceStack.Aws/src/ServiceStack.Aws/Sqs/SqsConnectionFactory.cs ... But I can't just work out how the combining is happening in order to perform the appropriate override/injection/subclass/etc.

[mythz]

You'd need to create a copy of SqsMqMessageFactory.cs then change it to return custom version of SqsMqClient and SqsMqMessageProducer:

https://github.com/ServiceStack/ServiceStack/blob/6298a3b84c41e9a5fe4dcba1ed7ad48cc79b69e2/ServiceStack.Aws/src/ServiceStack.Aws/Sqs/SqsMqMessageFactory.cs#L64-L80

public IMessageQueueClient CreateMessageQueueClient()
{
    return new SqsMqClient(sqsMqBufferFactory, sqsQueueManager) {
        SendMessageRequestFilter = SendMessageRequestFilter,
        ReceiveMessageRequestFilter = ReceiveMessageRequestFilter,
        ReceiveMessageResponseFilter = ReceiveMessageResponseFilter,
        DeleteMessageRequestFilter = DeleteMessageRequestFilter,
        ChangeMessageVisibilityRequestFilter = ChangeMessageVisibilityRequestFilter,
    };
}

public IMessageProducer CreateMessageProducer()
{
    return new SqsMqMessageProducer(sqsMqBufferFactory, sqsQueueManager) {
        SendMessageRequestFilter = SendMessageRequestFilter,
    };
}

Depending on the customizations required it may be easier to utilize a local modified class of all ServiceStack.Aws/Sqs classes so you're not restricted trying to modify existing implementations.

[techdragon]

Thanks, I'm just looking through how many places will need an override to get this working now that I see what needs doing. Given the nature of ongoing updates, I'd be concerned about potential future incompatible changes & all the normal "forked library risks" if we were to go ahead and use our own local modified copy of all of ServiceStack.Aws/Sqs, given the nature of the changes (keeping the current behaviour is simple) is there any way to contribute back a "fix" for this use case? Since the source code is also distributed under commercial terms I figure there's probably no way to contribute it back or if there is its with a CLA or something like that.

[mythz]

If you approve the Contributor License Agreement you can send a PR that I can review, please try to maintain backwards compatibility wherever possible.

[techdragon]

@mythz I've signed the Contributor License Agreement and I've created ServiceStack/ServiceStack#1307

I haven't included tests since I realised that as an optional parameter, any tests I write would need to be driven by the CI with an appropriate AWS Account Id and I have no simple way to set that up. I've got some tests I can add if you need them to accept the pull request, but I did also make sure that the existing tests continue to pass with the new code added.

[mythz]

Cheers, merged as PR looks fairly straight forward. If needed this is available in the latest v6.5.1+ on MyGet.