Skip to content
This repository was archived by the owner on Jan 13, 2021. It is now read-only.

Support higher throughputΒ #1

@jlhood

Description

@jlhood

Currently, the SQSPoller is invoked by the CW Event schedule and invokes the message processor to process batches of messages synchronously. The current algorithm works and is cost-efficient, but cannot support high throughput. An alternative would be if the SQSPoller and MessageDispatcher classes were put into separate lambda functions and the SQSPoller invoked the MessageDispatcher lambda function asynchronously to process a batch of messages, then continued on. The MessageDispatcher lambda function would handle updating the queue with results of the execution of its batch of messages. This would dramatically increase parallelism of message processing. One caveat is that a high volume queue could lead to too many lambdas being invoked, causing the account to hit its concurrency limit. However, setting a concurrency limit on the MessageProcessor function could be used to control how many concurrent invocations would be supported.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions