[FEA]: Add support for multiple source/sink pairs

[drobison00]

Is this a new feature, an improvement, or a change to existing functionality?

Improvement

How would you describe the priority of this feature request

Significant improvement

Task Description

This feature will allow us to support multiple ingress message brokers, or alternative data sources, which will need to be paired with an appropriate sink type to handle the message results.

Acceptance Criteria

• Configuration

  • Add support to pipeline.py for specifying some number of input source / output sink pairs
  • Allow specification of any required configuration parameters

• Implementation

  • Create pydantic schema validators for all input source/sink configurations
  • Update JobSpec schemas so that they support a new tag indicating which sink they should be routed to
  • Add appropriate lookup mechanisms for mapping source/sink specifications to stage/module constructors and appropriate error handling
  • Add functional tests the verify routing behavior

Describe the feature, and optionally a solution or implementation and any alternatives

This issue currently depends on the availability of router nodes in Morpheus.

graph LR;
    %% Input Sources
    A[Redis] --> B(Central Pipeline)
    C[Kafka] --> B(Central Pipeline)
    D[RabbitMQ] --> B(Central Pipeline)
    H[File System] --> B(Central Pipeline)

    %% Output Sinks
    B(Central Pipeline) --> E[Redis Sink]
    B(Central Pipeline) --> F[Kafka Sink]
    B(Central Pipeline) --> G[RabbitMQ Sink]
    B(Central Pipeline) --> I[File System Sink]

Loading