[FEA]: Enhanced tracing for individual pipeline stages - raw output

[jdye64]

Is this a new feature, an improvement, or a change to existing functionality?

New Feature

How would you describe the priority of this feature request

Currently preventing usage

Please provide a clear description of problem this feature solves

Several people have requested being able to view the output from each individual stage in a pipeline. Since we now have our opentelemetry global and unique trace-id I think it would make sense to add a configuration that would allow for each stage in the morpheus pipeline to output/write it raw data to a folder that has the same name as the trace-id and then the file named the same as the stage. This would allow for users to view the output for each stage and individually analyze it as well.

While this could possibly be done with zipkin and opentelemetry itself given we mostly deal with images I think the output would be far to large to capture there.

Describe the feature, and optionally a solution or implementation and any alternatives

Capture output from each individual stage in a complete Morpheus pipeline when a yet to be added configuration is enabled.

Additional context

No response

[drobison00]

Would like to brainstorm a bit on this. I'd like to try and formulate this problem in a way that lets us add or extend the existing function decorator patterns, and does the final write as a post-response stage.