add pipeline support to validation tests

[c1ly]

• Moved the validations tests outside of otap-dataflow to directly under the rust directory
• Added pipeline support to run pipelines and validate the output
• Allows for multiple pipelines to be defined and added to the validation test, currently only supports non transformative processors

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.80%. Comparing base (05ea7a9) to head (7e2517e).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1751      +/-   ##
==========================================
- Coverage   84.33%   81.80%   -2.53%     
==========================================
  Files         495      181     -314     
  Lines      144232    51480   -92752     
==========================================
- Hits       121640    42114   -79526     
+ Misses      22058     8832   -13226     
  Partials      534      534              

Components	Coverage Δ
otap-dataflow	∅ <ø> (∅)
query_abstraction	80.61% <ø> (ø)
query_engine	90.17% <ø> (-0.35%)	⬇️
syslog_cef_receivers	∅ <ø> (∅)
otel-arrow-go	53.50% <ø> (ø)
quiver	∅ <ø> (∅)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[albertlockett]

going to try closing and reopening to trigger CI

edit: this didn't work

[albertlockett]

@drewrelmas any idea why this PR is not running the rust CI job?

I assume we have some checks in place that this needs to run for any changes under the rust/ directory.

It looks like this should run for any pull request targeting main, so I'm confused about why this isn't running.

otel-arrow/.github/workflows/rust-ci.yml

Lines 2 to 12 in 31f3a17

	permissions:
	contents: read
	on:
	push:
	branches:
	- main
	pull_request:
	branches:
	- main
	merge_group:

[drewrelmas]

Does this need to be a matrix strategy if only processing one folder? Also, there are duplicate fail-fast definitions here - that could be why the Rust-CI didn't run.

[drewrelmas]

@albertlockett See the comment here about fail-fast, I think this is the issue with the CI.

[albertlockett]

great catch, thanks @drewrelmas!