Skip to content

Conversation

@lquerel
Copy link
Contributor

@lquerel lquerel commented Jan 19, 2026

Closes #1812

This PR adds support for multiple pipeline groups and multiple pipelines per group. It is still possible to start the engine with a single pipeline, but it is now also possible to start the engine with a configuration file describing one or more pipeline groups. An example configuration is included in this PR.

The controller documentation has been updated to provide more details. Optimal placement of the ITS and pipelines per NUMA node is not yet implemented. Named channels are not implemented yet.

I did not observe any performance regression with this PR.

@github-actions github-actions bot added the rust Pull requests that update Rust code label Jan 19, 2026
@lquerel lquerel changed the title Support multiple groups and pipelines [WIP] Support multiple groups and pipelines Jan 19, 2026
@lquerel lquerel self-assigned this Jan 19, 2026
@lquerel lquerel added the engine-capability Internal engine features label Jan 19, 2026
@codecov
Copy link

codecov bot commented Jan 19, 2026

Codecov Report

❌ Patch coverage is 18.59649% with 232 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.62%. Comparing base (d8a0d6d) to head (e97c423).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1823      +/-   ##
==========================================
- Coverage   84.70%   84.62%   -0.08%     
==========================================
  Files         498      498              
  Lines      146255   146385     +130     
==========================================
- Hits       123879   123875       -4     
- Misses      21842    21976     +134     
  Partials      534      534              
Components Coverage Δ
otap-dataflow 86.01% <18.59%> (-0.13%) ⬇️
query_abstraction 80.61% <ø> (ø)
query_engine 90.52% <ø> (ø)
syslog_cef_receivers ∅ <ø> (∅)
otel-arrow-go 53.50% <ø> (ø)
quiver 90.66% <ø> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@lquerel lquerel changed the title [WIP] Support multiple groups and pipelines Support multiple groups and pipelines Jan 20, 2026
@lquerel lquerel marked this pull request as ready for review January 20, 2026 00:20
@lquerel lquerel requested a review from a team as a code owner January 20, 2026 00:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

engine-capability Internal engine features rust Pull requests that update Rust code

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

Support for multiple pipeline groups and multiple pipelines

1 participant