feat(aggregator): support profile/system counter aggregation and custom metric Arrow output

[rayandrew]

Summary

This PR extends the aggregation pipeline to handle DFTracer counter events
(ph="C") as first-class aggregated outputs, including:

• separate aggregation of profile counters and system counters
• support for custom metric fields carried in event args
• Arrow output/schema updates for the new aggregation data
• plain-file byte-range reading support in TraceReader
• expanded C++ and Python test coverage
• documentation updates for Python and CLI surfaces

Aggregation behavior

• Added distinct profile_aggregations and system_aggregations alongside
  regular event aggregations.
• Updated chunk-level aggregation to recognize:
  • profile counters: ph="C" with cat != "sys"
  • system counters: ph="C" with cat == "sys"
• Added support for aggregating custom numeric fields from args via
  custom_metric_fields.
• Preserved merged aggregation behavior across chunk outputs and end-to-end
  utility collection.

Arrow / Python API

• Extended aggregator Arrow output with:
  • batch_type
  • ts
  • te
  • dynamic <field>_total|min|max|mean|std columns for custom metrics
• Updated Python binding signatures and docs to expose:
  • custom_metric_fields
  • compute_percentiles
• Updated Python-facing docs/examples to reflect the current schema and API.

TraceReader / byte-range handling

• Added plain-file byte-range line reading support in TraceReader.
• Kept byte-range semantics that finish the in-flight line past end_byte.
• Fixed async plain-file byte-range generation to preserve empty lines.

Tests

• Added new C++ tests for:
  • chunk aggregation of event/profile/system rows
  • merged collection of event/profile/system maps
  • end-to-end AggregatorUtility::process()
  • plain-file TraceReader byte-range behavior
  • async/sync empty-line parity for plain-file byte-range reads
• Added Python tests for:
  • profile/system/custom-metric aggregation through AggregatorUtility
  • iter_arrow() batch separation by batch_type
  • updated Arrow schema expectations
• Made server integration tests skip cleanly when TCP sockets are unavailable.

Testing

C++

Ran from build/build-tests:

ctest -j8 --output-on-failure

Result:

• 134/134 tests passed

Python

Targeted Python verification:

pytest tests/python/test_aggregator.py tests/python/test_arrow_ipc.py -q

Result:

• 14 passed

Notes

• The Arrow schema for aggregator output is no longer the old fixed 18-column shape. It now includes batch_type, ts, te, and optional custom metric columns.
• ph="C" rows are intentionally split into profile vs system outputs based on cat == "sys".
• The async plain-file byte-range reader now preserves empty lines; previous tests were aligned to the old, incorrect behavior.

[Copilot]

Pull request overview

This PR extends the DFTracer aggregation pipeline to treat counter events (ph="C") as first-class aggregated outputs (separating profile vs system counters), adds dynamic custom-metric aggregation from args, and updates Arrow/Python surfaces and TraceReader byte-range behavior accordingly.

Changes:

• Add profile/system counter aggregation paths and merge behavior across chunk/event aggregators.
• Extend Arrow output schema with batch_type, ts, te, and dynamic custom metric columns, and update Python bindings/tests/docs.
• Add plain-file byte-range line reading (sync/async parity, line completion past end_byte) and test coverage.

Reviewed changes

Copilot reviewed 27 out of 27 changed files in this pull request and generated 14 comments.

Show a summary per file

File	Description
tests/utilities/reader/test_trace_reader.cpp	Adds plain-file byte-range read_lines tests (line completion, skipping partial first line, query filter).
tests/utilities/fileio/lines/sources/test_async_plain_file_bytes_generator.cpp	Updates sync reference implementation and adds async/sync parity tests (empty lines, line completion past end).
tests/utilities/composites/dft/aggregators/test_event_aggregator_utility.cpp	New C++ test for merging event/profile/system aggregation maps.
tests/utilities/composites/dft/aggregators/test_chunk_aggregator_utility.cpp	New C++ tests for chunk aggregation separation and query/byte-range semantics for counters.
tests/utilities/composites/dft/aggregators/test_aggregator_utility.cpp	New end-to-end C++ test ensuring event/profile/system batches and custom metrics are emitted.
tests/utilities/CMakeLists.txt	Adds the new aggregator utility test sources to the utilities test target.
tests/python/test_arrow_ipc.py	Updates Arrow IPC tests to assert the new base schema columns.
tests/python/test_aggregator.py	Adds Python-level tests for profile/system/custom-metric aggregation and batch_type separation.
tests/binaries/test_dftracer_server.cpp	Skips server integration tests when TCP sockets are unavailable.
src/dftracer/utils/utilities/reader/trace_reader.cpp	Adds plain-file byte-range path using async_plain_file_bytes with query filtering.
src/dftracer/utils/utilities/composites/dft/aggregators/event_aggregator_utility.cpp	Extends merge to include profile and system aggregation maps.
src/dftracer/utils/utilities/composites/dft/aggregators/chunk_aggregator_utility.cpp	Switches chunk reading to TraceReader, adds counter handling and custom-metric aggregation for counters, splits outputs into event/profile/system maps.
src/dftracer/utils/utilities/composites/dft/aggregators/aggregator_utility.cpp	Adds batch_type and dynamic schema generation for extra keys/custom metrics, emits separate batches for event/profile/system, and avoids index building for uncompressed inputs.
src/dftracer/utils/utilities/composites/dft/aggregators/aggregation_metrics.cpp	Uses CustomMetricsMap alias for custom metric storage.
src/dftracer/utils/python/utilities/aggregator.cpp	Exposes custom_metric_fields and compute_percentiles to Python, adds shared list parsing helper.
src/dftracer/utils/python/trace_reader.cpp	Adds Arrow normalization support, string arena, and new iter_arrow params (flatten_objects, normalize).
python/dftracer/utils/utilities/_aggregator.pyi	Updates Python stub signatures for new aggregator parameters.
python/dftracer/utils/dftracer_utils_ext.pyi	Updates extension stub signatures for new aggregator parameters.
include/dftracer/utils/utilities/fileio/lines/sources/async_plain_file_bytes_generator.h	Adjusts byte-range generator to preserve empty lines and complete in-flight line past end_byte.
include/dftracer/utils/utilities/composites/dft/event.h	Adds convenience predicates for counter/profile/system/event classification.
include/dftracer/utils/utilities/composites/dft/aggregators/aggregator_utility.h	Introduces AggregationBatchType and stores it in AggregationBatch.
include/dftracer/utils/utilities/composites/dft/aggregators/aggregation_output.h	Extends aggregation outputs with profile/system maps.
include/dftracer/utils/utilities/composites/dft/aggregators/aggregation_metrics.h	Introduces CustomMetricsMap with transparent hashing and updates AggregationMetrics to use it.
docs/source/quickstart.rst	Documents custom metric aggregation + percentile option in Python quickstart.
docs/source/index.rst	Adds a short example using custom_metric_fields in the index page.
docs/source/cli.rst	Documents the three logical row types and Arrow base/custom-metric columns; updates CLI examples.
docs/source/api/utilities.rst	Documents updated Arrow schema and new Python API options.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.