Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove references to deprecated processor names otel_trace_raw and service_map_stateful #5017

Merged
merged 1 commit into from
Oct 3, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Trace Peer Forwarder Processor

This processor is used to reduce the number of Events that will be forwarded in a Trace Analytics pipeline by half when using [Peer Forwarder](https://github.com/opensearch-project/data-prepper/blob/main/docs/peer_forwarder.md).
It groups the events based on `trace_id` similar to `service_map_stateful` and `otel_trace_raw ` processors.
It groups the events based on `trace_id` similar to `service_map` and `otel_traces ` processors.

In [Trace Analytics pipeline](https://github.com/opensearch-project/data-prepper/blob/main/docs/trace_analytics.md#trace-analytics-pipeline) each event is duplicated, when it is sent from `otel-trace-pipeline` to `raw-pipeline` and `service-map-pipeline`.
So, the event will be forwarded once in each pipeline. Using this processor event will be forwarded only once in `otel-trace-pipeline` to correct peer.
Expand All @@ -26,7 +26,7 @@ raw-pipeline:
pipeline:
name: "entry-pipeline"
processor:
- otel_trace_raw:
- otel_traces:
sink:
- opensearch:
service-map-pipeline:
Expand All @@ -35,7 +35,7 @@ service-map-pipeline:
pipeline:
name: "entry-pipeline"
processor:
- service_map_stateful:
- service_map:
sink:
- opensearch:
```
Expand Down
4 changes: 2 additions & 2 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ raw-pipeline:
pipeline:
name: "entry-pipeline"
processor:
- otel_trace_raw:
- otel_traces:
sink:
- stdout:
service-map-pipeline:
Expand All @@ -62,7 +62,7 @@ service-map-pipeline:
pipeline:
name: "entry-pipeline"
processor:
- service_map_stateful:
- service_map:
sink:
- stdout:
```
Expand Down
2 changes: 1 addition & 1 deletion docs/core_apis.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ authentication:

### Peer Forwarder
Peer forwarder can be configured to enable stateful aggregation across multiple Data Prepper nodes. For more information on configuring Peer Forwarder, see [Peer Forwarder Configuration](https://github.com/opensearch-project/data-prepper/blob/main/docs/peer_forwarder.md).
It is supported by `service_map_stateful`, `otel_trace_raw` and `aggregate` processors.
It is supported by `service_map`, `otel_traces` and `aggregate` processors.

### Shutdown Timeouts
When the DataPrepper `shutdown` API is invoked, the sink and processor `ExecutorService`'s are given time to gracefully shutdown and clear any in-flight data. The default graceful shutdown timeout for these `ExecutorService`'s is 10 seconds. This can be configured with the following optional parameters:
Expand Down
4 changes: 2 additions & 2 deletions docs/peer_forwarder.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@

## Peer Forwarder
An HTTP service which performs peer forwarding of `Event` between Data Prepper nodes for aggregation. Currently, supported by `aggregate`, `service_map_stateful`, `otel_trace_raw` processors.
An HTTP service which performs peer forwarding of `Event` between Data Prepper nodes for aggregation. Currently, supported by `aggregate`, `service_map`, `otel_traces` processors.

Peer Forwarder groups events based on the identification keys provided the processors.
For `service_map_stateful` and `otel_trace_raw` it's `traceId` by default and can not be configured.
For `service_map` and `otel_traces` it's `traceId` by default and can not be configured.
It's configurable for `aggregate` processor using `identification_keys` configuration option. You can find more information about identification keys [here](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys).

---
Expand Down
10 changes: 5 additions & 5 deletions docs/trace_analytics.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,9 +33,9 @@ The [OpenTelemetry source](../data-prepper-plugins/otel-trace-source/README.md)
### Processor

We have two processor for the Trace Analytics feature,
* *otel_trace_raw* - This is a processor that receives collection of [Span](../data-prepper-api/src/main/java/org/opensearch/dataprepper/model/trace/Span.java) records sent from [otel-trace-source](../data-prepper-plugins/otel-trace-source/README.md), does stateful processing on extracting and filling-in trace group related fields.
* *otel_traces* - This is a processor that receives collection of [Span](../data-prepper-api/src/main/java/org/opensearch/dataprepper/model/trace/Span.java) records sent from [otel-trace-source](../data-prepper-plugins/otel-trace-source/README.md), does stateful processing on extracting and filling-in trace group related fields.
* *otel_trace_group* - This is a processor that fills in the missing trace group related fields in the collection of [Span](../data-prepper-api/src/main/java/org/opensearch/dataprepper/model/trace/Span.java) records by looking up the opensearch backend.
* *service_map_stateful* - This processor performs the required preprocessing on the trace data and build metadata to display the service-map OpenSearch Dashboards dashboards.
* *service_map* - This processor performs the required preprocessing on the trace data and build metadata to display the service-map OpenSearch Dashboards dashboards.


### OpenSearch sink
Expand Down Expand Up @@ -118,7 +118,7 @@ raw-pipeline:
# The raw processor does bulk request to your OpenSearch sink, so configure the batch_size higher.
batch_size: 3200
processor:
- otel_trace_raw:
- otel_traces:
- otel_trace_group:
hosts: [ "https://localhost:9200" ]
# Change to your credentials
Expand Down Expand Up @@ -152,7 +152,7 @@ service-map-pipeline:
pipeline:
name: "otel-trace-pipeline"
processor:
- service_map_stateful:
- service_map:
# The window duration is the maximum length of time the data prepper stores the most recent trace data to evaluvate service-map relationships.
# The default is 3 minutes, this means we can detect relationships between services from spans reported in last 3 minutes.
# Set higher value if your applications have higher latency.
Expand Down Expand Up @@ -250,7 +250,7 @@ pipeline authors the ability to configure other processors to modify spans or tr

To provide a migration path, Data Prepper 1.4 introduced the following changes.
* The `otel_trace_source` has an optional parameter `record_type` which can be set to `event`. When configured, it will output event objects.
* The `otel_trace_raw` replaces `otel_trace_raw_prepper` for event-based spans.
* The `otel_traces` replaces `otel_trace_raw_prepper` for event-based spans.
* The `otel_trace_group` replaces `otel_trace_group_prepper` for event-based spans.

In Data Prepper 2.0, the `otel_trace_source` will only output Events. Data Prepper 2.0 also removes
Expand Down
4 changes: 2 additions & 2 deletions examples/config/example-pipelines.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ raw-pipeline:
pipeline:
name: "entry-pipeline"
processor:
- otel_trace_raw:
- otel_traces:
sink:
- opensearch:
hosts: [ "https://<your-domain>.<region>.es.amazonaws.com" ]
Expand All @@ -26,7 +26,7 @@ service-map-pipeline:
pipeline:
name: "entry-pipeline"
processor:
- service_map_stateful:
- service_map:
sink:
- opensearch:
hosts: [ "https://<your-domain>.<region>.es.amazonaws.com" ]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ Resources:
pipeline:
name: "entry-pipeline"
processor:
- otel_trace_raw:
- otel_traces:
sink:
- stdout:
- Name: DATA_PREPPER_CONFIG_YAML
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ raw-pipeline:
pipeline:
name: "entry-pipeline"
processor:
- otel_trace_raw:
- otel_traces:
# trace_flush_interval: 6
sink:
- stdout:
4 changes: 2 additions & 2 deletions examples/dev/k8s/data-prepper.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ data:
pipeline:
name: "entry-pipeline"
processor:
- otel_trace_raw:
- otel_traces:
sink:
- opensearch:
hosts: [ "https://opensearch:9200" ]
Expand All @@ -40,7 +40,7 @@ data:
pipeline:
name: "entry-pipeline"
processor:
- service_map_stateful:
- service_map:
sink:
- opensearch:
hosts: ["https://opensearch:9200"]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ raw-pipeline:
pipeline:
name: "entry-pipeline"
processor:
- otel_trace_raw:
- otel_traces:
sink:
- opensearch:
hosts: [ "https://node-0.example.com:9200" ]
Expand All @@ -30,7 +30,7 @@ service-map-pipeline:
pipeline:
name: "entry-pipeline"
processor:
- service_map_stateful:
- service_map:
sink:
- opensearch:
hosts: ["https://node-0.example.com:9200"]
Expand Down
4 changes: 2 additions & 2 deletions examples/jaeger-hotrod/pipelines.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ raw-pipeline:
pipeline:
name: "entry-pipeline"
processor:
- otel_trace_raw:
- otel_traces:
sink:
- opensearch:
hosts: [ "https://node-0.example.com:9200" ]
Expand All @@ -27,7 +27,7 @@ service-map-pipeline:
pipeline:
name: "entry-pipeline"
processor:
- service_map_stateful:
- service_map:
sink:
- opensearch:
hosts: ["https://node-0.example.com:9200"]
Expand Down
4 changes: 2 additions & 2 deletions examples/trace_analytics.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ raw-pipeline:
pipeline:
name: "entry-pipeline"
processor:
- otel_trace_raw:
- otel_traces:
sink:
- opensearch:
hosts: [ "https://node-0.example.com:9200" ]
Expand All @@ -29,7 +29,7 @@ service-map-pipeline:
pipeline:
name: "entry-pipeline"
processor:
- service_map_stateful:
- service_map:
sink:
- opensearch:
hosts: ["https://node-0.example.com:9200"]
Expand Down
4 changes: 2 additions & 2 deletions examples/trace_analytics_no_ssl.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ raw-pipeline:
pipeline:
name: "entry-pipeline"
processor:
- otel_trace_raw:
- otel_traces:
sink:
- opensearch:
hosts: [ "https://node-0.example.com:9200" ]
Expand All @@ -28,7 +28,7 @@ service-map-pipeline:
pipeline:
name: "entry-pipeline"
processor:
- service_map_stateful:
- service_map:
sink:
- opensearch:
hosts: ["https://node-0.example.com:9200"]
Expand Down
4 changes: 2 additions & 2 deletions examples/trace_analytics_no_ssl_2x.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ raw-pipeline:
pipeline:
name: "entry-pipeline"
processor:
- otel_trace_raw:
- otel_traces:
sink:
- opensearch:
hosts: [ "https://node-0.example.com:9200" ]
Expand All @@ -27,7 +27,7 @@ service-map-pipeline:
pipeline:
name: "entry-pipeline"
processor:
- service_map_stateful:
- service_map:
sink:
- opensearch:
hosts: ["https://node-0.example.com:9200"]
Expand Down
Loading