[Bug]: After upgrading 2.14.1 -> 2.15.1 SPM stopped working #3537
Description
What happened?
After upgrading to 2.15.1 from 2.14.1, SPM (Service Performance Monitoring) stopped working. There are traces created, however, i am seeing No data yet! Please see these instructions on how to set up your span metrics. for all services and kinds. The same configuration works well with the previous version.
Steps to reproduce
- Deploy with the given configurations.
- Make sure traces are being created, then switch to Monitor tab.
- It will say "No data" even there is.
Expected behavior
- Deploy with the given configurations.
- Make sure traces are being created, then switch to Monitor tab.
- You should be able to see data by selecting the corresponding service/kind.
Relevant log output
Screenshot
No response
Additional context
No response
Jaeger backend version
No response
SDK
No response
Pipeline
No response
Stogage backend
No response
Operating system
No response
Deployment model
No response
Deployment configs
extensions:
jaeger_storage:
backends:
opensearch_storage:
opensearch:
server_urls:
- http://OPENSEARCH_ADDRESS
indices:
index_prefix: "jaeger-main"
spans:
date_layout: "2006-01-02"
rollover_frequency: "day"
shards: 5
replicas: 1
services:
date_layout: "2006-01-02"
rollover_frequency: "day"
shards: 5
replicas: 1
dependencies:
date_layout: "2006-01-02"
rollover_frequency: "day"
shards: 5
replicas: 1
sampling:
date_layout: "2006-01-02"
rollover_frequency: "day"
shards: 5
replicas: 1
metric_backends:
prometheus_storage:
prometheus:
endpoint: http://PROMETHEUS_ADDRESS
normalize_calls: true
normalize_duration: true
jaeger_query:
storage:
traces: opensearch_storage
metrics: prometheus_storage
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
batch: {}
connectors:
spanmetrics:
histogram:
explicit:
buckets: [5ms, 10ms, 25ms, 50ms, 100ms, 250ms, 500ms, 1s, 2.5s, 5s, 10s]
exporters:
debug: {}
jaeger_storage_exporter:
trace_storage: opensearch_storage
prometheus:
endpoint: 0.0.0.0:9464
service:
extensions: [jaeger_storage, jaeger_query]
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [spanmetrics, jaeger_storage_exporter]
metrics/spanmetrics:
receivers: [spanmetrics]
processors: []
exporters: [prometheus]