[storage] Add OTLP Scope metadata validation test (part of #7050)

[SoumyaRaikwar]

Summary

Part of #7050 - Adds OTLP Scope validation test for v2 Storage API.

Changes

• integration.go: Added testOTLPScopePreservation() test + helper methods
• otlp_scope_attributes.json: OTLP fixture with InstrumentationScope metadata
• otlp_span_links.json: OTLP fixture with span links

Why

Validates OTLP-specific features (InstrumentationScope name/version) not testable with v1 model. Tests run automatically on all storage backends.

Part of #7050

[SoumyaRaikwar]

Hi @yurishkuro, @jkowall

This PR upgrades the integration tests to use v2 Storage API with OTLP fixtures
for issue #7050.

Changes:

• Updated writeTrace() to accept ptrace.Traces (v2 API)
• Added OTLP fixture loading infrastructure
• Created 3 sample OTLP fixtures with scope/resource attributes
• All tests pass ✓

For the remaining v1 fixtures: What's your preferred approach? Should I convert
all of them in a follow-up PR, or would you like a different strategy?

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.64%. Comparing base (5b64125) to head (3c1052b).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7632      +/-   ##
==========================================
- Coverage   95.67%   95.64%   -0.03%     
==========================================
  Files         311      311              
  Lines       15641    15641              
==========================================
- Hits        14964    14960       -4     
- Misses        532      535       +3     
- Partials      145      146       +1     

Flag	Coverage Δ
badger_v1	9.73% <ø> (ø)
badger_v2	2.03% <ø> (ø)
cassandra-4.x-v1-manual	14.09% <ø> (ø)
cassandra-4.x-v2-auto	2.02% <ø> (ø)
cassandra-4.x-v2-manual	2.02% <ø> (ø)
cassandra-5.x-v1-manual	14.09% <ø> (ø)
cassandra-5.x-v2-auto	2.02% <ø> (ø)
cassandra-5.x-v2-manual	2.02% <ø> (ø)
clickhouse	1.95% <ø> (ø)
elasticsearch-6.x-v1	18.60% <ø> (ø)
elasticsearch-7.x-v1	18.63% <ø> (ø)
elasticsearch-8.x-v1	18.79% <ø> (ø)
elasticsearch-8.x-v2	2.03% <ø> (ø)
elasticsearch-9.x-v2	2.03% <ø> (ø)
grpc_v1	9.61% <ø> (ø)
grpc_v2	2.03% <ø> (ø)
kafka-3.x-v2	2.03% <ø> (ø)
memory_v2	?
opensearch-1.x-v1	18.68% <ø> (ø)
opensearch-2.x-v1	18.68% <ø> (ø)
opensearch-2.x-v2	2.03% <ø> (ø)
opensearch-3.x-v2	2.03% <ø> (ø)
query	2.03% <ø> (ø)
tailsampling-processor	0.59% <ø> (ø)
unittests	94.22% <ø> (-0.03%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

Metrics Comparison Summary

Total changes across all snapshots: 53

Detailed changes per snapshot

summary_metrics_snapshot_cassandra

📊 Metrics Diff Summary

Total Changes: 53

• 🆕 Added: 0 metrics
• ❌ Removed: 53 metrics
• 🔄 Modified: 0 metrics

❌ Removed Metrics

• http_server_request_body_size_bytes (18 variants)

View diff sample

-http_server_request_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="+Inf",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.64.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_request_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="0",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.64.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_request_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="10",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.64.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_request_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="100",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.64.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_request_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="1000",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.64.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_request_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="10000",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.64.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_request_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="25",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.64.0",server_address="localhost",server_port="13133",url_scheme="http"}
...

- `http_server_request_duration_seconds` (17 variants)

View diff sample

-http_server_request_duration_seconds{http_request_method="GET",http_response_status_code="503",le="+Inf",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.64.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_request_duration_seconds{http_request_method="GET",http_response_status_code="503",le="0.005",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.64.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_request_duration_seconds{http_request_method="GET",http_response_status_code="503",le="0.01",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.64.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_request_duration_seconds{http_request_method="GET",http_response_status_code="503",le="0.025",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.64.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_request_duration_seconds{http_request_method="GET",http_response_status_code="503",le="0.05",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.64.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_request_duration_seconds{http_request_method="GET",http_response_status_code="503",le="0.075",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.64.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_request_duration_seconds{http_request_method="GET",http_response_status_code="503",le="0.1",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.64.0",server_address="localhost",server_port="13133",url_scheme="http"}
...

- `http_server_response_body_size_bytes` (18 variants)

View diff sample

-http_server_response_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="+Inf",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.64.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_response_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="0",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.64.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_response_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="10",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.64.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_response_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="100",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.64.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_response_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="1000",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.64.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_response_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="10000",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.64.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_response_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="25",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.64.0",server_address="localhost",server_port="13133",url_scheme="http"}
...

➡️ View full metrics file

[yurishkuro]

It does not fix 7050. I just see a patchwork of different threads.

[SoumyaRaikwar]

@yurishkuro - Thanks for the feedback! Cleaned up unused code in latest commit.

Question on scope: This PR currently upgrades writeTrace() to v2 API and adds OTLP scope validation.

To fully complete #7050, I'd need to also convert all 30 fixture files and update 7 methods from *model.Trace → ptrace.Traces .

Should I:

1. Expand this PR to include full migration?
2. Keep it focused and do full migration in follow-up PR(s)?

Happy to do either - just want to confirm approach before implementing!

[yurishkuro]

The full scope is defined in the issue. I'm ok to merge incremental changes, but they must be labeled "part of" not "fixes" for the issue.

[SoumyaRaikwar]

@yurishkuro - Thanks for clarifying! My apologies for it.
Updated PR to reflect this is "part of #7050" as an incremental change. Will submit follow-up PRs for fixture conversion and remaining method migration.

[SoumyaRaikwar]

@yurishkuro - Added OTLP Scope validation test. Ready for review.

Changes:

• Added testOTLPScopePreservation() - validates Scope name/version preservation
• Test runs on all storage backends automatically

[Copilot]

Pull Request Overview

This PR adds integration tests to validate OTLP InstrumentationScope metadata preservation through the v2 Storage API, as part of #7050.

• Adds testOTLPScopePreservation() to verify scope name and version are preserved through storage backends
• Refactors writeTrace() to accept ptrace.Traces directly instead of *model.Trace, enabling native OTLP testing
• Includes two OTLP fixture files for testing scope attributes and span links

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
internal/storage/integration/integration.go	Refactored writeTrace() signature to accept ptrace.Traces, added testOTLPScopePreservation() test with helper functions loadOTLPFixture() and extractTraceID(), updated all callers to convert v1 traces to OTLP format before writing
internal/storage/integration/fixtures/traces/otlp_scope_attributes.json	OTLP fixture with InstrumentationScope metadata (name, version, attributes) and span data for validation testing
internal/storage/integration/fixtures/traces/otlp_span_links.json	OTLP fixture with span links data, currently unused in tests

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[SoumyaRaikwar]

@yurishkuro ,Fixed OTLPScopePreservation test failures by normalizing OTLP fixture timestamps to recent time window (similar to correctTime() for v1 fixtures). Also addressed Copilot feedback: removed unused scope attributes from fixture and cleaned up log message. All ES/OpenSearch tests now passing.

could you please review

[Manik2708]

why the new test has old model.Trace? we want to upgrade the tests completely to use otlp traces. if new fixtures are added, why we need to convert them back to old models?

[SoumyaRaikwar]

it is now resolved: the scope test is fully OTLP-native.

[Manik2708]

checking only lengths and scope name and versions is not enough, we need to check whether the whole trace is preserved or not. this is the reason of this issue, as scope attributes were missed in conversion. we might have to do

      assert(t, expected, retrieved)

it might fail in some backends because some normalization is done without any loss of data but as for now, checking for memory backend is enough.

[Manik2708]

this is the reason why most of the tests are failing. @yurishkuro we are preserving only two type of attributes of span link in form of reference child_of and follows_from with key opentracing.ref_type. do we need to preserve random attributes like these for links?

[yurishkuro]

The goal is to ensure storing of OTLP data is lossless. But not all backends implement that today. The actual ticket is about upgrading e2e tests to use OTLP, not to add new functional requirements, which would be a separate issue.

[Manik2708]

yes, but tests are failing, that's because of the above reason I specified. So just wanna ask should we test memory as for now (by introducing a flag similar to SpanKind in e2e tests) or change the fixtures?

[yurishkuro]

don't introduce attributes in the test fixtures where they are not supported. Create a dedicated fixture that tests that and exclude it by name from some of the tests for backends that don't support them (ES and CH should properly support all attributes).

[Manik2708]

For ES, this fixture would fail because we are handling link attributes in this way:

jaeger/internal/storage/v2/elasticsearch/tracestore/to_dbmodel.go

Lines 360 to 375 in b728d61

	func refTypeFromLink(link ptrace.SpanLink) dbmodel.ReferenceType {
	refTypeAttr, ok := link.Attributes().Get(conventions.AttributeOpentracingRefType)
	if !ok {
	return dbmodel.FollowsFrom
	}
	return strToDbSpanRefType(refTypeAttr.Str())
	}
	func strToDbSpanRefType(attr string) dbmodel.ReferenceType {
	if attr == conventions.AttributeOpentracingRefTypeChildOf {
	return dbmodel.ChildOf
	}
	// There are only 2 types of SpanRefType we assume that everything
	// that's not a model.ChildOf is a model.FollowsFrom
	return dbmodel.FollowsFrom
	}

So my question was, do we get only these 2 types of attributes from links of OTLP traces (it is assumed here in ptrace.Traces to model.Trace too)? because in ES we are preserving only these two attributes via Reference (Here SpanLink has a random attribute "link.type", that's why all tests are failing except memory.

[yurishkuro]

all attributes must be preserved in the storage.

[SoumyaRaikwar]

@yurishkuro @Manik2708

Thanks for the feedback! Updated:

• Removed unsupported custom attributes from span link fixtures
• Added comprehensive assertions verifying each link's TraceID and SpanID
• Scope test now validates full metadata (name + version)

Tests confirm OTLP structure preservation through v2 API for memory backend.
could you please review again

[SoumyaRaikwar]

@yurishkuro sorry for the noise

[SoumyaRaikwar]

Closing this PR as it got messy with multiple rebases and conflicts. I've created a clean PR with a better approach for #7050.

New PR: #7761 - Focuses on removing v1 downgrades from integration test read paths as the first step in a 2-PR migration strategy.

The OTLP Scope metadata validation work from this PR may be revisited as a follow-up enhancement after the core migration is complete.

Thanks for the reviews and guidance, @yurishkuro!