Getting Google.Protobuf.InvalidProtocolBufferException when running the exported in Aspire

[nikneem]

Package

OpenTelemetry.Exporter.OpenTelemetryProtocol

Package Version

1.11.1

Runtime Version

net9.0

Description

I noticed when I change the data access protection code in line 34 of Program.cs (WebApplication1) from using DefaultAzureCredential to use VisualStudioCredential I do not get the error. If I change it to ManagedIdentityCredential I do get the exception. Since I am running locally I do not have a managed identity and it seems that the error that gets thrown by that call failure is causing the telemetry to fail.

I also noticed that even if I use DefaultAzureCredential but I do not use Azure.Monitor.OpenTelemetry.AspNetCore nuget package in the service defaults project to send my telemetry to application insights I do not get the exception. It seems the issue is sending the failed managed identity call from DefaultAzureCredential through the Azure.Monitor.OpenTelemetry.AspNetCore nuget package to application insights

Found the problem is in the OpenTelemetry.Exporter.OpenTelemetryProtocol NuGet package. I've recently upgraded this package leaving me with the most recent version (at this time) of 1.11.1 that contains the problem. Degrading the version to 1.11.0 still contains this problem. But degrading to 1.10.0 eliminates the problem. So with the bump of version 1.10 to 1.11, something was introduced to cause this problem.

Steps to Reproduce

I have created a repo for you so you can see the exception:

https://github.com/KenBrannigan/AspireDefaultAzureCredentialIssue

After running the app wait a few seconds and the exception will display.

Expected Result

No exceptions to be thrown when running Aspire.

Actual Result

info: Aspire.Hosting.DistributedApplication[0]
Distributed application started. Press Ctrl+C to shut down.
fail: Aspire.Hosting.Dashboard.Grpc.AspNetCore.Server.ServerCallHandler[6]
Error when executing service method 'Export'.
Google.Protobuf.InvalidProtocolBufferException: While parsing a protocol message, the input ended unexpectedly in the middle of a field. This could mean either that the input has been truncated or that an embedded message misreported its own length.
at Google.Protobuf.SegmentedBufferHelper.RefillFromReadOnlySequence(ReadOnlySpan1& buffer, ParserInternalState& state, Boolean mustSucceed) at Google.Protobuf.SegmentedBufferHelper.RefillBuffer(ReadOnlySpan1& buffer, ParserInternalState& state, Boolean mustSucceed)
at Google.Protobuf.ParsingPrimitives.ReadRawByte(ReadOnlySpan1& buffer, ParserInternalState& state) at Google.Protobuf.ParsingPrimitives.ParseRawVarint32SlowPath(ReadOnlySpan1& buffer, ParserInternalState& state)
at Google.Protobuf.ParsingPrimitives.ParseRawVarint32(ReadOnlySpan1& buffer, ParserInternalState& state) at OpenTelemetry.Proto.Trace.V1.Status.pb::Google.Protobuf.IBufferMessage.InternalMergeFrom(ParseContext& input) in //artifacts/obj/Aspire.Dashboard/win-x64/Release/net8.0/win-x64/opentelemetry/proto/trace/v1/Trace.cs:line 2612 at Google.Protobuf.ParsingPrimitivesMessages.ReadRawMessage(ParseContext& ctx, IMessage message) at Google.Protobuf.ParsingPrimitivesMessages.ReadMessage(ParseContext& ctx, IMessage message) at OpenTelemetry.Proto.Trace.V1.Span.pb::Google.Protobuf.IBufferMessage.InternalMergeFrom(ParseContext& input) in //artifacts/obj/Aspire.Dashboard/win-x64/Release/net8.0/win-x64/opentelemetry/proto/trace/v1/Trace.cs:line 1642 at Google.Protobuf.ParsingPrimitivesMessages.ReadRawMessage(ParseContext& ctx, IMessage message) at Google.Protobuf.ParsingPrimitivesMessages.ReadMessage(ParseContext& ctx, IMessage message) at Google.Protobuf.FieldCodec.<>c__DisplayClass32_01.b__0(ParseContext& ctx)
at Google.Protobuf.Collections.RepeatedField1.AddEntriesFrom(ParseContext& ctx, FieldCodec1 codec)
at OpenTelemetry.Proto.Trace.V1.ScopeSpans.pb::Google.Protobuf.IBufferMessage.InternalMergeFrom(ParseContext& input) in //artifacts/obj/Aspire.Dashboard/win-x64/Release/net8.0/win-x64/opentelemetry/proto/trace/v1/Trace.cs:line 844 at Google.Protobuf.ParsingPrimitivesMessages.ReadRawMessage(ParseContext& ctx, IMessage message)
at Google.Protobuf.ParsingPrimitivesMessages.ReadMessage(ParseContext& ctx, IMessage message)
at Google.Protobuf.FieldCodec.<>c__DisplayClass32_01.b__0(ParseContext& ctx) at Google.Protobuf.Collections.RepeatedField1.AddEntriesFrom(ParseContext& ctx, FieldCodec1 codec) at OpenTelemetry.Proto.Trace.V1.ResourceSpans.pb::Google.Protobuf.IBufferMessage.InternalMergeFrom(ParseContext& input) in /_/artifacts/obj/Aspire.Dashboard/win-x64/Release/net8.0/win-x64/opentelemetry/proto/trace/v1/Trace.cs:line 560 at Google.Protobuf.ParsingPrimitivesMessages.ReadRawMessage(ParseContext& ctx, IMessage message) at Google.Protobuf.ParsingPrimitivesMessages.ReadMessage(ParseContext& ctx, IMessage message) at Google.Protobuf.FieldCodec.<>c__DisplayClass32_01.b__0(ParseContext& ctx)
at Google.Protobuf.Collections.RepeatedField1.AddEntriesFrom(ParseContext& ctx, FieldCodec1 codec)
at OpenTelemetry.Proto.Collector.Trace.V1.ExportTraceServiceRequest.pb::Google.Protobuf.IBufferMessage.InternalMergeFrom(ParseContext& input) in //artifacts/obj/Aspire.Dashboard/win-x64/Release/net8.0/win-x64/opentelemetry/proto/collector/trace/v1/TraceService.cs:line 241
at Google.Protobuf.ParsingPrimitivesMessages.ReadRawMessage(ParseContext& ctx, IMessage message)
at Google.Protobuf.MessageExtensions.MergeFrom(IMessage message, ReadOnlySequence1 data, Boolean discardUnknownFields, ExtensionRegistry registry) at Google.Protobuf.MessageParser1.ParseFrom(ReadOnlySequence1 data) at OpenTelemetry.Proto.Collector.Trace.V1.TraceService.__Helper_DeserializeMessage[T](DeserializationContext context, MessageParser1 parser) in //artifacts/obj/Aspire.Dashboard/win-x64/Release/net8.0/win-x64/opentelemetry/proto/collector/trace/v1/TraceServiceGrpc.cs:line 62
at OpenTelemetry.Proto.Collector.Trace.V1.TraceService.<>c.<.cctor>b__13_0(DeserializationContext context) in //artifacts/obj/Aspire.Dashboard/win-x64/Release/net8.0/win-x64/opentelemetry/proto/collector/trace/v1/TraceServiceGrpc.cs:line 69
at Grpc.AspNetCore.Server.Internal.PipeExtensions.ReadSingleMessageAsync[T](PipeReader input, HttpContextServerCallContext serverCallContext, Func2 deserializer) at Grpc.AspNetCore.Server.Internal.CallHandlers.UnaryServerCallHandler3.HandleCallAsyncCore(HttpContext httpContext, HttpContextServerCallContext serverCallContext)
at Grpc.AspNetCore.Server.Internal.CallHandlers.ServerCallHandlerBase3.g__AwaitHandleCall|8_0(HttpContextServerCallContext serverCallContext, Method2 method, Task handleCall)

Additional Context

.NET Version info
.NET SDK:
Version: 9.0.102
Commit: cb83cd4923
Workload version: 9.0.100-manifests.4a54b1a6
MSBuild version: 17.12.18+ed8c6aec5

Runtime Environment:
OS Name: Windows
OS Version: 10.0.22631
OS Platform: Windows
RID: win-x64
Base Path: C:\Program Files\dotnet\sdk\9.0.102\

.NET workloads installed:
[aspire]
Installation Source: VS 17.12.35707.178
Manifest Version: 8.2.2/8.0.100
Manifest Path: C:\Program Files\dotnet\sdk-manifests\8.0.100\microsoft.net.sdk.aspire\8.2.2\WorkloadManifest.json
Install Type: Msi

[maccatalyst]
Installation Source: VS 17.12.35707.178
Manifest Version: 18.1.9163/9.0.100
Manifest Path: C:\Program Files\dotnet\sdk-manifests\9.0.100\microsoft.net.sdk.maccatalyst\18.1.9163\WorkloadManifest.json
Install Type: Msi

[maui-windows]
Installation Source: VS 17.12.35707.178
Manifest Version: 9.0.14/9.0.100
Manifest Path: C:\Program Files\dotnet\sdk-manifests\9.0.100\microsoft.net.sdk.maui\9.0.14\WorkloadManifest.json
Install Type: Msi

[ios]
Installation Source: VS 17.12.35707.178
Manifest Version: 18.1.9163/9.0.100
Manifest Path: C:\Program Files\dotnet\sdk-manifests\9.0.100\microsoft.net.sdk.ios\18.1.9163\WorkloadManifest.json
Install Type: Msi

[android]
Installation Source: VS 17.12.35707.178
Manifest Version: 35.0.7/9.0.100
Manifest Path: C:\Program Files\dotnet\sdk-manifests\9.0.100\microsoft.net.sdk.android\35.0.7\WorkloadManifest.json
Install Type: Msi

Configured to use loose manifests when installing new manifests.

Host:
Version: 9.0.1
Architecture: x64
Commit: c8acea2262

.NET SDKs installed:
9.0.102 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
Microsoft.AspNetCore.App 6.0.36 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 7.0.20 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 8.0.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 9.0.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 6.0.36 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 7.0.20 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 8.0.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 9.0.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.WindowsDesktop.App 6.0.36 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 7.0.20 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 8.0.12 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 9.0.1 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Other architectures found:
x86 [C:\Program Files (x86)\dotnet]
registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]

Environment variables:
Not set

global.json file:
Not found

Anything else?
I noticed when I change the data access protection code in line 34 of Program.cs (WebApplication1) from using DefaultAzureCredential to use VisualStudioCredential I do not get the error. If I change it to ManagedIdentityCredential I do get the exception. Since I am running locally I do not have a managed identity and it seems that the error that gets thrown by that call failure is causing the telemetry to fail.

I also noticed that even if I use DefaultAzureCredential but I do not use Azure.Monitor.OpenTelemetry.AspNetCore nuget package in the service defaults project to send my telemetry to application insights I do not get the exception. It seems the issue is sending the failed managed identity call from DefaultAzureCredential through the Azure.Monitor.OpenTelemetry.AspNetCore nuget package to application insights

Found the problem is in the OpenTelemetry.Exporter.OpenTelemetryProtocol NuGet package. I've recently upgraded this package leaving me with the most recent version (at this time) of 1.11.1 that contains the problem. Degrading the version to 1.11.0 still contains this problem. But degrading to 1.10.0 eliminates the problem. So with the bump of version 1.10 to 1.11, something was introduced to cause this problem.

[rajkumar-rangaraj]

To help with easier analysis and debugging, could you check if this issue can be reproduced outside of Aspire, ideally in a minimal, self-contained environment?

02/01 - Update - I got this recreated, will debug further to find the root cause.

[rajkumar-rangaraj]

I reproduced this exception (Google.Protobuf.InvalidProtocolBufferException) but did not observe any telemetry loss. Is this the same behavior others are experiencing as well?

[rajkumar-rangaraj]

Related to #6117. This occurs when the Activity.StatusDescription contains a lengthy string. The Azure.Storage.Blobs.BlobRestClient should consider adding this information to activity events instead of dumping the entire stack to the status description. This issue was resolved in the OTLP exporter last week through PR #6119.

Azure.RequestFailedException: Key based authentication is not permitted on this storage account.
RequestId:aaaaaaaaaa
Time:2025-02-03T20:34:52.3552470Z
Status: 403 (Key based authentication is not permitted on this storage account.)
ErrorCode: KeyBasedAuthenticationNotPermitted

Content:
<?xml version="1.0" encoding="utf-8"?><Error><Code>KeyBasedAuthenticationNotPermitted</Code><Message>Key based authentication is not permitted on this storage account.
RequestId:aaaaaaaaaa
Time:2025-02-03T20:34:52.3552470Z</Message></Error>

Headers:
Server: Microsoft-HTTPAPI/2.0
x-ms-request-id: aaaaaaaaaa
x-ms-error-code: KeyBasedAuthenticationNotPermitted
Date: Mon, 03 Feb 2025 20:34:51 GMT
Content-Length: 269
Content-Type: application/xml

   at Azure.Storage.Blobs.BlobRestClient.Download(String snapshot, String versionId, Nullable`1 timeout, String range, String leaseId, Nullable`1 rangeGetContentMD5, Nullable`1 rangeGetContentCRC64, String encryptionKey, String encryptionKeySha256, Nullable`1 encryptionAlgorithm, Nullable`1 ifModifiedSince, Nullable`1 ifUnmodifiedSince, String ifMatch, String ifNoneMatch, String ifTags, CancellationToken cancellationToken)
   at Azure.Storage.Blobs.Specialized.BlobBaseClient.StartDownloadAsync(HttpRange range, BlobRequestConditions conditions, DownloadTransferValidationOptions validationOptions, Int64 startOffset, Boolean async, CancellationToken cancellationToken)
   at Azure.Storage.Blobs.Specialized.BlobBaseClient.DownloadStreamingInternal(HttpRange range, BlobRequestConditions conditions, DownloadTransferValidationOptions transferValidationOverride, IProgress`1 progressHandler, String operationName, Boolean async, CancellationToken cancellationToken)
   at Azure.Storage.Blobs.PartitionedDownloader.DownloadToInternal(Stream destination, BlobRequestConditions conditions, Boolean async, CancellationToken cancellationToken)

[cijothomas]

should consider adding this information to activity events instead of dumping the entire stack to the status description

Or to Logs, instead of Spans if the below OTEP gets approved and made part of spec/conventions.
open-telemetry/opentelemetry-specification#4333