Bigtable: BigtableCloudMonitoringExporter fails with "unrecognized metric labels" for connection_pool/outstanding_rpcs metric

[jim-kleckner-hpe]

Environment details

1. Bigtable
2. Linux (GKE on Google Kubernetes Engine, us-central1)
3. OpenJDK 11
4. google-cloud-bigtable 2.73.0 (also reproducible with 2.69.0+)

Steps to reproduce

1. Create a BigtableDataClient with default settings (built-in metrics enabled by default)
2. Perform any Bigtable operation
3. Wait ~60 seconds for the metrics export cycle
4. Observe WARNING log from BigtableCloudMonitoringExporter$1 onFailure

Code example

BigtableDataSettings settings = BigtableDataSettings.newBuilder()
    .setProjectId("my-project")
    .setInstanceId("my-instance")
    .build();

BigtableDataClient client = BigtableDataClient.create(settings);

// Perform any operation, then wait ~60s for metrics export cycle

Stack trace

WARNING: createServiceTimeSeries request failed for application metrics.
shaded_com_google.api.gax.rpc.InvalidArgumentException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: One or more TimeSeries could not be written: timeSeries[3] (metric.type="bigtable.googleapis.com/internal/client/connection_pool/outstanding_rpcs", metric.labels={"app_profile": "", "project_id": "dev-main-01", "client_name": "bigtable-java/2.73.0", "instance": "analytics-data"}, resource.type="bigtable_client", resource.labels={"uuid": "java-32cd4f1a-8630-4708-b5d2-28f9d2f815cb1@mainstream-dequeue-6cb4c6d648-zjqr9-0", "cloud_platform": "gcp_kubernetes_engine", "host_name": "mainstream-dequeue-6cb4c6d648-zjqr9", "project_id": "dev-main-01", "region": "us-central1", "client_project": "dev-main-01", "app_profile": "", "instance": "analytics-data", "client_name": "java-bigtable/2.73.0", "host_id": "6238638445181415795"}): unrecognized metric labels [app_profile, project_id, client_name, instance]
    at shaded_com_google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:92)
    at shaded_com_google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:98)
    at shaded_com_google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:66)
    at shaded_com_google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97)
    at shaded_com_google.api.core.ApiFutures$1.onFailure(ApiFutures.java:84)
    at shaded_com_google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1125)
    at shaded_com_google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
    ...
Caused by: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: One or more TimeSeries could not be written: timeSeries[3] (metric.type="bigtable.googleapis.com/internal/client/connection_pool/outstanding_rpcs", metric.labels={"app_profile": "", "project_id": "dev-main-01", "client_name": "bigtable-java/2.73.0", "instance": "analytics-data"}, resource.type="bigtable_client"): unrecognized metric labels [app_profile, project_id, client_name, instance]
    at io.grpc.Status.asRuntimeException(Status.java:532)
    ... 18 more

External references such as API reference guides

• The outstanding_rpcs metric was introduced in v2.69.0 via feat(bigtable): expose a metric to track the number of outstanding rpcs (unary , streaming) in channel pool #2696 and feat(bigtable): add internal grpc subconnections metric and add outstanding rpcs to INTERNAL_VIEW #2700

Any additional information below

The error occurs every ~60 seconds on every pod, across multiple pods simultaneously. The Cloud Monitoring metric descriptor for bigtable.googleapis.com/internal/client/connection_pool/outstanding_rpcs does not recognize the labels [app_profile, project_id, client_name, instance] that the client sends. Either the server-side metric descriptor needs to be updated, or the client should not be sending these labels for this metric.

Bigtable data operations are unaffected — only client-side metrics export fails.

Workaround: disable built-in metrics with BigtableDataSettings.newBuilder().setMetricsProvider(NoopMetricsProvider.create()), but this disables all client-side Cloud Monitoring metrics.

[jim-kleckner-hpe]

See also: #2757