Skip to content
Open
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
131 changes: 109 additions & 22 deletions config/collector.yaml
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might require make generate to have these changes reflect into the manifests in k8s/ directory.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 9099496

Original file line number Diff line number Diff line change
Expand Up @@ -12,18 +12,53 @@
# See the License for the specific language governing permissions and
# limitations under the License.

exporters:
googlecloud:
log:
default_log_name: opentelemetry-collector
user_agent: Google-Cloud-OTLP manifests:0.2.0 OpenTelemetry Collector Built By Google/0.121.0 (linux/amd64)
googlemanagedprometheus:
user_agent: Google-Cloud-OTLP manifests:0.2.0 OpenTelemetry Collector Built By Google/0.121.0 (linux/amd64)
receivers:
# Open two OTLP servers:
# - On port 4317, open an OTLP GRPC server
# - On port 4318, open an OTLP HTTP server
#
# Docs:
# https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver/otlpreceiver
otlp:
protocols:
grpc:
endpoint: ${env:MY_POD_IP}:4317
http:
cors:
allowed_origins:
- http://*
- https://*
endpoint: ${env:MY_POD_IP}:4318

# Open an OTLP server on port 14317 that will receive self-metrics from
# the collector itself.
# See service::telemetry::metrics for more information about OTLP self-metrics.
#
# Docs:
# https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver/otlpreceiver
otlp/self-metrics:
protocols:
grpc:
endpoint: ${env:MY_POD_IP}:14317

extensions:
# Opens an endpoint on 13133 that can be used to check the
# status of the collector. Since this does not configure the
# `path` config value, the endpoint will default to `/`.
#
# Docs:
# https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/healthcheckextension
health_check:
endpoint: ${env:MY_POD_IP}:13133

processors:
# Filters out most of the self-metrics produced by the collector.
# If you would like more information, you can add them here or
# you can remove this processor if you want all available collector
# self metrics.
#
# Docs:
# https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/filterprocessor
filter/self-metrics:
metrics:
include:
Expand All @@ -33,11 +68,23 @@ processors:
- otelcol_process_memory_rss
- otelcol_grpc_io_client_completed_rpcs
- otelcol_googlecloudmonitoring_point_count

# The batch processor is in place to regulate both the number of requests
# being made and the size of those requests.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cloud Monitoring recommends using a batch size of 200, which is the maximum batch size allowed.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in b4875a4

#
# Docs:
# https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor/batchprocessor
batch:
send_batch_max_size: 200
send_batch_size: 200
timeout: 5s

# The k8sattributes processor will fetch Kubernetes metadata and attach
# the metadata as resource attributes on your telemetry. This is important
# for proper relation between telemetry and k8s resources.
#
# Docs:
# https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/k8sattributesprocessor
k8sattributes:
extract:
metadata:
Expand All @@ -62,11 +109,21 @@ processors:
name: k8s.pod.uid
- sources:
- from: connection

# The memorylimiter will check the memory usage of the collector process.
#
# Docs:
# https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor/memorylimiterprocessor
memory_limiter:
check_interval: 1s
limit_percentage: 65
spike_limit_percentage: 20

# Transforms the uptime metric to include a version label. This allows you
# to track the versions of your deployments.
#
# Docs:
# https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/metricstransformprocessor
metricstransform/self-metrics:
transforms:
- action: update
Expand All @@ -76,10 +133,24 @@ processors:
new_label: version
new_value: Google-Cloud-OTLP manifests:0.2.0 OpenTelemetry Collector Built By Google/0.121.0 (linux/amd64)

# The resourcedetection processor is configured to detect GCP resources.
# Resource attributes that represent the GCP resource the collector is
# running on will be attached to all telemetry that goes through this
# processor.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resource detection is required, otherwise you will write time series from many resources into the same time series in GCP and will get a lot of errors.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 27ae370

#
# Docs:
# https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/resourcedetectionprocessor
# https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/resourcedetectionprocessor#gcp-metadata
resourcedetection:
detectors: [gcp]
timeout: 10s

# The transform/collision processor ensures that any attributes that may
# collide with the googlemanagedprometheus exporter's monitored resource
# construction are moved to a similar name that is not reserved.
#
# Docs:
# https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/transformprocessor
transform/collision:
metric_statements:
- context: datapoint
Expand All @@ -102,6 +173,9 @@ processors:
# https://kubernetes.io/docs/concepts/workloads/controllers.
# The relative ordering of the other controllers in this list is inconsequential since they directly
# create pods.
#
# Docs:
# https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/transformprocessor
transform/aco-gke:
metric_statements:
- context: datapoint
Expand All @@ -119,21 +193,27 @@ processors:
- set(attributes["top_level_controller_type"], "CronJob") where resource.attributes["k8s.cronjob.name"] != nil
- set(attributes["top_level_controller_name"], resource.attributes["k8s.cronjob.name"]) where resource.attributes["k8s.cronjob.name"] != nil

receivers:
otlp:
protocols:
grpc:
endpoint: ${env:MY_POD_IP}:4317
http:
cors:
allowed_origins:
- http://*
- https://*
endpoint: ${env:MY_POD_IP}:4318
otlp/self-metrics:
protocols:
grpc:
endpoint: ${env:MY_POD_IP}:14317
exporters:
# The googlecloud exporter will export telemetry to different
# Google Cloud services:
# Logs -> Cloud Logging
# Metrics -> Cloud Monitoring
# Traces -> Cloud Trace
#
# Docs:
# https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/googlecloudexporter
googlecloud:
log:
default_log_name: opentelemetry-collector
user_agent: Google-Cloud-OTLP manifests:0.2.0 OpenTelemetry Collector Built By Google/0.121.0 (linux/amd64)

# The googlemanagedprometheus exporter will send metrics to
# Google Managed Service for Prometheus.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Google Cloud Managed Service for Prometheus. Unless you have a specific reason to do otherwise, we strongly recommend you send OTel metric data with the googlemanagedprometheus exporter to get the best query experience with the lowest cost.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 27ae370

#
# Docs:
# https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/googlemanagedprometheusexporter
googlemanagedprometheus:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this need the IntToDouble flag somewhere

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is turned on as part of the Google-Built OpenTelemetry Collector image's ENTRYPOINT. The way these manifests interact with the container as well as all other documented sources, the flag will always be on.

Copy link
Contributor

@psx95 psx95 Mar 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Should the information about intToDouble flag be mentioned here to help guide in case of issues due to that flag ?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 9099496

user_agent: Google-Cloud-OTLP manifests:0.2.0 OpenTelemetry Collector Built By Google/0.121.0 (linux/amd64)

service:
extensions:
Expand Down Expand Up @@ -184,9 +264,16 @@ service:
receivers:
- otlp
telemetry:
# Changing the log encoding to json makes it so
# GKE's default log collection can recognize and
# structured data and severity from the Collector's
# self-logs.
logs:
encoding: json
metrics:
# This reader will periodically sample the Collector's
# tracked self-metrics and export them to the configured
# otlp receiver.
readers:
- periodic:
exporter:
Expand Down