---

Getting Started   •   Migrating from Smart Agent   •   Migrating from Splunk Connect for Kubernetes

Configuration   •   Components   •   Monitoring   •   Security   •   Sizing   •   Troubleshooting

---

Splunk OpenTelemetry Collector for Kubernetes

The Splunk OpenTelemetry Collector for Kubernetes is a Helm chart for the Splunk Distribution of OpenTelemetry Collector. This chart creates a Kubernetes DaemonSet along with other Kubernetes objects in a Kubernetes cluster and provides a unified way to receive, process and export metric, trace, and log data for:

• Splunk Enterprise
• Splunk Cloud Platform
• Splunk Observability Cloud

Current Status

• The Splunk OpenTelemetry Collector for Kubernetes Helm chart is production tested; it is in use by a number of customers in their production environments
• Customers using the helm chart can receive direct help from official Splunk support within SLA's
• The core configuration experience for metrics, traces and logs collection is stable and well supported. Please review the versioning and breaking changes guidance for upgrade best practices.

Installations that use this distribution can receive direct help from Splunk's support teams. Customers are free to use the core OpenTelemetry OSS components (several do!). We will provide best effort guidance for using these components; however, only the Splunk distributions are in scope for official Splunk support and support-related SLAs.

This distribution currently supports:

• Splunk APM via the otlp_http exporter.
• Splunk Infrastructure Monitoring via the signalfx exporter. More information available here.
• Splunk Cloud or Splunk Enterprise via the splunk_hec exporter.

Supported Kubernetes distributions

The Helm chart works with default configurations of the main Kubernetes distributions. Use actively supported versions:

• Vanilla (unmodified version) Kubernetes
• Amazon Elastic Kubernetes Service including with Fargate profiles
• Azure Kubernetes Service
• Google Kubernetes Engine including GKE Autopilot
• Red Hat OpenShift

While this helm chart should work for other Kubernetes distributions, it may require additional configurations applied to values.yaml.

Versioning and breaking changes

Important

Chart versions are SemVer-formatted, but they do not follow SemVer compatibility guarantees; a minor version bump can contain breaking changes.

The chart version mirrors the bundled Splunk OpenTelemetry Collector image's appVersion: a release starts at the same patch as the image, and the patch is then bumped for a chart change or to pick up an image patch update. Because of this, breaking changes to chart values, templates, and rendered manifests can ship in what looks like a minor bump (for example 0.149.0 → 0.150.0).

To upgrade safely:

• Pin to an exact chart version. Avoid ~ / ^ version constraints, and be careful with GitOps tooling that auto-bumps minor versions, as these can apply a breaking upgrade without review.
• Read the release notes before every upgrade. Check UPGRADING.md for migration steps and the CHANGELOG.md / GitHub releases for items marked as breaking.
• Read the Collector image release notes. Each chart version bundles a matching Splunk OpenTelemetry Collector image (the chart's appVersion). Image-level changes such as component behavior, metric or attribute renames, default runtime settings, and upstream opentelemetry-collector-contrib component updates can change the telemetry emitted by components you configure, even when your chart values are unchanged. Review the image release notes for the appVersion you are moving to (each CHANGELOG.md entry links the image release it adopts).
• Check bundled dependencies. The chart ships versioned subcharts and dependencies, such as the OpenTelemetry Operator (and its CRDs). Their upgrades can change operator behavior or injected auto-instrumentation even when your chart values are unchanged, so review their release notes if those features are enabled.
• Test the upgrade in a non-production environment first, and diff the rendered manifests (helm template / helm diff upgrade) to catch unexpected changes.

Getting Started

Prerequisites

The following prerequisites are required to use the helm chart:

• Helm 3.9+ or Helm 4.x
  • Helm version support may vary depending on the supported Kubernetes version you are using and the Helm supported Kubernetes versions skew.
  • This project is tested with Helm v3.20.0 and v4.1.1.
  • Please note that Helm 2 is not supported.
• Administrator access to your Kubernetes cluster and familiarity with your Kubernetes configuration. You must know where your log information is being collected in your Kubernetes deployment.

To send data to Splunk Enterprise or Splunk Cloud

• Splunk Enterprise 8.0 or later.

• A minimum of one Splunk platform index ready to collect the log data. This index will be used for ingesting logs.

• An HTTP Event Collector (HEC) token and endpoint. See the following topics for more information:

  • Set up and use HTTP Event Collector in Splunk Web
  • Scale HTTP Event Collector with distributed deployments

To send data to Splunk Observability Cloud

• Splunk Observability Cloud:
  • Splunk Access Token
  • Splunk Realm

Advanced Configuration

To fully configure the Helm chart, see the advanced configuration.

How to install

In order to install Splunk OpenTelemetry Collector in a Kubernetes cluster, at least one of the destinations (splunkPlatform or splunkObservability) has to be configured.

For Splunk Enterprise/Cloud the following parameters are required:

• splunkPlatform.endpoint: URL to a Splunk instance, e.g. "http://localhost:8088/services/collector"
• splunkPlatform.token: Splunk HTTP Event Collector token

For Splunk Observability Cloud the following parameters are required:

• splunkObservability.realm: Splunk realm to send telemetry data to.
• splunkObservability.accessToken: Your Splunk Observability org access token.

The following parameter is required or optional depending on the Kubernetes distribution:

• clusterName: arbitrary value that identifies your Kubernetes cluster. The value will be associated with every trace, metric and log as "k8s.cluster.name" attribute.
  • Optional: If distribution is set to EKS, EKS/fargate, GKE, and GKE/autopilot. If clusterName is specified it will overwrite detected value.
  • Required: For all other distributions.

Run the following commands, replacing the parameters above with their appropriate values.

Add Helm repo

helm repo add splunk-otel-collector-chart https://signalfx.github.io/splunk-otel-collector-chart

Sending data to Splunk Observability Cloud

helm install my-splunk-otel-collector --set="splunkObservability.realm=us0,splunkObservability.accessToken=xxxxxx,clusterName=my-cluster" splunk-otel-collector-chart/splunk-otel-collector

To enable Splunk Remote Management for the collectors installed by this chart, add remoteManagement.enabled=true to the same Helm install or upgrade command. See Remote Management for advanced configuration and operational details about the workload this enables:

helm install my-splunk-otel-collector --set="splunkObservability.realm=us0,splunkObservability.accessToken=xxxxxx,clusterName=my-cluster,remoteManagement.enabled=true" splunk-otel-collector-chart/splunk-otel-collector

Sending data to Splunk Enterprise or Splunk Cloud

helm install my-splunk-otel-collector --set="splunkPlatform.endpoint=https://127.0.0.1:8088/services/collector,splunkPlatform.token=xxxxxx,splunkPlatform.metricsIndex=k8s-metrics,splunkPlatform.index=main,clusterName=my-cluster" splunk-otel-collector-chart/splunk-otel-collector

Sending data to both Splunk Observability Cloud and Splunk Enterprise or Splunk Cloud

helm install my-splunk-otel-collector --set="splunkPlatform.endpoint=https://127.0.0.1:8088/services/collector,splunkPlatform.token=xxxxxx,splunkPlatform.metricsIndex=k8s-metrics,splunkPlatform.index=main,splunkObservability.realm=us0,splunkObservability.accessToken=xxxxxx,clusterName=my-cluster" splunk-otel-collector-chart/splunk-otel-collector

You can specify a namespace to deploy the chart to with the -n argument. Here is an example showing how to deploy in the otel namespace:

helm -n otel install my-splunk-otel-collector -f values.yaml splunk-otel-collector-chart/splunk-otel-collector

Instead of setting helm values as arguments a YAML file can be provided:

helm install my-splunk-otel-collector --values my_values.yaml splunk-otel-collector-chart/splunk-otel-collector

The examples directory contains examples of typical use cases with pre-rendered Kubernetes resource manifests for each example.

How to upgrade

Make sure you run helm repo update before you upgrade

Important

Before upgrading, review the versioning and breaking changes guidance, and the CHANGELOG.md.

To upgrade a deployment follow the instructions for installing but use upgrade instead of install, for example:

helm upgrade my-splunk-otel-collector --values my_values.yaml splunk-otel-collector-chart/splunk-otel-collector

How to uninstall

To uninstall/delete a deployment with name my-splunk-otel-collector:

helm uninstall my-splunk-otel-collector

Advanced Configuration

To fully configure the Helm chart, see the advanced configuration.

Auto-instrumentation

For setting up auto-instrumentation, see the auto-instrumentation-introduction.md.

Zero-Code Instrumentation with eBPF

In addition to the OpenTelemetry Operator for injection-based auto-instrumentation of language runtimes, OpenTelemetry eBPF Instrumentation can be deployed for eBPF-based zero-code instrumentation. See the zero-code documentation for more details.

Contributing

We welcome feedback and contributions from the community! Please see our (contribution guidelines) for more information on how to get involved.

License

Apache Software License version 2.0.

ℹ️  SignalFx was acquired by Splunk in October 2019. See Splunk SignalFx for more information.