add document defining an OpenTelemetry Collector #4313
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,64 @@ | ||
| <!--- Hugo front matter used to generate the website version of this page: | ||
| path_base_for_github_subdir: | ||
| from: tmp/otel/specification/collector/_index.md | ||
| to: collector/README.md | ||
| ---> | ||
|
|
||
| # OpenTelemetry Collector | ||
codeboten marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| **Status**: [Development](../document-status.md) | ||
|
|
||
| The goal of this document is for users to be able to easily switch between | ||
| OpenTelemetry Collector Distros while also ensuring that components produced by | ||
| the OpenTelemetry Collector SIG are able to work with any vendor who claims | ||
| support for an OpenTelemetry Collector. | ||
|
Comment on lines
+11
to
+14
There was a problem hiding this comment. I'm not sure I understand this goal. If a vendor produces a collector distribution that has a subset of available components because those are the components relevant to their service offerings and that they're willing to support, where do any other components (whether hosted in an OTel repo or not) fit into that picture? Do we mean that a distribution must offer end users the ability to modify its source and create their own build? We should be explicit about that if that is the case. Given that the licensing of the collector's source code does not require that distribution of derivative works happen in source form I'm not sure that we have much ability here to enforce such a requirement. We can certainly try to use the "OpenTelemetry" mark as a cudgel, but I'm not sure it'll be as effective as may be desirable since the terms "collector" and "distribution" are very broad. It could perhaps be argued that "OpenTelemetry Collector" is a protectable mark and maybe even that "Collector" has acquired secondary meaning in this limited scope, but protecting such a mark against genericization is going to be a Sisyphean task. There was a problem hiding this comment. I see this definition as separation from the term Something is not an OpenTelemetry Collector if it cannot support OpenTelemetry Collector Components. Maybe the word There was a problem hiding this comment.
We potentially have leverage over:
I think we have enough leverage here to make this worth it There was a problem hiding this comment. I'm not sure that having a registered "OpenTelemetry Collector" mark is sufficient here as nominative fair use would allow anyone preparing a distribution (in the colloquial sense, not Obviously the project can control what it puts on its website and what marketing collateral is used in conjunction with events operated by LF/CNCF, but that doesn't seem like effective leverage over an actor who has no need or interest in such things. |
||
|
|
||
| - An OpenTelemetry Collector _MUST_ accept an [OpenTelemetry Collector configuration | ||
| file](#opentelemetry-collector-configuration-file). | ||
| - An OpenTelemetry Collector _MUST_ be able to include additional compatible | ||
| [Collector components](#opentelemetry-collector-components) that | ||
| the user wishes to include. | ||
|
|
||
| ## OpenTelemetry Collector configuration file | ||
jpkrohling marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| An OpenTelemetry Collector configuration file is defined as YAML and _MUST_ support | ||
| the following [minimum structure](https://pkg.go.dev/go.opentelemetry.io/collector/otelcol#Config): | ||
|
|
||
| ```yaml | ||
| receivers: | ||
| processors: | ||
| exporters: | ||
| connectors: | ||
| extensions: | ||
| service: | ||
codeboten marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| telemetry: | ||
| pipelines: | ||
| ``` | ||
|
|
||
| ## OpenTelemetry Collector components | ||
|
|
||
| For a library to be considered an OpenTelemetry Collector component, it _MUST_ | ||
| implement a [Component interface](https://pkg.go.dev/go.opentelemetry.io/collector/component#Component) | ||
| defined by the OpenTelemetry Collector SIG. | ||
|
|
||
| Components require a unique identfier as a `type` string to be included in an OpenTelemetry | ||
| Collector. It is possible that multiple components use the same identifier, in which | ||
| case the two components cannot be used simultaneously in a single OpenTelemetry Collector. In | ||
| order to resolve this, the clashing components must use a different identifier. | ||
codeboten marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| ### Compatibility requirements | ||
|
|
||
| A component is defined as compatible with an OpenTelemetry Collector when its dependencies are | ||
| source- and version-compatible with the Component interfaces of that Collector. | ||
|
|
||
| For example, a Collector derived from version tag v0.100.0 of the [OpenTelemetry Collector](https://github.com/open-telemetry/opentelemetry-collector) _MUST_ support all components that | ||
| are version-compatible with the Golang Component API defined in the `github.com/open-telemetry/opentelemetry-collector/component` module found in that repository for that version tag. | ||
|
|
||
codeboten marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| ## OpenTelemetry Collector Distribution | ||
codeboten marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| An OpenTelemetry Collector Distribution (Distro) is a compiled instance | ||
| of an OpenTelemetry Collector with a specific set of components and features. A | ||
| Distribution author _MUST_ provide users with tools and/or documentation for adding | ||
| their own components to the Distribution's components. Note that the resulting | ||
|
Comment on lines
+60
to
+62
There was a problem hiding this comment. This seems problematic to me as a I think if we want an identifier for compatible distributions that can be effectively controlled we will need a distinctive mark for a compatibility certification that can be granted to distributions that satisfy its requirements, similar to what @tedsuo seems to be describing here. Beyond those concerns, this requirement also seems excessively vague. What qualifies as "tools and/or documentation"? Is a link to https://go.dev/dl/ sufficient? This probably requires a definition similar to "Corresponding Source" from AGPL-3, which again reinforces the limitations that come from not having this be part of the license under which the collector source code is made available. There was a problem hiding this comment. The spirit here is to allow users to reuse their components when they move from one distribution to another: the engineering investments made should not be lost. If the distribution is open source and there's clear documentation how to add a new component to it, that's good enough for me. If the distribution is not open source but allows me to enter the Go module name on a web interface somewhere and get a binary out, that's also fine. I'd see that binary as "tainted" (to use the kernel terminology) and the final binary might not be officially supported (with SLAs) by a service provider, but as an end-user, I'm not locked in. There was a problem hiding this comment. This is a specification, so I don't think it's appropriate to leave ambiguity here and rely on interpretation of the "spirit" of the requirement. This was changed from
I would not expect, and do not think it reasonable to expect, that any vendor offering a closed-source, binary-only distribution will allow users to provide arbitrary code to be built into a new "tainted" binary by that vendor. Doing so would allow for a user to cause a vendor to distribute binaries built from code licensed under terms the vendor has no opportunity to review and which may require, for instance, that any code it is compiled with be distributed under the same terms. |
||
| binary from updating a Distribution to include new components | ||
| is a different Distribution. | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OpenTelemetry Collector is never defined. Is it a source code artifact? A binary?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's one of the things I was trying to get at here. Since there's no binary plugin mechanism it seems that the source would need to be available for it to be extended in the manner contemplated, but that's not clear or explicit in the current state.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is the lack of binary plugin mechanism something that the OpenTelemetry Collector SIG wants to solve? Are there technical blockers?
Binary and dynamic loading plugin seem to be an established pattern. For example:
https://github.com/fluent/fluent-bit-go
https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/yaml/plugins-section
While Fluent Bit comes with a variety of built-in plugins, it also supports loading external plugins at runtime. This feature is especially useful for loading Go or Wasm plugins that are built as shared object files (.so).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The Fluent Bit example is not necessarily apposite as it involves a C application dynamically loading shared libraries built with the Go toolchain using CGO (which is generally prohibited in the Collector codebase).
Go does have a native plugin mechanism, though it comes with many caveats and is widely regarded as a bad idea that can't be dropped due to compatibility guarantees. Its documentation sums up its litany of restrictions in this way, which sounds a lot like a suggestion to use something like
ocb: