Enabling tracing by usage of opentelemetry

[rem1-dev]

Enabling tracing by usage of opentelemetry

[rem1-dev]

@shesek this is same as PR #99 but applied to fresh branch created out of new-index. Commits were squashed.

[rem1-dev]

I'm working on solving those conflict files

[rem1-dev]

@shesek conflicts resolved, it's ready to be merged.

[philippem]

@rem1-dev when testing locally we ran into some errors

tempo-1           | level=warn ts=2024-11-13T20:09:28.400374334Z caller=instance.go:42 msg="TRACE_TOO_LARGE: max size of trace (5000000) exceeded while adding 185917 bytes to trace 6bafc5e34f7227d985e36a2142d95fb6 for tenant single-tenant"
tempo-1           | level=warn ts=2024-11-13T20:09:28.426261953Z caller=instance.go:42 msg="TRACE_TOO_LARGE: max size of trace (5000000) exceeded while adding 185912 bytes to trace 6bafc5e34f7227d985e36a2142d95fb6 for tenant single-tenant"

which may be related to cardinality or size of the spans.

Possible causes:

- Large Trace Data: The trace contains too many spans or overly large data (such as logs or metadata).
- High Cardinality: Excessive number of unique tags, attributes, or data points being attached to spans, leading to large trace size.
- Excessive Span Detail: Individual spans may contain large payloads or detailed information that increase the trace size.
- Improper Sampling Rate: Sampling configuration might be too high, causing too many traces to be captured, some of which may exceed the size limit.
- Large Payloads in Spans: If spans contain large amounts of data (e.g., large JSON bodies, files, or detailed logs), it could push the trace size over the limit.
- Too Many Traces Collected: An overly aggressive tracing configuration that collects an excessive number of traces, increasing the likelihood of exceeding the size limit.
- Long or Complex Transactions: If a trace represents a very large or complex operation (e.g., a multi-step workflow or a distributed operation), it might naturally grow too large.

[rem1-dev]

@philippem how can I reproduce this locally?

[philippem]

@shesek what do you think about adding a new macro to reduce the amount of copy-paste? will require new module

[rem1-dev]

I provided a macro that wraps long #[tracing::instrument(args...)] macro call into simplified #[instrumented] call with optional capturing of instrumented function arguments. @philippem @shesek please take a look.

[shesek]

Looks good! I like the new macro :)

Now that we have it, perhaps we could also implement my suggestion from the prior PR to expand #[trace] into a no-op when tracing is disabled?

That way we won't need the negative no-otlp-tracing feature, can remove the manual compile_errors for invalid feature combos, and avoid the dependency on the tracing crate when tracing is disabled.