set_global_default does not seem to work

[joseph-henry]

Bug Report

Version

│   └── tracing v0.1.41
│       ├── tracing-attributes v0.1.28 (proc-macro)
│       └── tracing-core v0.1.33
│   │   │   └── tracing v0.1.41 (*)
│   │       │   └── tracing v0.1.41 (*)
│   │       │   │   └── tracing v0.1.41 (*)
│   │       │   └── tracing v0.1.41 (*)
│   │       └── tracing v0.1.41 (*)
│   └── tracing v0.1.41 (*)
├── tracing v0.1.41 (*)
├── tracing-opentelemetry v0.28.0
│   ├── tracing v0.1.41 (*)
│   ├── tracing-core v0.1.33 (*)
│   ├── tracing-log v0.2.0
│   │   └── tracing-core v0.1.33 (*)
│   └── tracing-subscriber v0.3.19
│       └── tracing-core v0.1.33 (*)
└── tracing-subscriber v0.3.19 (*)

Platform

Linux 6.9.3-76060903-generic #202405300957~1738770968~22.04~d5f7c84 SMP PREEMPT_DYNAMIC Wed F x86_64 x86_64 x86_64 GNU/Linux

Crates

I'm not yet sure where the failure is. I suspect it's in tracing but here's my Cargo.toml:

tracing = { version = "0.1.35", default-features = false, features = ["std", "attributes"] }
opentelemetry = { version = "0.27.0", features = ["trace", "metrics"] }
opentelemetry_sdk = { version = "0.27.0", default-features = false, features = ["trace", "rt-tokio"] }
opentelemetry-stdout = { version = "0.27.0", features = ["trace", "metrics"] }
tracing-subscriber = { version = "0.3.0", default-features = false, features = ["registry", "std", "fmt"] }
tracing-opentelemetry = "0.28.0"
opentelemetry-otlp = "0.27.0"
opentelemetry-semantic-conventions = "0.27.0" # for service name
tokio = { version = "1.41.1", features = ["full"] 

Description

Hello. I'm trying to set the default subscriber for my entire program using set_global_default but am not getting the result that I expect. I can use set_default successfully, and I can use .with_default in a closure successfully.

When I use method 1 or 2 in the code below, my tracing events are sent to my collector, but when I use method 1 (set_global_default) nothing is sent to my collector and no error is emitted when I check its state. Any help would be greatly appreciated. Thanks!

use opentelemetry::trace::TracerProvider as _;
use opentelemetry::KeyValue;
use opentelemetry_otlp::WithExportConfig;
use opentelemetry_sdk::trace::TracerProvider;
use opentelemetry_sdk::{runtime, Resource};
use opentelemetry_semantic_conventions::resource::SERVICE_NAME;
use tracing::{error, info, span, trace};
use tracing_subscriber::layer::SubscriberExt;
use tracing_subscriber::Registry;

#[tokio::main]
async fn main() {
    let exporter = opentelemetry_otlp::SpanExporter::builder()
        .with_tonic()
        .with_endpoint("http://localhost:4317")
        .build()
        .unwrap();

    let provider = TracerProvider::builder()
        .with_batch_exporter(exporter, runtime::Tokio)
        .with_resource(Resource::new(vec![KeyValue::new(
            SERVICE_NAME,
            "test-service",
        )]))
        .build();

    let tracer = provider.tracer("global_tracer");
    let telemetry = tracing_opentelemetry::layer().with_tracer(tracer.clone());
    let subscriber = Registry::default().with(telemetry);

    // Method 1: Set subscribe as global default (Doesn't work)
    tracing::subscriber::set_global_default(subscriber).expect("setting default subscriber failed");
    let root = span!(tracing::Level::TRACE, "app_start", work_units = 2);
    let _enter = root.enter();
    trace!("This is a trace event.");
    error!("This is an error event.");
    info!("This is an info event.");

    // Method 2: Set subscriber as default (Works)
    /*
    let _guard = tracing::subscriber::set_default(subscriber);
    let root = span!(tracing::Level::TRACE, "app_start", work_units = 2);
    let _enter = root.enter();
    trace!("This is a trace event.");
    error!("This is an error event.");
    info!("This is an info event.");
    */

    // Method 3: Trace executed code inside of a closure (Works)
    /*
    tracing::subscriber::with_default(subscriber, || {
        // Spans will be sent to the configured OpenTelemetry exporter
        let root = span!(tracing::Level::TRACE, "app_start", work_units = 2);
        let _enter = root.enter();
        trace!("This is a trace event");
    });
    */

    let current_span = tracing::Span::current();
    println!("Current span: {:?}", current_span);
}

Thanks!

[joseph-henry]

It seems like this example although different from what I'm trying to do does demonstrate that set_global_default can work, but why would the subscriber make any difference here? The subscriber clearly works in my original example when I use set_default.

use tracing::{info, Level};
use tracing_subscriber::FmtSubscriber;

fn main() {
    // a builder for `FmtSubscriber`.
    let subscriber = FmtSubscriber::builder()
        // all spans/events with a level higher than TRACE (e.g, debug, info, warn, etc.)
        // will be written to stdout.
        .with_max_level(Level::TRACE)
        // completes the builder.
        .finish();

    tracing::subscriber::set_global_default(subscriber)
        .expect("setting default subscriber failed");

    let number_of_yaks = 3;
    // this creates a new event, outside of any spans.
    info!(number_of_yaks, "preparing to shave yaks");

    let number_shaved = yak_shave::shave_all(number_of_yaks);
    info!(
        all_yaks_shaved = number_shaved == number_of_yaks,
        "yak shaving completed."
    );
}

[kaffarell]

Just skimming quickly over open-telemetry/opentelemetry-rust#1961 it seems that this is known and expected.
set_global_default has been removed from any examples but maybe it would be nice to include a big comment on top of the tracing_opentelemetry::layer::OpenTelemetryLayer struct.

In any case, this is not a tracing issue.

[cijothomas]

@joseph-henry It looks like you are missing shutdown call on tracer_provider. Make an explicit shutdown() call, and this should work.

As to why this works when you use with_default and not when you use global (simplified)

1. Shutdown is automatically invoked when last reference of provider is dropped
2. This means shutdown would occur automatically at end of program as provider will be dropped at that point.
3. But when you use global_ - the provider is help on by tracing::subscriber and not dropped and hence not shutdown.
4. When you use with_default, provider is dropped at the end of the scope and end of program drops the last ref also, leading to shutdown being triggered.

In short, explicit shutdown should do the trick.

[cijothomas]

In any case, this is not a tracing issue.

Yes. This issue can be closed here.

[joseph-henry]

Thanks for the help but unfortunately adding the following did not help:

if let Err(err) = provider.shutdown() {
    eprintln!("Provider shutdown error: {:?}", err);
}

I can agree to close the issue soon since this isn't a tracing issue but if there's anything else I should try I'd love to know what that could be. Thanks.

[cijothomas]

@joseph-henry Check if you are following the examples from https://github.com/open-telemetry/opentelemetry-rust/tree/main/opentelemetry-otlp/examples

If yes and having issues, it's better to report it and discuss on the OpenTelemetry repo itself.

[joseph-henry]

I'll do just that. Thanks again guys for the assistance.