DEBUG-3573 send AppClientConfigurationChange telemetry event when subsequent telemetry instances start

[p-datadog]

What does this PR do?

When tracer is reconfigured (via Datadog.configure), and telemetry is enabled in both previous and new configurations, telemetry will now emit AppClientConfigurationChange from the second instance when previously no event was emitted.

Motivation:

Previously, only AppStarted event was emitted for programmatic and environment configuration, and only the first time the tracer was configured. This caused configuration state to not be sent to the backend when Datadog.configure was used to enable a product (for example, dynamic instrumentation).

Change log entry
Yes: improve dynamic instrumentation UI reporting of application and host status

Additional Notes:

AppClientConfigurationChange currently sends the complete configuration (same as what AppStarted would have sent). Ideally it should only send changed parameters.

This PR does not change behavior of tracer when telemetry is turned on or off. For example, turning telemetry off would NOT presently produce AppClosing event (assuming tracer is reconfigured rather than shut down).

Besides AppClientConfigurationChange, there is also AppIntegrationsChange. This event is not changed by this PR, it is still not being sent on second and subsequent telemetry initializations. It probably should be sent in theory but there is no immediate demand for it. (And it should also send changed state, not all state.)

This PR also makes two fixes to telemetry:

1. Previously, submitting events to telemetry would discard them if the initial event (used to be AppStarted, now it's either AppStarted or AppClientConfigurationChange) hasn't been sent yet. Now the submitted events are added to the queue and will be sent out after the initial event. The queue is (and was) already bounded. Since the initial event can fail and will be retried, there is a window of what looks to be 50 seconds where events could have been getting discarded).

2. The initial event is sent when the worker is performing sending of some other event. Previously, If initial event fails to send, the other event would be dropped. Now the dequeued event is put back into the queue. (Currently the dequeued event is added to the end of the queue which is not the correct place for it, to avoid shifting all other events down one.)

How to test the change?
Integration tests have been added that verify the AppClientConfigurationChange is being sent when dynamic instrumentation is enabled programmatically along with several other smaller scope tests.

The change is also tested manually.

[github-actions]

Thank you for updating Change log entry section 👏

^{Visited at: 2025-05-27 21:11:12 UTC}

[datadog-datadog-prod-us1]

Datadog Report

Branch report: telemetry-lifecycle
Commit report: 0023b94
Test service: dd-trace-rb

✅ 0 Failed, 21706 Passed, 1299 Skipped, 4m 28.29s Total Time

[pr-commenter]

Benchmarks

Benchmark execution time: 2025-05-27 22:03:46

Comparing candidate commit 0023b94 in PR branch telemetry-lifecycle with baseline commit ff8ae5b in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 42 metrics, 3 unstable metrics.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 99.63899% with 1 line in your changes missing coverage. Please review.

Project coverage is 97.64%. Comparing base (ff8ae5b) to head (0023b94).

Files with missing lines	Patch %	Lines
spec/datadog/core/telemetry/worker_spec.rb	94.44%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4678      +/-   ##
==========================================
- Coverage   97.64%   97.64%   -0.01%     
==========================================
  Files        1470     1474       +4     
  Lines       87685    87829     +144     
  Branches     4544     4552       +8     
==========================================
+ Hits        85624    85760     +136     
- Misses       2061     2069       +8     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Strech]

The change is a very good one, but I think we should consider moving it into the components tree instead of introducing new abstraction somewhere else.

[Strech]

I think we need to convert it into the boolean, otherwise we may return nil

irb(main):001> nil&.started?
=> nil

Suggested change

	@remote_started = components.remote&.started?
	@remote_started = !!components.remote&.started?
	``

[Strech]

I have a question about design decision here.

I see the ease of use of ComponentsState as a separate abstraction which is able to analyze some details about the components tree, but why it's not a part of the components tree itself?

A good example of the current design flaw is this part

          telemetry.emit_closing! unless replacement&.telemetry&.enabled

when inside the components itself we have to use directly some internals of the other components tree, instead of asking the components tree itself (violation of encapsulation), like this:

          telemetry.emit_closing! unless replacement.telemetry_enabled?

IMHO we should incorporate the state into components tree and ask questions, instead of introducing the missing API somewhere else.