feat(data-pipeline)!: add flush_and_close to the trace buffer (#2313)

# What does this PR do?

Add `TraceBuffer::flush_and_shutdown(timeout)`, a synchronous method to flush the trace buffer and reject further writes.

# Motivation

dd-trace-py stops and joins its background export thread before every fork. The trace buffer had no equivalent operation: force_flush only queues a flush request, and `TraceBuffer::new()` gives the caller a `Sender`, not a `WorkerHandle` or the `SharedRuntime`. Without `flush_and_shutdown`, dd-trace-py could not get a bounded, synchronous guarantee that buffered chunks reach the exporter and that no new chunk can arrive before a fork.

`flush_and_shutdown` triggers a flush of the current batch, blocks until the exporter exports that batch or timeout elapses, then marks the channel shut down. The channel shuts down even on timeout, so a caller that must tear down before a fork always gets the "no further chunks" guarantee, whether or not the flush finished in time.

This also fixes a separate drop bug in `Worker::shutdown`: `PausableWorker::pause`'s biased `select!` cancels the trigger-and-run loop without letting it hand off a pending batch, so `Worker::shutdown` used to ack without exporting. It now drains and exports the pending batch before it acks.

# Additional Notes

### Changes

- `Sender::trigger_flush` returns the `BatchGeneration` of the batch it flushes, so a caller can wait for that exact batch.
- `Sender::flush_and_shutdown` / `TraceBuffer::flush_and_shutdown(timeout)`: flush, wait, then shut down. Shuts down even when the flush times out.
- `Receiver::drain_pending`: take whatever sits in the batch, regardless of flush_needed. `Worker::shutdown` calls this before it acks, so a paused worker no longer drops its last batch.
- `Waiter::mark_shutdown` and `TraceExporterWorker::export_and_ack`: extracted from duplicated code paths.

# How to test the change?

- New tests cover: `flush_and_shutdown` exporting a pending chunk, marking the channel shut down even when the flush times out, and returning `AlreadyShutdown` on a second call.
- `cargo test -p libdd-data-pipeline trace_buffer`, `cargo +stable clippy -p libdd-data-pipeline --all-targets -- -D warnings`, and `cargo +nightly-2026-07-26 fmt --all -- --check all` pass.


Co-authored-by: brett.langdon <brett.langdon@datadoghq.com>

Author: brettlangdon