[execution] Remove deprecated disk manager configuration API

[megakaizo]

Which issue does this PR close?

• Part of Remove old deprecated code #23080

Rationale for this change

DiskManagerConfig and DiskManager::try_new were deprecated in version 48.0.0 in favor of the new DiskManagerBuilder. This PR remove this deprecated methods and update Runtime env initialization logic

What changes are included in this PR?

• Removed RuntimeEnvBuilder::with_disk_manager depercated method.
• Removed deprecated DiskManagerConfig enum and its associated constructor methods.
• Removed deprecated DiskManager::try_new method.
• Refactored RuntimeEnvBuilder to store Option<Arc<DiskManager>> instead of the old configuration enum.
• Update RuntimeEnvBuilder::build logic by using matching to handle the existing manager, builder, or default initialization.

Are these changes tested?

verifed by running local tests

Are there any user-facing changes?

Yes. This removes the deprecated public Rust APIs RuntimeEnvBuilder::with_disk_manager, DiskManagerConfig and DiskManager::try_new. Downstream users who need to configure the disk manager should migrate to using DiskManager::builder.

This is an API change and should be labeled as api change.

[github-actions]

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion-execution v54.0.0 (current)
       Built [  32.351s] (current)
     Parsing datafusion-execution v54.0.0 (current)
      Parsed [   0.026s] (current)
    Building datafusion-execution v54.0.0 (baseline)
       Built [  28.267s] (baseline)
     Parsing datafusion-execution v54.0.0 (baseline)
      Parsed [   0.025s] (baseline)
    Checking datafusion-execution v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.327s] 223 checks: 221 pass, 2 fail, 0 warn, 30 skip

--- failure enum_missing: pub enum removed or renamed ---

Description:
A publicly-visible enum cannot be imported by its prior path. A `pub use` may have been removed, or the enum itself may have been renamed or removed entirely.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/enum_missing.ron

Failed in:
  enum datafusion_execution::disk_manager::DiskManagerConfig, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/b8fa4006b8f4d1f29613f6541c5cb62a3d3543b5/datafusion/execution/src/disk_manager.rs:125

--- failure inherent_method_missing: pub method removed or renamed ---

Description:
A publicly-visible method or associated fn is no longer available under its prior name. It may have been renamed or removed entirely.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/inherent_method_missing.ron

Failed in:
  DiskManager::try_new, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/b8fa4006b8f4d1f29613f6541c5cb62a3d3543b5/datafusion/execution/src/disk_manager.rs:198
  DiskManager::try_new, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/b8fa4006b8f4d1f29613f6541c5cb62a3d3543b5/datafusion/execution/src/disk_manager.rs:198
  RuntimeEnvBuilder::with_disk_manager, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/b8fa4006b8f4d1f29613f6541c5cb62a3d3543b5/datafusion/execution/src/runtime_env.rs:377

     Summary semver requires new major version: 2 major and 0 minor checks failed
    Finished [  62.444s] datafusion-execution

[Dodothereal]

@alamb (and any other reviewer) — another #23080 partial removal: DiskManagerConfig and DiskManager::try_new from the execution crate (deprecated since 48.0.0, replaced by DiskManagerBuilder). 75-line deletion in disk_manager.rs + callsite updates in runtime_env.rs. All 35 CI checks (cargo test amd64/macos-aarch64, clippy, rustfmt, license-header, semver, MSRV, sqllogictest, codeql) are green. Happy to address any feedback.

[2010YOUY01]

LGTM, thank you!