Fix spark.kubernetes.file.upload.path permission

[pan3793]

Why are the changes needed?

The default behavior of HDFS is to set the permission of a file created with FileSystem.create or FileSystem.mkdirs to (P & ^umask), where P is the permission in the API call and umask is a system value set by fs.permissions.umask-mode and defaults to 0022. This means, with default settings, any mkdirs call can have at most 755 permissions.

The same issue also got reported in SPARK-30860

How was this patch tested?

Manual test.

Was this patch authored or co-authored using generative AI tooling?

No.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 0% with 3 lines in your changes missing coverage. Please review.

Project coverage is 0.00%. Comparing base (a3f1e51) to head (7527060).

Files with missing lines	Patch %	Lines
...ache/kyuubi/engine/spark/SparkProcessBuilder.scala	0.00%	3 Missing ⚠️

Additional details and impacted files

@@          Coverage Diff           @@
##           master   #7148   +/-   ##
======================================
  Coverage    0.00%   0.00%           
======================================
  Files         701     701           
  Lines       43500   43501    +1     
  Branches     5895    5895           
======================================
- Misses      43500   43501    +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.