[FEA] we should support hdfs federation

[nvliyuan]

Currently we only support hdfs path: https://github.com/NVIDIA/spark-rapids-tools/blob/3db52ef3a4c56abce9d1d3337daa05ae176d9622/user_tools/src/spark_rapids_tools/storagelib/hdfs/hdfspath.py

spark-rapids-tools/user_tools/src/spark_rapids_tools/cmdli/argprocessor.py

Line 174 in 3db52ef

def detect_platform_from_eventlogs_prefix(self):

But some customer use hdfs federation, it would be nice if we could support viewfs://

[amahussein]

Thanks @nvliyuan !
We are actually in the process of switching the workflow a little bit in the CLI.
The --platform is required. The reason to do that is that we found that auto-detecting the platform based on eventlogpath is not always accurate.
To support viewfs requires the following:

Core-tools

• Investigate how to support the federation in Spark configuration/properties and how to set it to the java cmd.

user-tools

Our cross-platform storage uses pyarrow.filesystem. Fortunatley, it supports viewfs.
So we need to:

• In general: we need to adds that scheme to the HDFS interface in our python.
• For distributed mode: we need to make sure that the distributed-mode can be configured to use viewfs (CC: @parthosa )
• How to set the java CLI with the correct configurations.

[parthosa]

@amahussein The distributed-mode uses our cross-platform storage for all file access. As you mentioned that pyarrow supports viewfs, it should simplify things for distributed-mode as well.