[BUG] Pinned pool size setting may be missing from the autotuner recommendation

[kuhushukla]

Describe the bug
We do not always recommend the user to have a pinned pool size setting with an appropriate overhead memory setting.
spark.rapids.memory.pinnedPool.size . This was seen in on-prem environments.

We should verify if this setting is set appropriately with respect to other params as well. eg. spark.executor.memory and spark.executor.memoryOverhead

[mattahrens]

Pinned memory logic is in one of two places:

• https://github.com/NVIDIA/spark-rapids-tools/blob/dev/core/src/main/scala/com/nvidia/spark/rapids/tool/profiling/AutoTuner.scala#L518-L519
• https://github.com/NVIDIA/spark-rapids-tools/blob/dev/core/src/main/scala/com/nvidia/spark/rapids/tool/profiling/AutoTuner.scala#L518-L519

Can you give exact repro information where pinned pool wasn't being recommended? I wonder of cluster memory info was missing in input.

[parthosa]

We do not always recommend the user to have a pinned pool size setting with an appropriate overhead memory setting.
spark.rapids.memory.pinnedPool.size

I investigated this and was able to find some bugs that could have caused this:

Case 1:
@kuhushukla Do you remember if the recommended configs did not have an entry for spark.rapids.memory.pinnedPool.size but the comments had this line?

- 'spark.rapids.memory.pinnedPool.size' should be set to 2048m.

This is a bug in our code due to our incorrect cluster memory calculation when dynamic allocation is enabled. This should be fixed by my dev branch that address #1121

Case 2:
The AutoTuner comment had this line:

- 'spark.executor.memoryOverhead' must be set if using 'spark.rapids.memory.pinnedPool.size

This is a bug in our code where we compare a memory value string (4G) with the string spark.executor.memoryOverhead. We should also fix this.

Case 3:
Was the job using standalone mode? In that case recommending spark.executor.memoryOverhead is disabled as it is not supported.