If we pack a multi-node submission with test cases that require fewer than one node's resources (particularly GPU resources), how should the visible devices be managed? Can we do this? Is it even desirable?
Work with experts (e.g., Kokkos) to better understand the problem and solution space. Implement a solution, or provide configuration guidance for appropriately batching tests in a multi-node environment if it doesn't make sense to manage. Put in guardrails to prevent inadvertent misuse or having things behave unexpectedly.
Update this issue description when time permits for greater clarity/specificity.
If we pack a multi-node submission with test cases that require fewer than one node's resources (particularly GPU resources), how should the visible devices be managed? Can we do this? Is it even desirable?
Work with experts (e.g., Kokkos) to better understand the problem and solution space. Implement a solution, or provide configuration guidance for appropriately batching tests in a multi-node environment if it doesn't make sense to manage. Put in guardrails to prevent inadvertent misuse or having things behave unexpectedly.
Update this issue description when time permits for greater clarity/specificity.