Potential race condition in axom::copy when Umpire is enabled.

[BradWhitlock]

In the new heavily_mixed MIR example, I was calling axom::copy() in an axom::for_all<axom::OMP_EXEC>() loop to copy some data into a slice of a 3D array. This was to speed up material construction on 3D meshes since I could have many 2D slices constructed in parallel.

I noticed that the test for heavily_mixed would crash intermittently on dane. Basically, the first time axom::copy is called, the Umpire resource manager is constructed as a singleton. Since axom::copy() was being called on multiple OpenMP threads at the same time, there was a race condition in creating the resource manager that would sometimes cause it to throw exceptions and terminate the program.

I worked around the problem in the heavily_mixed example by calling axom::copy() prior to the OpenMP loop. That makes sure the Umpire resource manager exists by the time the loop is called.

OpenMP loops like this are probably rare but is there something more general that should be done to avoid the problem?

• Add a mutex into axom::copy around where the resource manager instance is retrieved?

[rhornung67]

@BradWhitlock my gut tells me that the test is tickling a case that would be rare in a real code/problem setting, which would most likely have created the Umpire resource manager in serial section someplace early (like accessing allocators).

Would it be better to call the Umpire method to get the resource before anything else is done in your test code. Calling axom::copy() seems cryptic to me.

[BradWhitlock]

@rhornung67 - I agree this is probably rare. Calling the Umpire function to make the resource manager is a more explicit workaround. Still, I ran into it because Umpire is not front and center in Axom so I did not realize I might run into problems calling axom::copy from an OpenMP loop. I thought I'd note the issue. I could also switch to memcpy() in this instance.

[rhornung67]

We should document this and leave it unsafe.