Load balance high residual and seg fault with EB

[KZhu-ME]

When trying to use load balancing with the input file from #2538, with the changes from #2401, and adding return statements to the beginning of WarpX::ComputeEdgeLengths, WarpX::ComputeFaceAreas, WarpX::ScaleEdges, and WarpX::ScaleAreas, we get a very high residual (3e289) on step 4 (load_balance_intervals = 5), causing the run to fail and exit. If we set a high absolute tolerance for the MLMG solver (3e290) to bypass this high residual, the program will segfault at the beginning of step 5 with backtraces from some of the processes.

The input file we used is inputs2d.txt.

This is one backtrace file we got using a debug build, when running with the conditions above to reach the segfault on step 5. This simulation was run with 8 mpi cores and 1 omp thread. Backtrace.0.0.txt

[roelof-groenewald]

At least part of the problem is that phi_fp is not reset in WarpX::RemakeLevel. With the reset in we longer see the segfaults mentioned above. I will continue testing after fixing to see if there are other visible issues.

[roelof-groenewald]

There is definitely still a big problem with load balancing as can be seen in the attached figures of ion density.
(Note the plots below were generated with a different input script than the one attached above so don't be surprised by dimensions not agreeing)
Without load balancing on:

With load balancing on:

[ax3l]

attn @WeiqunZhang @lgiacome :)

[ax3l]

Is it possible that we forget to load balance a couple of EB-related MultiFabs?
https://github.com/ECP-WarpX/WarpX/blob/ed9a8edc1e6c07b7f6942c0c9b86ab8d2fda6690/Source/WarpX.H#L1039-L1051

For an example on where to add them for load balance, see #2555

[ax3l]

@WeiqunZhang @lgiacome can you please add inline Doxygen comments to these members on the way?

//! This stores the blabobb of the glubberish for EB, scaled by blimbub. Only used with labalabala
amrex::Vector<std::array< std::unique_ptr<amrex::MultiFab>, 3 > > Venl; 

This helps people understand what the 100s of member variables in WarpX.H are used for :)

[lgiacome]

@ax3l, I really don't know well this part of the code, but I think EB is being handled at the beginning of Warpx::RemakeLevel.
Is it correct?

https://github.com/ECP-WarpX/WarpX/blob/ee3fa1d0388af4ed43a578097cc86b0a6e0cbb21/Source/Parallelization/WarpXRegrid.cpp#L150-L159

[WeiqunZhang]

I think one needs to rebuild the mulitfabs before calling ComputeEdgeLengths etc., otherwise m_edge_lengths etc. are still in the old DistributionMapping.