Make Adami loop flipping optimization optional

[efaulhaber]

Depends on #498.

I now get an error of 0.0 for the dam break validation between a single threads and 128 threads on an AMD Threadripper 3990X. So we can generate all validation files on as many threads as we have available.

[codecov]

Codecov Report

Attention: Patch coverage is 57.14286% with 3 lines in your changes missing coverage. Please review.

Project coverage is 68.50%. Comparing base (f051b55) to head (10df125).
Report is 5 commits behind head on main.

Files with missing lines	Patch %	Lines
...chemes/boundary/dummy_particles/dummy_particles.jl	57.14%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #581      +/-   ##
==========================================
- Coverage   68.51%   68.50%   -0.02%     
==========================================
  Files          87       87              
  Lines        5406     5407       +1     
==========================================
  Hits         3704     3704              
- Misses       1702     1703       +1     

Flag	Coverage Δ
unit	68.50% <57.14%> (-0.02%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[efaulhaber]

@svchb do you remember what we did with this? Did we implement it somewhere else? Or did we just forget about it?

[svchb]

The optimization is there because there are cases where the larger system is looped over and there are no/only a small number of neighbors between systems. In my simulations this made up to a 20% difference in run time.

One side effect is though that if you run the validation cases with different numbers of threads the results change.

[efaulhaber]

Yes, I know. This PR is based on your PR, so the diff is messy. But did we make the flipping optional and disable it for validiation? Or did we forget about it? Or did we decide not to do this for some reason?

[efaulhaber]

Just rebased to make the diff useful.

[svchb]

No this was not merged. You also did not create all the validation files correctly. I think this was the latest on this PR.