Best practices for ensuring proper grid nesting?

[desjardi]

As I'm pushing towards more refinement levels in my amrcore-based simulations (say 8 or 9 levels), I'm finding that regridding is increasingly likely to generate improperly nested grids. My blocking_factor=8 with nghost=2 satisfies the formula from #2140. I've tried playing around with some parameters (specifically n_error_buf), but I haven't had much success (i.e., increasing n_error_buf delays but doesn't solver the problem). I also notice that MakeNewGrid's nesting check is compiled out - is it supposed to be guaranteed by design? My intuition tells me that it's a question of error tagging: there may be implicit requirements in how we tag cells for refinement that are not explicitly enforced by AMReX and left for the user to address. However, I'm not certain what these are.

What are the best practices for ensuring proper nesting in deep hierarchies? Are there parameter choices that help, or is it common practice to ensure consistency across levels in the tagging prior to feeding tags to AMReX?

[WeiqunZhang]

Have you tried amr.n_proper? https://amrex-codes.github.io/amrex/docs_html/RuntimeParameters.html?highlight=n_proper#amr.n_proper

[WeiqunZhang]

As for improperly nested grids, is it improper in FillPatch functions or somewhere else? If it's in FillPatch, there is a function for checking.
https://amrex-codes.github.io/amrex/doxygen/namespaceamrex.html#a36b10976708081700c810d70417da8c9

[desjardi]

I had not tried amr.n_proper. I just gave it a quick try in 2D, and it doesn't seem to really impact my grids? With blocking_ratio=8 and rref=2, should I expect to have always 8 coarse cells of buffer around my finer grids? Or is not visual change expected?

And yeah, it's improper in fillpatch - we've used the function you pointed out and it confirms it. The question is what's the proper knob to turn to address this, or alternatively, is there a way to design my error tagging code to ensure a safe behavior?

[WeiqunZhang]

You should expect >= 4 coarse cells.

[WeiqunZhang]

If you want 8, you need to set blocking factor to 16 or n_proper to 2.

[desjardi]

Okay. To clarify:

• with blocking_factor=8, rref=2, and n_proper=1: I should see 4 coarse cells
• with blocking_factor=8, rref=2, and n_proper=2: I should see 8 coarse cells
  Is that correct?
  Can you help me understand what conditions can create improper nesting though? My main production run has been blocking_factor=8, rref=2, and n_proper=1, and I'm getting improper nesting - that means that "should see" does not mean "guarantied to always get", right? That gap is what I'd like to understand.

[WeiqunZhang]

Yes, they are correct. If you do not get 4 coarse cells with blocking_factor=8, rref=2, and n_proper=1, then that is not expected behavior. If you can provide a reproducer, I can take a look.

[desjardi]

Okay, I greatly appreciate it. We have now a few different cases where remeshing via MakeNewGrids deterministically generates non-nested grid. The issue is that it's on 8-9 levels... Smallest case is probably about 800 millions cells. All run with blocking_factor=8, rref=2, and n_proper=1. We tend to see little corners of the finest level sticking out of the level below, along the diagonal directions (image attached below). We are trying to tag thin features (shocks and liquid-gas interfaces), so maybe that's the problem? Happy to send all the details, but I'm not sure what's the best way to give you the necessary data. I imagine that only the ba's matter, but we also would need to give you the tags? Or you're welcome to run our code directly from our checkpoint? Or if you prefer, you could tell me what to do/check in amrex.

[asalmgren]

Another question -- do you know whether this case (where the proper nesting is violated) happens when only the green level is being re-created, or are multiple levels being re-created during this step?

[desjardi]

Thanks, I appreciate it.

[WeiqunZhang]

I started working on it and realized BoxArray has some internal meta-data that might be hard to dump to the disk. That could well be the reason why the issue exists. So the most reliable way for me to reproduce what you see is for me to run your code from your checkpoint. As for how to obtain your data, you can contact me by email. Thanks!

[desjardi]

Okay, sounds good - thanks again for your help with this, we've been stuck trying to understand what's going on for a while now!
The code is here: https://github.com/desjardi/NGA2/tree/develop. Apologies in advance, but it's fortran :)
I'll contact you asap to provide you with the restart file and the input, and I can provide assistance to compile the code as well if needed.

[WeiqunZhang]

Are you on NERSC, OLCF or ALCF machines?

[desjardi]

No, sorry, all on DoD machines (and some NSF-access too I guess)

[WeiqunZhang]

I think I have found the issue. It's a bug in amrex. A few years ago, we were trying to be more efficient, but unfortunately I broke it. When a fine level lev grows rapidly, the level lev-2 needs to be buffered wide enough to properly nest level lev-1, which in turn needs to properly nest level lev. But there was some an error in how we do it. I will try to fix it ASAP.

Thanks for reporting the issue! We have not noticed it before because applications usually do regridding every two steps on every coarse level. But with regridding every ten level 0 steps and a deep hierarchy, it's more likely to have the issue.

[WeiqunZhang]

#5267

[chaselee42]

The tagboxarray_ngt branch has mostly fixed the regrid issues @desjardi and I were experiencing after running a few jobs over the weekend!

I also saw that you made a new branch (regrid_proj) that also fixed the number of ghost cells that were added to TagBoxArray. Do you recall if there were any other bug fixes or differences between the tagboxarray_ngt and regrid_proj branches?

[WeiqunZhang]

The two branches are fixing the same bug with different approaches. We are going to go with the approach in #5269, because it's more memory efficient. They do not have any other differences.

[desjardi]

Thanks so much Weiqun! It looks like this fixes our issue - our tests have been running just fine with the modification.

[desjardi]

So I'm not fully sure I understand the specific details of your question. However, I'm doing interface capturing on the finest level only, for which I have my own band-growth algorithm to ensure sufficient coverage: i.e., I don't tag interface cells and rely on amrex, instead I tag interface cells, and grow a band of N cells which ensures I'll be safe for N time steps without regridding. Then I tag those cells. However, it's unclear to me how that could directly impact amrex's machinery.

Could you give me more info on what you're observing?

[WeiqunZhang]

Most likely it's a bug in amrex. The TagBoxArray with ghost cells is created by amrex. It needs to be safe even if every cells are tagged for refinement. In you case, new grids on level 7 are created first. Then we need to create level 6 grids based tagging on level 5 plus the constraint that level 6 needs to properly nest the level 7 new grids. We do the latter by adding new tags to level 5 on the amrex side (even if on the user side nothing is tagged). I believe that part has a flaw when there are periodic boundaries.

Sorry for the gibberish. But writing this down helps me think.

[desjardi]

No worries! Happy to discuss more if you want. I've been reading the tagging code quite a bit to try to understand whether there were implicit assumptions (i.e., not being able to use tag that are not monotonically evolving across levels, for example) while I was trying to understand if I had a bug on my end...

[WeiqunZhang]

The test is periodic in y and z directions. The checkpoint I am examining has no grids near periodic boundaries. So that is unlikely the issue.

[desjardi]

yeah, I wouldn't expect much to be happening near the y/z periodic boundaries in this case indeed

[desjardi]

@WeiqunZhang - I wanted to give you a quick follow-up. We've been running a range of cases with increasingly deep hierarchy, and so far everything has been good. Thanks again for your fix!

[desjardi]

Hi @WeiqunZhang! Very sorry to be revisiting this one...

We've hit a strange crash in the MLMG solver in an incompressible sim, using the same regridding strategy you've seen from us before (regrid every 10 steps, fairly deep hierarchy). Not fully diagnosed yet, but it's looking like regridding is the culprit, and the key clue is that setting amr.n_proper = 2 makes it go away fully.

The one big difference from before: this time we have active, complex regridding across a periodic direction. Does that ring any bells? Any reason to think your fix could be insufficient across periodic boundaries?

[desjardi]

MLABecLaplacian, cell-centered, multi-level starting from level 0.

[WeiqunZhang]

Thanks for the information. I will think about it. Is the blocking factor 8 and the ref ratio 2?

In MLABecLaplacian, the B coefficients are defined on the faces. If they are not consistent across periodic boundaries, the solver will have issues. That's a possibility, but it does not explain why n_proper=2 works. However, it's still worth looking it first. How do you compute the face coefficients?

[desjardi]

block factor is 8, ref_ratio is 2. I ensure consistency of the face coefficients by doing proper parallel comm with OverrideSync(cg->periodicity()). I'll don't think I'm missing anything, but I should double-check.

[WeiqunZhang]

Just in case: cg should be Geometry for that level, because Periodicity is not just a boolean flag.

[desjardi]

yes, I oversimplified in the reply, but I have it done correctly (I think!)