Issue with box refinement

[rthedin]

I'm running a few tests with ERF, trying very simple features to get familiar with it. So my potential bug report here is very simple.

I'm have periodic domain in x and y and was trying to have a box refinement region touching the xlo and ylo, but not xhi and yhi. If I try that, I get the following error within a few time steps:

amrex::Abort::63::Bad theta in check_for_negative_theta !!!

The error is gone if my refinement spans the entire x and y extents (that is, touching all four side boundaries). I also get the same fatal error when the refinement is simply not touching any periodic boundary (but touching zhi).

I'm attaching a reproducible MWE. I'm using the latest version as of this morning (6d41455).

PS: I believe there is a mistake in the documentation for the refinements. The following line (and a few others in the same page) should read amr.ref_ratio_vec.

ERF/Docs/sphinx_doc/MeshRefinement.rst

Line 49 in 6d41455

amr.ref_ratio = 2 2 2

input_sounding.txt
mwe_theta_issue.txt

[asalmgren]

@rthedin -- trying to reproduce now :-)

[asalmgren]

@rthedin -- re the ref_ratio vs ref_ratio_vect -- both are actually possible but they have different uses/formats -- I will add extra documentation that explains -- thanks for pointing that out!

[asalmgren]

@rthedin -- I see this line in your mwe_theta_issue.txt

erf.init_type = "input_sounding.txt"

but that generates an error with the current code because it needs to be "input_sounding" not "input_sounding.txt" ie you should replace that line by these two lines:

erf.init_type = "input_sounding"
erf.input_sounding_file = "input_sounding.txt"

So I'm confused as to how you could have run with this inputs file ...

[rthedin]

The txt I added just to be able to upload to github. I didn't try with txt. I made an effort to edit the main input file to include the txt, but went untested.

[rthedin]

(turns out github is a bit annoying with file extensions)

[hgopalan]

Ann we missed the geom[lev_for_box].CellSize() to geom[lev_for_box-1].CellSize() correction here: https://github.com/erf-model/ERF/blob/development/Source/ERF_Tagging.cpp#L509C1-L509C68

[rthedin]

I tried adding the -1 that Harish pointed out- no luck.

[asalmgren]

@rthedin -- I've been unable to reproduce your bug. Can you try with 1) the most recent version of ERF development branch, 2) the lines below for controlling the refinement, and 3) the lines below that for specifying init_type ... let me know if this runs for you. It does run for me locally

PROBLEM SIZE & GEOMETRY

geometry.prob_lo = 0 0 0
geometry.prob_hi = 256 256 128

amr.n_cell = 256 256 128

geometry.is_periodic = 1 1 0

REFINEMENT / REGRIDDING

amr.max_level = 1
amr.ref_ratio_vect = 2 2 2

erf.refinement_indicators = box1
erf.box1.max_level = 1

Not touching any periodic boundary: does not work

erf.box1.in_box_lo = 64 64 0
erf.box1.in_box_hi = 128 128 128

erf.init_type = "input_sounding"
erf.init_sounding_file = "input_sounding.txt"
erf.sounding_type = Ideal

[rthedin]

If I'm following it correctly, the only difference from your suggested inputs is the in_box_lo. I just tried with b379d50 on NREL's Kestrel. Still getting the same error. I can try building/running it on GPUs if you think it might help debug it. Right now I only ran on 2 CPU nodes.

I'm attaching the complete log file-- maybe you can spot something else that I'm missing.

log.erf.txt

[asalmgren]

I don't see anything wrong and not sure how to fix it if I can't break it. Could you try building in DEBUG mode and then see if that pops up anything.

[asalmgren]

@hgopalan -- could you give this case a try and see if it breaks for you as well? Do CPU only.

[rthedin]

Ann - here is the log with the debug build. I have also asked Harish to try running this on the same machine. I'm also attaching a single self-contained script that is doing everything (cloning, building, writing files, etc) and reproduces the bug on the machine I'm on. This log was produced by running this script.

log.erf.txt
0_erfbug.sh

[hgopalan]

I can reproduce the issue. ERF was compiled using GNU compiler for this run. Wonder if it can be issue?

[asalmgren]

I was actually just able to get it break as well on a smaller problem (that fits on my workstation). Working on a fix now :-)

[asalmgren]

Good news / bad news -- I know what's going on but need to think for a bit about how to fix it...