Allow restart calculations from a p-refined solution #30052

YaqiWang · 2025-03-07T04:16:01Z

This possibly will also enable p-refinement in mesh generators.

Closes #30048.

moosebuild · 2025-03-08T01:03:15Z

Job Documentation, step Docs: sync website on 3c161ae wanted to post the following:

View the site here

This comment will be updated on new commits.

YaqiWang · 2025-03-11T15:48:52Z

Are the test failures related with my change?

roystgnr · 2025-03-11T19:34:55Z

Are the test failures related with my change?

Well, they're definitely not a fluke. Most of these failures look the same to me: we're hitting a material's computeQpProperties() via FEProblemBase::initElementStatefulProps(), but when we try to set a value for qp 0 we discover that the underlying MooseArray is still sitting at size 0, so it screams at us in debug builds and crashes in opt builds.

I don't see how they could be triggered by your changes here, though. Unless the changes from update() to meshChanged() in SetupMeshCompleteAction::act() are inadvertently causing some initialization to be omitted??

YaqiWang · 2025-03-11T19:38:57Z

I don't see how they could be triggered by your changes here, though.

Me neither ;-( MooseMesh::meshChanged is calling MooseMesh::update. Without it, I actually saw a test crash on my desktop, (the test is using pre-split mesh with a h uniform refinement) which makes sense to me. I think the failing tests are there without it.

YaqiWang · 2025-03-11T19:44:46Z

Eh, I saw one function MooseMesh::buildRefinementAndCoarseningMaps could be a bit suspicious. Let me check.

YaqiWang · 2025-03-11T20:46:51Z

Still cannot find any... One thing is that EmptyAction is no longer build in THM tests with my change, but that does not seem matter because it is an empty action.

YaqiWang · 2025-03-11T21:52:54Z

framework/src/mesh/MooseMesh.C

@@ -614,6 +614,13 @@ MooseMesh::update()
  cacheInfo();
  buildElemIDInfo();

+  // this will make moose mesh aware of p-refinement added by mesh generators including
+  // a file mesh generator loading a restart checkpoint file
+  for (const auto & elem : *getActiveLocalElementRange())


Something quite insane is happening here. If I replace this with

for (const auto & elem : getMesh().active_local_element_ptr_range())

the error is gone. While I admit I should directly use the libMesh element range, but this code is not affecting anything supposedly. I will push the change momentarily.

I'd admit that MOOSE has a lot of shims that are basically "add a second way to do a libMesh thing for no better reason than a preference for camelCase over snake_case", there generally aren't any shims that are "add a second way to do a libMesh thing, but make it broken this time".

... unless this is one, in this context? getActiveLocalElementRange() doesn't just get two libMesh iterators defining a range, it copies the entire range into a new StoredRange, and it caches that copy. This isn't insane (we use StoredRange in our threading APIs, and the copy is too expensive to repeat for every threaded section), but: if anyone adds an element to the mesh or deletes an element from the mesh, then tries to use that cached range again before calling MooseMesh::meshChanged(), the result is undefined behavior. A crash if you're lucky.

lindsayad · 2025-03-11T22:04:11Z

framework/src/actions/SetupMeshCompleteAction.C

+  const auto families_enum = AddVariableAction::getNonlinearVariableFamilies();
+  MultiMooseEnum disable_p_refinement_for_families(families_enum.getRawNames());
+  params.addParam<MultiMooseEnum>("disable_p_refinement_for_families",
+                                  disable_p_refinement_for_families,
+                                  "What families we should disable p-refinement for.");


This feels very strange to have algebraic information in a mesh action

lindsayad · 2025-03-11T22:06:09Z

test/tests/adaptivity/dont-p-refine/tests

@@ -22,71 +22,71 @@
      input = test.i
      csvdiff = test_out.csv
      prereq = 'disable_p_refinement/lagrange_second_order_mesh'
-      cli_args = 'Adaptivity/disable_p_refinement_for_families="hermite" AuxVariables/test/family=HERMITE AuxVariables/test/order=THIRD'
+      cli_args = 'Mesh/disable_p_refinement_for_families="hermite" AuxVariables/test/family=HERMITE AuxVariables/test/order=THIRD'


yea, I'm not a fan of this at all. What's the point of having an Adaptivity block at this point

The mesh could be p-refined by loading a checkpoint file. In such a case, we do not have [Adpativity] but we still need to disable certain shape function families. I think this is the reason that we need to move this into [Mesh]. We could move this into [Problem], but that is a syntax used for creating custom problems and may create implications. Having this parameter in [Mesh] just mean that the created mesh will not support these shape functions. Most of times, users can just use the default I guess.

I think it would be much better to save the information about what families to disable p-refinement for as restartable data in the problem/assembly

A mesh has absolutely no data about finite element families

It might be possible to move the code directly in FEProblemBase::init at its beginning. If so, disable_p_refinement_for_families can be a parameter of [Problem].

If it is a restartable data, it won't be available before calling FEProblemBase::init. We will have to call doingPRefinement after data are restarted in FEProblemBase::initialSetup. Will that be too late?

I don't think it would be too late, but I am not 100% sure. It just needs to happen before a FE::reinit occurs. I would like to at least try and see becuase I still think this parameter is a better fit in the Adaptivity block as it is about adaptivity

moosebuild · 2025-03-12T05:36:54Z

Job OpenMPI on aec4026 : invalidated by @YaqiWang

Random failure? Cannot produce this on my mac.

moosebuild · 2025-03-12T16:23:12Z

Job Modules parallel on aec4026 : invalidated by @YaqiWang

Random failure? Cannot produce this on my mac.

moosebuild · 2025-03-12T19:19:20Z

Job Coverage, step Generate coverage on 3c161ae wanted to post the following:

Framework coverage

	a5ad50	#30052 3c161a
	Total	Total	+/-	New
Rate	85.29%	85.29%	+0.00%	93.33%
Hits	109051	109070	+19	70
Misses	18807	18810	+3	5

Diff coverage report

Full coverage report

Modules coverage

Level set

	a5ad50	#30052 3c161a
	Total	Total	+/-	New
Rate	85.49%	85.49%	-	100.00%
Hits	324	324	-	1
Misses	55	55	-	0

Diff coverage report

Full coverage report

Full coverage reports

Reports

This comment will be updated on new commits.

moosebuild · 2025-03-12T19:19:21Z

Job Coverage, step Verify coverage on aec4026 wanted to post the following:

The following coverage requirement(s) failed:

Failed to generate subchannel coverage rate (required: 87.0%)

lindsayad · 2025-03-13T05:33:35Z

framework/src/problems/FEProblemBase.C

+    if (_displaced_mesh)
+      _displaced_mesh->doingPRefinement(true);
+    if (_displaced_problem)
+    {
+      _displaced_problem->mesh().doingPRefinement(true);


These two are the same

lindsayad · 2025-03-13T05:44:02Z

test/tests/adaptivity/dont-p-refine/tests

@@ -22,71 +22,71 @@
      input = test.i
      csvdiff = test_out.csv
      prereq = 'disable_p_refinement/lagrange_second_order_mesh'
-      cli_args = 'Adaptivity/disable_p_refinement_for_families="hermite" AuxVariables/test/family=HERMITE AuxVariables/test/order=THIRD'
+      cli_args = 'Mesh/disable_p_refinement_for_families="hermite" AuxVariables/test/family=HERMITE AuxVariables/test/order=THIRD'


I don't think it would be too late, but I am not 100% sure. It just needs to happen before a FE::reinit occurs. I would like to at least try and see becuase I still think this parameter is a better fit in the Adaptivity block as it is about adaptivity

YaqiWang · 2025-03-13T19:35:31Z

The issue is that this parameter disable_p_refinement_for_families is needed for a p-refined mesh in a checkpoint file loaded with file mesh generator, or in general any mesh generator performing p-refinement. I have to say that this parameter does not belong to Adaptivity although it is the only place using it currently. I will try making it restartable in FEProblemBase and report back.

YaqiWang · 2025-03-13T19:52:02Z

Got a PETSc error with the new test I added if I move the doingPRefinement call after restart in FEProblemBase::initialSetup:

restart/p_refinement_restart.steady_from_steady_restart: [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
restart/p_refinement_restart.steady_from_steady_restart: [0]PETSC ERROR: Object is in wrong state
restart/p_refinement_restart.steady_from_steady_restart: [0]PETSC ERROR: Matrix is missing diagonal entry 33

lindsayad · 2025-03-13T21:29:05Z

Let me take a look. Thanks for trying

lindsayad · 2025-03-13T21:29:41Z

Let me take a look. Thanks for trying

lindsayad · 2025-03-14T00:10:45Z

Can you push a commit with the change to use restartable data so I can figure out why we are getting the PETSc error?

YaqiWang · 2025-03-14T16:27:48Z

Can you push a commit with the change to use restartable data so I can figure out why we are getting the PETSc error?

The test in this PR can reproduce it.

lindsayad · 2025-03-16T22:04:42Z

Did you not try declaring restartable data then?

YaqiWang · 2025-03-17T03:26:12Z

Did you not try declaring restartable data then?

No. I stopped after seeing the error by moving the doingPRefinement call after restart in FEProblemBase::initialSetup.

YaqiWang · 2025-03-17T17:23:03Z

Per our discussion today, we want to have a parameter Variables/*/disable_p_refinement replace the current Problem/disable_p_refinement_for_families. The parameter will have default value false for all families not in the list. We will have the same parameter for aux variables.

…a crash in tests/mesh/split_uniform_refine test

YaqiWang · 2025-03-18T04:44:06Z

@lindsayad this is ready again.

lindsayad

Not quite done, but I'll comment with what I have for now

lindsayad · 2025-03-18T22:39:39Z

framework/include/base/Assembly.h

@@ -1923,7 +1923,7 @@ class Assembly
   *   of quadrature points reflects the element p-level)
   * @param disable_p_refinement_for_families Families that we should disable p-refinement for
   */
-  void havePRefinement(const std::vector<FEFamily> & disable_p_refinement_for_families);
+  void havePRefinement(const std::set<FEFamily> & disable_p_refinement_for_families);


Suggested change

void havePRefinement(const std::set<FEFamily> & disable_p_refinement_for_families);

void havePRefinement(const std::unordered_set<FEFamily> & disable_p_refinement_for_families);

lindsayad · 2025-03-18T22:55:13Z

framework/include/problems/SubProblem.h

@@ -1203,6 +1206,17 @@ class SubProblem : public Problem
  /// Whether p-refinement has been requested at any point during the simulation
  bool _have_p_refinement;

+  /// Indicate whether a family is disabled for p-refinement
+  std::map<FEFamily, bool> _family_for_p_refinement;


Suggested change

std::map<FEFamily, bool> _family_for_p_refinement;

std::unordered_map<FEFamily, bool> _family_for_p_refinement;

lindsayad · 2025-03-18T22:55:27Z

framework/include/problems/SubProblem.h

+  /// Indicate whether a family is disabled for p-refinement
+  std::map<FEFamily, bool> _family_for_p_refinement;
+  /// The set of variable families by default disable p-refinement
+  std::set<FEFamily> _default_families_without_p_refinement = {libMesh::LAGRANGE,


Suggested change

std::set<FEFamily> _default_families_without_p_refinement = {libMesh::LAGRANGE,

std::unordered_set<FEFamily> _default_families_without_p_refinement = {libMesh::LAGRANGE,

lindsayad · 2025-03-18T22:55:46Z

framework/include/problems/SubProblem.h

+  std::set<FEFamily> _default_families_without_p_refinement = {libMesh::LAGRANGE,
+                                                               libMesh::NEDELEC_ONE,
+                                                               libMesh::RAVIART_THOMAS,
+                                                               libMesh::LAGRANGE_VEC,
+                                                               libMesh::CLOUGH,
+                                                               libMesh::BERNSTEIN,
+                                                               libMesh::RATIONAL_BERNSTEIN};


Populate this in the constructor plz

YaqiWang requested review from lindsayad, roystgnr and GiudGiud as code owners March 7, 2025 04:16

YaqiWang force-pushed the p_restart_30048 branch 4 times, most recently from ebb96a0 to 9027bde Compare March 7, 2025 22:29

YaqiWang commented Mar 11, 2025

View reviewed changes

YaqiWang force-pushed the p_restart_30048 branch from 6297756 to ed501fb Compare March 11, 2025 21:54

lindsayad requested changes Mar 11, 2025

View reviewed changes

YaqiWang force-pushed the p_restart_30048 branch from ed501fb to aec4026 Compare March 12, 2025 01:31

lindsayad reviewed Mar 13, 2025

View reviewed changes

YaqiWang force-pushed the p_restart_30048 branch from aec4026 to acb8350 Compare March 18, 2025 04:39

should call full mesh changed after uniform refinement otherwise got …

625f8b4

…a crash in tests/mesh/split_uniform_refine test

YaqiWang force-pushed the p_restart_30048 branch from acb8350 to 3c161ae Compare March 18, 2025 04:41

lindsayad requested changes Mar 18, 2025

View reviewed changes

allow restart calculations from a p-refined solution idaholab#30048

fd3d5ba

YaqiWang force-pushed the p_restart_30048 branch from 3c161ae to fd3d5ba Compare March 19, 2025 20:15

	void havePRefinement(const std::set<FEFamily> & disable_p_refinement_for_families);
	void havePRefinement(const std::unordered_set<FEFamily> & disable_p_refinement_for_families);

	std::map<FEFamily, bool> _family_for_p_refinement;
	std::unordered_map<FEFamily, bool> _family_for_p_refinement;

	std::set<FEFamily> _default_families_without_p_refinement = {libMesh::LAGRANGE,
	std::unordered_set<FEFamily> _default_families_without_p_refinement = {libMesh::LAGRANGE,

Allow restart calculations from a p-refined solution #30052

Are you sure you want to change the base?

Allow restart calculations from a p-refined solution #30052

Conversation

YaqiWang commented Mar 7, 2025

moosebuild commented Mar 8, 2025 • edited Loading

YaqiWang commented Mar 11, 2025

roystgnr commented Mar 11, 2025

YaqiWang commented Mar 11, 2025

YaqiWang commented Mar 11, 2025

YaqiWang commented Mar 11, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

YaqiWang Mar 11, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

moosebuild commented Mar 12, 2025

moosebuild commented Mar 12, 2025

moosebuild commented Mar 12, 2025 • edited Loading

Framework coverage

Modules coverage

Level set

Full coverage reports

moosebuild commented Mar 12, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

YaqiWang commented Mar 13, 2025

YaqiWang commented Mar 13, 2025

lindsayad commented Mar 13, 2025

lindsayad commented Mar 13, 2025

lindsayad commented Mar 14, 2025

YaqiWang commented Mar 14, 2025

lindsayad commented Mar 16, 2025

YaqiWang commented Mar 17, 2025

YaqiWang commented Mar 17, 2025 • edited Loading

YaqiWang commented Mar 18, 2025

lindsayad left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

moosebuild commented Mar 8, 2025 •

edited

Loading

YaqiWang Mar 11, 2025 •

edited

Loading

moosebuild commented Mar 12, 2025 •

edited

Loading

YaqiWang commented Mar 17, 2025 •

edited

Loading