Symplectic integration in a user-defined magnetostatic vector potential #964

cemitch99 · 2025-05-15T01:55:44Z

This PR adds a new element that allows the user to track through a region with a specified magnetostatic vector potential. Symplectic integration is performed using the exact form of the nonlinear relativistic Hamiltonian. We use the semiexplicit integrator appearing in:

B. Jayawardana and T. Ohsawa, "Semiexplicit symplectic integrators for non-separable Hamiltonian systems," Math. Comput. 92, pp. 251-281 (2022),
https://doi.org/10.1090/mcom/3778

To do:

for more information, see https://pre-commit.ci

src/elements/VectorPotential.H

for more information, see https://pre-commit.ci

src/elements/VectorPotential.H

Reduce parallelism to avoid job shutdown

ax3l · 2025-08-08T20:53:46Z

The tolerances seem to fail with MPI, maybe just need a small relaxation @cemitch99 ?

Errors while running CTest
	485 - examples-fodo-vector-potential.py.analysis (Failed)
	491 - examples-exact-quad-vector-potential.py.analysis (Failed)
	512 - examples-solenoid-vector-potential.py.analysis (Failed)

ax3l · 2025-08-11T23:38:09Z

src/elements/MagnetostaticVectorPotential.H

+            // Evaluate the vector potential and its derivatives
+            auto const ax = m_dfunc_ax(x, y, z) * m_scale;
+            auto const ay = m_dfunc_ay(x, y, z) * m_scale;
+            auto const daxdx = m_dfunc_daxdx(x, y, z) * m_scale;
+            auto const daxdy = m_dfunc_daxdy(x, y, z) * m_scale;
+            auto const daydx = m_dfunc_daydx(x, y, z) * m_scale;
+            auto const daydy = m_dfunc_daydy(x, y, z) * m_scale;
+            auto const dazdx = m_dfunc_dazdx(x, y, z) * m_scale;
+            auto const dazdy = m_dfunc_dazdy(x, y, z) * m_scale;


@WeiqunZhang I fear we use too many parsers for the CUDA CI (?) 😢

Possible hack to confirm

Suggested change

// Evaluate the vector potential and its derivatives

auto const ax = m_dfunc_ax(x, y, z) * m_scale;

auto const ay = m_dfunc_ay(x, y, z) * m_scale;

auto const daxdx = m_dfunc_daxdx(x, y, z) * m_scale;

auto const daxdy = m_dfunc_daxdy(x, y, z) * m_scale;

auto const daydx = m_dfunc_daydx(x, y, z) * m_scale;

auto const daydy = m_dfunc_daydy(x, y, z) * m_scale;

auto const dazdx = m_dfunc_dazdx(x, y, z) * m_scale;

auto const dazdy = m_dfunc_dazdy(x, y, z) * m_scale;

// Evaluate the vector potential and its derivatives

auto const ax = 0_prt;

auto const ay = 0_prt;

auto const daxdx = 0_prt;

auto const daxdy = 0_prt;

auto const daydx = 0_prt;

auto const daydy = 0_prt;

auto const dazdx = 0_prt;

auto const dazdy = 0_prt;

We probably could try to merge them into one Parser. Let me think about it.

Or maybe it does not help. I think we need to force noinline them.

How about this?

amrex::GpuArray<amrex::ParserExecutor<3>,8> df{m_dfunc_ax, m_dfunc_ay, ....}; // on host // on device amrex::Real results[8]; for (int i = 0; i < 8; ++i) { results[i] = df[i](x,y,z) * m_scale; }

You might add a pragma to make sure the loop is not unrolled.

As for the noinline approach, you can add a helper in your code that that marked as noinline. Something like

AMREX_GPU_HOST_DEVICE AMREX_NO_INLINE template <int N, typename... T> auto call_parser (ParserExecutor<N> const& f, T... xyz) { return f(xyz...); }

I think that might be good!

So far, reducing the compile to -j1 for CUDA helped, but calling a non-inline for this case would be super helpful.

@WeiqunZhang do you like to pus hthis to the PR or a follow-up?

I can put the helper function in amrex. Then we can use it here or in another PR. (I tested it. It did work.)

AMReX-Codes/amrex#4606

.github/workflows/cuda.yml

…linear_element

ax3l · 2025-08-13T06:29:33Z

Something is still off with the Python tests. Besides the tolerance issues, they seem to run significantly longer than their app/executable counterparts?

ctest --test-dir build -R vector
      Start 481: fodo-vector-potential.run
 1/18 Test #481: fodo-vector-potential.run .................   Passed    3.65 sec
      Start 482: fodo-vector-potential.analysis
 2/18 Test #482: fodo-vector-potential.analysis ............   Passed    0.30 sec
      Start 483: fodo-vector-potential.cleanup
 3/18 Test #483: fodo-vector-potential.cleanup .............   Passed    0.01 sec
      Start 484: fodo-vector-potential.py.run
 4/18 Test #484: fodo-vector-potential.py.run ..............   Passed  268.34 sec
      Start 485: fodo-vector-potential.py.analysis
 5/18 Test #485: fodo-vector-potential.py.analysis .........***Failed    0.26 sec
      Start 486: fodo-vector-potential.py.cleanup
 6/18 Test #486: fodo-vector-potential.py.cleanup ..........   Passed    0.01 sec
      Start 487: exact-quad-vector-potential.run
 7/18 Test #487: exact-quad-vector-potential.run ...........   Passed    3.73 sec
      Start 488: exact-quad-vector-potential.analysis
 8/18 Test #488: exact-quad-vector-potential.analysis ......   Passed    0.57 sec
      Start 489: exact-quad-vector-potential.cleanup
 9/18 Test #489: exact-quad-vector-potential.cleanup .......   Passed    0.01 sec
      Start 490: exact-quad-vector-potential.py.run
10/18 Test #490: exact-quad-vector-potential.py.run ........   Passed   11.98 sec
      Start 491: exact-quad-vector-potential.py.analysis
11/18 Test #491: exact-quad-vector-potential.py.analysis ...***Failed    0.57 sec
      Start 492: exact-quad-vector-potential.py.cleanup
12/18 Test #492: exact-quad-vector-potential.py.cleanup ....   Passed    0.01 sec
      Start 508: solenoid-vector-potential.run
13/18 Test #508: solenoid-vector-potential.run .............   Passed    6.24 sec
      Start 509: solenoid-vector-potential.analysis
14/18 Test #509: solenoid-vector-potential.analysis ........   Passed    0.59 sec
      Start 510: solenoid-vector-potential.cleanup
15/18 Test #510: solenoid-vector-potential.cleanup .........   Passed    0.01 sec
      Start 511: solenoid-vector-potential.py.run
16/18 Test #511: solenoid-vector-potential.py.run ..........   Passed   23.76 sec
      Start 512: solenoid-vector-potential.py.analysis
17/18 Test #512: solenoid-vector-potential.py.analysis .....***Failed    0.58 sec
      Start 513: solenoid-vector-potential.py.cleanup
18/18 Test #513: solenoid-vector-potential.py.cleanup ......   Passed    0.01 sec

ax3l · 2025-08-13T06:37:32Z

docs/source/usage/parameters.rst

+This element requires these additional parameters:
+
+* ``<element_name>.ds`` (``float``, in meters) the segment length
+* ``<element_name>.unit`` (``integer``) specification of units for the vector potential (default: ``0``)


This is called unit while the inputs file (app API) calls it units

Was not the same input as used in the app example

examples/vector_potential/run_exact_quad_vector_potential.py

Was not the same input as used in the app example

ax3l · 2025-08-13T07:06:38Z

@cemitch99 the three python run files are not yet 100% identical with the app inputs files, which should be the origin of the failing tests. I fixed the things I spotted, but some differences remain.

This works on lambdas, functors, normal functions. But it does not work on overloaded functions like std::sin. If needed, one could however wrap functions like std::sin inside a lambda function. Here is the motivation behind this PR. In this impactx PR (BLAST-ImpactX/impactx#964), a GPU kernel uses 8 amrex::Parser's. The CUDA CI fails if more than one job is used in build. Apparently the kernel is too big because all those parser functions are inlined. This PR provides a way to reduce the size by forcing noinline.

This works on lambdas, functors, normal functions. But it does not work on overloaded functions like std::sin. If needed, one could however wrap functions like std::sin inside a lambda function. It also does not work with normal functions for SYCL and one would have to wrap it inside a lambda. Here is the motivation behind this PR. In this impactx PR (BLAST-ImpactX/impactx#964), a GPU kernel uses 8 amrex::Parser's. The CUDA CI fails if more than one job is used in build. Apparently the kernel is too big because all those parser functions are inlined. This PR provides a way to reduce the size by forcing noinline.

…xact_quad_vector_potential.py.

cemitch99 · 2025-08-29T17:25:54Z

The three new tests currently fail only in the case OpenMP / GCC w/ MPI w/ Python. Note that the execution time (of the Python test) is very long - 32.42 sec for the solenoid example, which is only 0.56 sec on macOS / AppleClang (with similar behavior for the other two tests). Also, although the solenoid example runs successfully, the initial and final beam moments agree to all digits (and they should not), which appears to indicate that no tracking push was applied to the beam.

ax3l · 2025-08-29T18:52:41Z

Thanks! If they run a bit on the longer end, let us add the slow label on these (in CMakeLists.txt).

examples/vector_potential/analysis_fodo_vector_potential.py

cemitch99 and others added 2 commits May 14, 2025 18:40

Add initial element and integrator routines in draft form.

77509bc

[pre-commit.ci] auto fixes from pre-commit.com hooks

d169e14

for more information, see https://pre-commit.ci

cemitch99 marked this pull request as draft May 15, 2025 01:59

Add literals.

c812000

cemitch99 marked this pull request as ready for review May 15, 2025 02:14

ax3l self-requested a review June 18, 2025 18:48

ax3l added component: elements Elements/external fields tracking: particles labels Jun 18, 2025

ax3l added this to the HTU LDRD milestone Jun 18, 2025

cemitch99 and others added 11 commits June 25, 2025 15:42

Resolve conflicts, and add comments.

a1ab4b3

Merge branch 'development' into add_user_nonlinear_element

d5cf7da

[pre-commit.ci] auto fixes from pre-commit.com hooks

1048cd3

for more information, see https://pre-commit.ci

Update Integrators.H

5a0c003

Fix parameter documentation in new integrator step.

d46ed68

Add both particle copies as arguments to internal push map.

2a27e8a

Clean-up and addition of map for a general Hamiltonian.

9cded9e

[pre-commit.ci] auto fixes from pre-commit.com hooks

a0ec35d

for more information, see https://pre-commit.ci

Add preliminary initialization of VectorPotential element.

91b5f53

[pre-commit.ci] auto fixes from pre-commit.com hooks

3916603

for more information, see https://pre-commit.ci

Eliminate calls to Print to avoid errors.

1e41d52

github-advanced-security bot found potential problems Jul 17, 2025

View reviewed changes

src/elements/VectorPotential.H Fixed Show fixed Hide fixed

cemitch99 and others added 5 commits July 16, 2025 18:46

Add maybe_unused to avoid unused variable warnings.

4969aa4

Attempt to eliminate host-device error.

c5db4f9

Remove debug print statement.

2e8beca

Update map using Hamiltonian derivatives.

606fd2f

[pre-commit.ci] auto fixes from pre-commit.com hooks

a260f86

for more information, see https://pre-commit.ci

cemitch99 changed the title ~~[WIP] Symplectic integration in a user-defined vector potential~~ Symplectic integration in a user-defined vector potential Jul 17, 2025

Delete unnecessary comments.

d4dcaf5

cemitch99 commented Jul 17, 2025

View reviewed changes

src/elements/VectorPotential.H Outdated Show resolved Hide resolved

Comment unused az.

436a2e1

CUDA CI: -j 3

59abec1

Reduce parallelism to avoid job shutdown

ax3l and others added 3 commits August 8, 2025 14:01

CUDA CI: -j 2

2fb715d

Relax tolerance analysis_exact_quad_vector_potential.py

90d1ffe

Relax tolerance analysis_solenoid_vector_potential.py

91b8eb0

ax3l reviewed Aug 11, 2025

View reviewed changes

.github/workflows/cuda.yml Outdated Show resolved Hide resolved

ax3l and others added 6 commits August 11, 2025 16:38

CUDA CI: -j 1

4d2f67d

Merge branch 'development' into add_user_nonlinear_element

ed67c24

CMake: Unify Test Names

66555cb

Relax Tolerances

7ff35f6

Relax Tolerances

2aa0e1c

Merge remote-tracking branch 'mainline/development' into add_user_non…

3be5620

…linear_element

ax3l reviewed Aug 13, 2025

View reviewed changes

ax3l added 2 commits August 12, 2025 23:55

Fix (Some): Py FODO Vector Example

ae1ac85

Was not the same input as used in the app example

Fix (Some): Py Solenoid Vector Example

0bdf431

Was not the same input as used in the app example

ax3l reviewed Aug 13, 2025

View reviewed changes

examples/vector_potential/run_exact_quad_vector_potential.py Show resolved Hide resolved

Fix (Some): Py Quad Vector Example

801f5a2

Was not the same input as used in the app example

WeiqunZhang mentioned this pull request Aug 13, 2025

amrex::callNoinline: Call given function without inline AMReX-Codes/amrex#4606

Merged

cemitch99 added 2 commits August 28, 2025 15:54

Make input_exact_quad_vector_potential.in exactly coincide with run_e…

ee56530

…xact_quad_vector_potential.py.

Delete irrelevant comments in input_solenoid_vector_potential.in

faf0546

Merge branch 'development' into add_user_nonlinear_element

4888129

cemitch99 commented Oct 7, 2025

View reviewed changes

examples/vector_potential/analysis_fodo_vector_potential.py Outdated Show resolved Hide resolved

Update examples/vector_potential/analysis_fodo_vector_potential.py

10101e4

Symplectic integration in a user-defined magnetostatic vector potential #964

Are you sure you want to change the base?

Symplectic integration in a user-defined magnetostatic vector potential #964

Uh oh!

Conversation

cemitch99 commented May 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ax3l commented Aug 8, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ax3l commented Aug 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ax3l Aug 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ax3l commented Aug 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cemitch99 commented Aug 29, 2025

Uh oh!

ax3l commented Aug 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

cemitch99 commented May 15, 2025 •

edited

Loading

ax3l commented Aug 13, 2025 •

edited

Loading

ax3l Aug 13, 2025 •

edited

Loading

ax3l commented Aug 13, 2025 •

edited

Loading

ax3l commented Aug 29, 2025 •

edited

Loading