Particle Buffer for Backtransformed Diagnostics (#1898)

* move BTD call before Redistribute

* define particle buffer, BTD particle functor, particle filter

* select particles on the slice

* add particle functors

* add Lorentz Transform

* fix conflict

* add call to loretnz operator

* storing Particles in buffer

* This is WIP. Added particle buffers, filled them, sent them for flush with number already flushed. Does not work for multiple flushes. Crashes for OpenPMD.

* trailing endif after rebase

* adding print statements and not flushing particles in OPENPMD if numpart is 0

* last timestep flush is ensured at the end of evolve loop

* fix bug in declaring uy uy new and computing uzp

* set particle Geom, BA, and DMAP for particle flush with plotfile

* set Particle BA Geom DMAP for particle buffer and no BTD transform for force flush

* separate compute and pack from flush

* Fix Typo: resizeable -> resizable

Fixes HDF5 BTD particle output.

* new class for plotfile particles for BTD

* copy particle_H and DATA and Header. some WIP print statements

* Merge plotfile

* clean print statements

* fix warning message

* struct declaration in header, fix warning

* doxygen comments and copyright

* clean print statements

* fix eol and override function warning

* tile data

* fix output species array size bug

* fix access for particle buffer size

* clean and move time-update

* add cur_time update back

* remove cur time update which was called twice

* dont access particles flushed already for full diagnostics

* cur time must be updated for RigidInjection BTD CI test to pass

* temporarily move call to BTD

* updating time and calling BTD before movewindow

* cleanup

* reset benchmarks and analysis script

* clean and add comments

* fix particle box array, geom, dmap

* reset benchmarks for multi_J rz and ElectrostaticSphereEB_mixedBC

* wip commit

* wip commit

* add SI conversion

* abort for openpmd bp backend if species is selected. Also write particle output for BTD only if write_species is 1

* add documentation for aborting if adios is used with openpmd and add other BTD input parameters

* Apply suggestions from code review

commit Axel's suggestions from review

Co-authored-by: Axel Huebl <[email protected]>

* use bool instead of int

* fix doxygen format

* using h5 as backend in example test to ensure consistency with abort for particle output.

* fix doxygen comment

* reset benchmark again for comoving_2d_psatd galilean_2d_psatd multi_J_rz_psatd

* reset benchmark for background_mcc

* self-review suggestions

* reset benchmarks. Update with last snapshot full info

* Axel's PR suggestions

Co-authored-by: Axel Huebl <[email protected]>

* Axel's doxygen fix

Co-authored-by: Axel Huebl <[email protected]>

* add comments

* fix eol

* improved exception handling for stringsteam

* PR suggestions

* Axels' suggestions from code review

Co-authored-by: Axel Huebl <[email protected]>

* Axel's suggestions :)

Co-authored-by: Axel Huebl <[email protected]>

* simplify logic

* suggestions from review (Axel/Reva)

Co-authored-by: Axel Huebl <[email protected]>

* variable name change for clarity

* if num particles in tmp array is 0, return

* Use new BTD inputs to set up BTD for particles in the corresponding particle container

* unused var

* fix logic error

* speciesID undefined

* separate particle and field buffer calls and initialization for BTD. Data common to both are initialized separately

* rename variable so it does not ghost existing varname

* add more comments

* Assert that fields are on

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* redundant function definition

* unused variable

* unused variable zp

* 1D 2D 3D definition

* fix else

* Apply suggestions from code review

Add Prabhat's suggestion

Co-authored-by: Prabhat Kumar <[email protected]>

* missing semicolon and ignore xp yp for 1D

* resetting benchmarks for boosted sims and mcc sim

* temporarily changing tolerance since the relative difference for momentum_z is 3.68e-3 and the current tolerance is 2.5e-3

Co-authored-by: Axel Huebl <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Prabhat Kumar <[email protected]>

Author: 4 people

Docs/source/usage/parameters.rst

@@ -1782,8 +1782,8 @@ In-situ capabilities can be used by turning on Sensei or Ascent (provided they a
     If this is `1`, the last timestep is dumped regardless of ``<diag_name>.period``.
 
 * ``<diag_name>.diag_type`` (`string`)
-    Type of diagnostics. So far, only ``Full`` is supported.
-    example: ``diag1.diag_type = Full``.
+    Type of diagnostics. ``Full`` and ``BackTransformed``
+    example: ``diag1.diag_type = Full`` or ``diag1.diag_type = BackTransformed``
 
 * ``<diag_name>.format`` (`string` optional, default ``plotfile``)
     Flush format. Possible values are:
@@ -1814,6 +1814,7 @@ In-situ capabilities can be used by turning on Sensei or Ascent (provided they a
     ``bp`` is the `ADIOS I/O library <https://csmd.ornl.gov/adios>`_, ``h5`` is the `HDF5 format <https://www.hdfgroup.org/solutions/hdf5/>`_, and ``json`` is a `simple text format <https://en.wikipedia.org/wiki/JSON>`_.
     ``json`` only works with serial/single-rank jobs.
     When WarpX is compiled with openPMD support, the first available backend in the order given above is taken.
+    Note that when using ``BackTransformed`` diagnostic type, the openpmd format supports only ``h5`` backend for both species and fields, while ``bp`` backend can be used for visualizing fields, but not particles. The code will abort if ``bp`` is selected for particle output.
 
 * ``<diag_name>.openpmd_encoding`` (optional, ``v`` (variable based), ``f`` (file based) or ``g`` (group based) ) only read if ``<diag_name>.format = openpmd``.
      openPMD `file output encoding <https://openpmd-api.readthedocs.io/en/0.14.0/usage/concepts.html#iteration-and-series>`__.
@@ -1930,8 +1931,43 @@ In-situ capabilities can be used by turning on Sensei or Ascent (provided they a
 
 .. _running-cpp-parameters-diagnostics-btd:
 
-Back-Transformed Diagnostics
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+BackTransformed Diagnostics (with support for Plotfile/openPMD output)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+``BackTransformed`` diag type are used when running a simulation in a boosted frame, to reconstruct output data to the lab frame. This option can be set using ``<diag_name>.diag_type = BackTransformed``. Additional options for this diagnostic include:
+
+* ``<diag_name>.num_snapshots_lab`` (`integer`)
+    Only used when ``<diag_name>.diag_type`` is ``BackTransformed``.
+    The number of lab-frame snapshots that will be written.
+
+* ``<diag_name>.dt_snapshots_lab`` (`float`, in seconds)
+    Only used when ``<diag_name>.diag_type`` is ``BackTransformed``.
+    The time interval inbetween the lab-frame snapshots (where this
+    time interval is expressed in the laboratory frame).
+
+* ``<diag_name>.dz_snapshots_lab`` (`float`, in meters)
+    Only used when ``<diag_name>.diag_type`` is ``BackTransformed``.
+    Distance between the lab-frame snapshots (expressed in the laboratory
+    frame). ``dt_snapshots_lab`` is then computed by
+    ``dt_snapshots_lab = dz_snapshots_lab/c``. Either `dt_snapshots_lab`
+    or `dz_snapshot_lab` is required.
+
+* ``<diag_name>.buffer_size`` (`integer`)
+    Only used when ``<diag_name>.diag_type`` is ``BackTransformed``.
+    The default size of the back transformed diagnostic buffers used to generate lab-frame
+    data is 256. That is, when the multifab with lab-frame data has 256 z-slices,
+    the data will be flushed out. However, if many lab-frame snapshots are required for
+    diagnostics and visualization, the GPU may run out of memory with many large boxes with
+    a size of 256 in the z-direction. This input parameter can then be used to set a
+    smaller buffer-size, preferably multiples of 8, such that, a large number of
+    lab-frame snapshot data can be generated without running out of gpu memory.
+    The downside to using a small buffer size, is that the I/O time may increase due
+    to frequent flushes of the lab-frame data. The other option is to keep the default
+    value for buffer size and use slices to reduce the memory footprint and maintain
+    optimum I/O performance.
+
+Back-Transformed Diagnostics (legacy output)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 ``BackTransformedDiagnostics`` are used when running a simulation in a boosted frame, to reconstruct output data to the lab frame, and
 

Examples/Modules/boosted_diags/inputs_3d_slice

@@ -124,6 +124,7 @@ btd_openpmd.dz_snapshots_lab = 0.001
 btd_openpmd.fields_to_plot = Ex Ey Ez Bx By Bz jx jy jz rho
 btd_openpmd.format = openpmd
 btd_openpmd.buffer_size = 32
+btd_openpmd.openpmd_backend = h5
 
 btd_pltfile.diag_type = BackTransformed
 btd_pltfile.do_back_transformed_fields = 1

Examples/Physics_applications/capacitive_discharge/analysis.py

@@ -15,5 +15,5 @@
 
 my_check = checksumAPI.evaluate_checksum(
     'background_mcc', 'Python_background_mcc_plt00050',
-    do_particles=True, rtol=2.5e-3
+    do_particles=True, rtol=3.7e-3
 )

Examples/Tests/ElectrostaticDirichletBC/analysis.py

@@ -30,7 +30,7 @@
 for ii, file in enumerate(files):
     ds = yt.load( file )
     times[ii] = (
-        ds.current_time.item() - float(ds.parameters.get('warpx.const_dt'))
+        ds.current_time.item()
     )
     data = ds.covering_grid(
         level=0, left_edge=ds.domain_left_edge, dims=ds.domain_dimensions

Regression/Checksum/benchmarks_json/LaserAccelerationBoost.json

@@ -4,41 +4,41 @@
     "particle_id": 500500.0,
     "particle_momentum_x": 4.40462890925364e-19,
     "particle_momentum_y": 4.375137239523109e-19,
-    "particle_momentum_z": 2.680882004211874e-18,
+    "particle_momentum_z": 2.6808820042118736e-18,
     "particle_position_x": 0.0008126719835048633,
     "particle_position_y": 0.3596147573048867,
     "particle_weight": 62415090744.60765
   },
   "electrons": {
-    "particle_cpu": 1140.0,
-    "particle_id": 1254690.0,
-    "particle_momentum_x": 4.851331230863139e-27,
-    "particle_momentum_y": 2.198180048358965e-22,
-    "particle_momentum_z": 3.0976485630303324e-18,
-    "particle_position_x": 0.06840000000361958,
-    "particle_position_y": 0.03930011372955363,
-    "particle_weight": 6.286568807682775e+16
+    "particle_cpu": 1200.0,
+    "particle_id": 1392000.0,
+    "particle_momentum_x": 6.2983505352251814e-27,
+    "particle_momentum_y": 2.5611486433369074e-22,
+    "particle_momentum_z": 3.260682694944147e-18,
+    "particle_position_x": 0.07200000000522772,
+    "particle_position_y": 0.040410946815682504,
+    "particle_weight": 7.205670325760674e+16
   },
   "ions": {
-    "particle_cpu": 1140.0,
-    "particle_id": 1390110.0,
-    "particle_momentum_x": 3.360052546384035e-30,
-    "particle_momentum_y": 2.198180119275469e-22,
-    "particle_momentum_z": 5.687755728408193e-15,
-    "particle_position_x": 0.0684,
-    "particle_position_y": 0.03930011366678667,
-    "particle_weight": 6.286568807682775e+16
+    "particle_cpu": 1200.0,
+    "particle_id": 1538400.0,
+    "particle_momentum_x": 7.73523770301275e-30,
+    "particle_momentum_y": 2.5611486596061174e-22,
+    "particle_momentum_z": 5.987111293060833e-15,
+    "particle_position_x": 0.072,
+    "particle_position_y": 0.04041094673206954,
+    "particle_weight": 7.205670325760674e+16
   },
   "lev=0": {
-    "Bx": 7187.996086766834,
-    "By": 16.42954217429746,
-    "Bz": 426.1198913459503,
-    "Ex": 4509306942.33143,
-    "Ey": 1984367969782.724,
-    "Ez": 6309586883.984778,
-    "jx": 4725348020.173394,
-    "jy": 551784861830530.6,
-    "jz": 449864633308.4492,
-    "rho": 1568.420379535507
+    "Bx": 7184.641132345461,
+    "By": 16.42963921477926,
+    "Bz": 425.9213641032897,
+    "Ex": 4509329694.602806,
+    "Ey": 1984276461005.9048,
+    "Ez": 6310104620.917465,
+    "jx": 4752150463.6297245,
+    "jy": 553920892529501.3,
+    "jz": 450362837774.94794,
+    "rho": 1569.6222851908453
   }
 }

Regression/Checksum/benchmarks_json/PlasmaAccelerationBoost2d.json

@@ -2,7 +2,7 @@
   "beam": {
     "particle_cpu": 0.0,
     "particle_id": 1500500.0,
-    "particle_momentum_x": 4.331952826767249e-19,
+    "particle_momentum_x": 4.3319528267672493e-19,
     "particle_momentum_y": 4.501000169972693e-19,
     "particle_momentum_z": 2.730833780425186e-17,
     "particle_position_x": 0.0003765650634942415,
@@ -12,42 +12,42 @@
   "driver": {
     "particle_cpu": 0.0,
     "particle_id": 500500.0,
-    "particle_momentum_x": 4.404628898427658e-19,
+    "particle_momentum_x": 4.404628898428432e-19,
     "particle_momentum_y": 4.375137239499453e-19,
-    "particle_momentum_z": 2.74242453057886e-15,
+    "particle_momentum_z": 2.7424245305788595e-15,
     "particle_position_x": 0.001625341951975183,
-    "particle_position_y": 0.3195380522103931,
+    "particle_position_y": 0.31953805221039305,
     "particle_weight": 93622636116911.45
   },
   "lev=0": {
-    "Bx": 0.3939499532670673,
-    "By": 9563.615015137155,
-    "Bz": 0.3591030714778953,
-    "Ex": 1057402563551.624,
-    "Ey": 144244338.6803337,
-    "Ez": 3010491714956.791,
-    "jx": 3215863168.881746,
-    "jy": 5068791682.024239,
-    "jz": 102420308627362.4
+    "Bx": 0.3939499531816347,
+    "By": 9563.615022654387,
+    "Bz": 0.35910306957800636,
+    "Ex": 1057402535441.4385,
+    "Ey": 144244335.46921343,
+    "Ez": 3010491709183.074,
+    "jx": 3227910749.573446,
+    "jy": 5068791609.127937,
+    "jz": 102420312560044.6
   },
   "plasma_e": {
     "particle_cpu": 504.0,
     "particle_id": 236124.0,
-    "particle_momentum_x": 1.261492278768167e-23,
-    "particle_momentum_y": 1.539229073145743e-27,
-    "particle_momentum_z": 1.369490412666826e-18,
-    "particle_position_x": 0.01771877489305462,
+    "particle_momentum_x": 1.2614870751260742e-23,
+    "particle_momentum_y": 1.539223405828654e-27,
+    "particle_momentum_z": 1.3694904126667137e-18,
+    "particle_position_x": 0.017718774893004597,
     "particle_position_y": 0.02062006553416991,
-    "particle_weight": 677858497765697.2
+    "particle_weight": 727797380314273.1
   },
   "plasma_p": {
     "particle_cpu": 504.0,
     "particle_id": 272412.0,
-    "particle_momentum_x": 1.261489389649524e-23,
-    "particle_momentum_y": 1.539254935054062e-27,
-    "particle_momentum_z": 2.514586739415174e-15,
-    "particle_position_x": 0.017718749996674485,
-    "particle_position_y": 0.02062006546321032,
-    "particle_weight": 677858497765697.2
+    "particle_momentum_x": 1.2614841860109576e-23,
+    "particle_momentum_y": 1.5392492676789103e-27,
+    "particle_momentum_z": 2.5145867394151743e-15,
+    "particle_position_x": 0.017718749986442628,
+    "particle_position_y": 0.02062006546318015,
+    "particle_weight": 727797380314273.1
   }
 }

Regression/Checksum/benchmarks_json/PlasmaAccelerationBoost3d.json

@@ -2,7 +2,7 @@
   "beam": {
     "particle_cpu": 0.0,
     "particle_id": 1500500.0,
-    "particle_momentum_x": 4.178482505909376e-19,
+    "particle_momentum_x": 4.1784825059093756e-19,
     "particle_momentum_y": 4.56492260137707e-19,
     "particle_momentum_z": 2.733972888170628e-17,
     "particle_position_x": 0.0003995213395426269,
@@ -13,11 +13,11 @@
   "driver": {
     "particle_cpu": 0.0,
     "particle_id": 500500.0,
-    "particle_momentum_x": 4.700436405078562e+21,
-    "particle_momentum_y": 4.678586211309308e+21,
+    "particle_momentum_x": 4.7004364050785617e+21,
+    "particle_momentum_y": 4.6785862113093076e+21,
     "particle_momentum_z": 2.999596093018443e+25,
-    "particle_position_x": 0.001581171564560721,
-    "particle_position_y": 0.001621241612001656,
+    "particle_position_x": 0.0015811715645607214,
+    "particle_position_y": 0.0016212416120016558,
     "particle_position_z": 0.3737871668388371,
     "particle_weight": 6241509074.460762
   },
@@ -27,42 +27,42 @@
     "particle_momentum_x": 4.813131349021333e+21,
     "particle_momentum_y": 5.165480740901231e+21,
     "particle_momentum_z": 3.005830430844926e+25,
-    "particle_position_x": 0.001648993262358345,
-    "particle_position_y": 0.001618626767559713,
-    "particle_position_z": 0.4203715138122838,
+    "particle_position_x": 0.0016489932623583448,
+    "particle_position_y": 0.0016186267675597135,
+    "particle_position_z": 0.42037151381228377,
     "particle_weight": 6241509074.460762
   },
   "lev=0": {
     "Bx": 44900.36241701626,
-    "By": 44925.149756099,
+    "By": 44925.149756098996,
     "Bz": 6.522149557730263,
-    "Ex": 1740395711918.568,
-    "Ey": 1739383965652.243,
-    "Ez": 15904996412605.86,
-    "jx": 181761783637.0391,
-    "jy": 211922069538.1537,
-    "jz": 1808517835456762.0
+    "Ex": 1740395711918.5684,
+    "Ey": 1739383965652.2427,
+    "Ez": 15904996412605.855,
+    "jx": 181761783637.03912,
+    "jy": 211922069538.15366,
+    "jz": 1808517835456761.5
   },
   "plasma_e": {
     "particle_cpu": 3600.0,
     "particle_id": 9721800.0,
-    "particle_momentum_x": 1.252095679647391e-25,
+    "particle_momentum_x": 1.2520956796473912e-25,
     "particle_momentum_y": 1.255131061786229e-25,
     "particle_momentum_z": 9.782048215409952e-18,
-    "particle_position_x": 0.126562500284578,
-    "particle_position_y": 0.1265625002853938,
-    "particle_position_z": 0.06789191625822297,
-    "particle_weight": 3869115015.619221
+    "particle_position_x": 0.12656250028457797,
+    "particle_position_y": 0.12656250028539381,
+    "particle_position_z": 0.06789191622060867,
+    "particle_weight": 4834401345.035692
   },
   "plasma_p": {
     "particle_cpu": 3600.0,
     "particle_id": 16201800.0,
-    "particle_momentum_x": 1.252095682124398e-25,
-    "particle_momentum_y": 1.255131063319169e-25,
-    "particle_momentum_z": 1.7961333877559242e-14,
-    "particle_position_x": 0.12656249999772107,
-    "particle_position_y": 0.12656249999790195,
-    "particle_position_z": 0.06789191621901314,
-    "particle_weight": 3869115015.619221
+    "particle_momentum_x": 1.2520956821243977e-25,
+    "particle_momentum_y": 1.2551310633191694e-25,
+    "particle_momentum_z": 1.796133387912909e-14,
+    "particle_position_x": 0.12656249999984504,
+    "particle_position_y": 0.12656249999984454,
+    "particle_position_z": 0.06789191621903365,
+    "particle_weight": 4834401345.035692
   }
 }

Regression/Checksum/benchmarks_json/background_mcc.json

@@ -1,26 +1,26 @@
 {
   "electrons": {
-    "particle_cpu": 190805.0,
-    "particle_id": 58572159554.0,
-    "particle_momentum_x": 1.0096555379894353e-18,
-    "particle_momentum_y": 2.792308861947816e-19,
-    "particle_momentum_z": 2.7912608371353956e-19,
-    "particle_position_x": 17158.992965684767,
-    "particle_position_y": 941.2938975939338,
-    "particle_weight": 61532370208.74024
+    "particle_cpu": 187358.0,
+    "particle_id": 58199703731.0,
+    "particle_momentum_x": 1.0141001808730319e-18,
+    "particle_momentum_y": 2.8070977171592375e-19,
+    "particle_momentum_z": 2.8029049329565064e-19,
+    "particle_position_x": 17128.025846186716,
+    "particle_position_y": 933.6068577115554,
+    "particle_weight": 61098512084.960945
   },
   "he_ions": {
-    "particle_cpu": 262726.0,
-    "particle_id": 206636597294.0,
-    "particle_momentum_x": 2.8337541918025354e-18,
-    "particle_momentum_y": 2.194599746445289e-18,
-    "particle_momentum_z": 2.1971043484994987e-18,
-    "particle_position_x": 17599.236953657848,
-    "particle_position_y": 1099.5894721489742,
-    "particle_weight": 71945924133.3008
+    "particle_cpu": 262797.0,
+    "particle_id": 206713337554.0,
+    "particle_momentum_x": 2.887261723355975e-18,
+    "particle_momentum_y": 2.1945524403708785e-18,
+    "particle_momentum_z": 2.1982471595658953e-18,
+    "particle_position_x": 17605.41372853014,
+    "particle_position_y": 1099.8462212522177,
+    "particle_weight": 71965788177.49025
   },
   "lev=0": {
-    "rho_electrons": 0.03590777765584312,
-    "rho_he_ions": 0.041907813459370793
+    "rho_electrons": 0.035651817593500675,
+    "rho_he_ions": 0.04191959356643796
   }
 }