Memaster update with upstream/development (#106)

* Add Python Wrappers for F,G in PML (BLAST-WarpX#2481)

* Add Python Wrappers for F,G in PML

* Add Getters for F,G Nodal Flags

* Fix Bug in <F,G>FPPMLWrapper (Default Level)

* Fix Bug in F,G Nodal Flags

* Use GetPML Method for F,G Nodal Flags

* PICMI: Add max_grid_size, blocking_factor in (x,y,z) (BLAST-WarpX#2524)

* PICMI: Add amr.max_grid_size_<x,y,z>

* Set All Flags in Python

* PICMI: Add amr.blocking_factor_<x,y,z>

* added wrappers to get particle structs for the particles in the boundary buffers (BLAST-WarpX#2498)

* Doxygen: Fix Docs (BLAST-WarpX#2526)

* Fix Bug with Tilebox for G in PML (BLAST-WarpX#2527)

* Fix Bug with Tilebox for G in PML

* Reset Benchmark

* AMReX/PICSAR: Weekly Update (BLAST-WarpX#2533)

Weekly update to latest AMReX.
Weekly update to latest PICSAR (no changes).

```
./Tools/Release/updatePICSAR.py
./Tools/Release/updateAMReX.py
```

* Install pre-commit (BLAST-WarpX#2532)

* Add pre-commit

Add basis for automated pre-commit checks.
Install locally via:

```bash
python3 -m pip install -U pre-commit
pre-commit install
```

See: https://pre-commit.com

* Cleanup: Whitespaces

* Cleanup: requirements.txt order

* Summit: Update Numpy Hints (BLAST-WarpX#2535)

Make sure `numpy` can be rebuilt when and were needed.
To achieve that, move numpy-specific installation hints on OpenBLAS
to the WarpX profile.

* Fix some issues with Fujitsu compiler (BLAST-WarpX#2529)

* make some code compilable with Fujitsu compiler in clang mode

* update documentation

* Fix BLAST-WarpX#2522: Gaussian beam positions do not change with warpx.random_seed (BLAST-WarpX#2523)

* Draw Gaussian beam position with amrex random engine

* Update benchmarks

* Update tolerance in space-charge tests

* Update benchmark for space charge initialization test

* Update benchmarks

* Update benchmark

* Clean-up code

* Update benchmarks

* BLAST-WarpX#2534: Don't access position vector values beyond the configured dimension (BLAST-WarpX#2536)

* BLAST-WarpX#2534: Don't access position vector values beyond the configured dimension

* Fix particle position component used in XZ configuration

* Handle 1D case

* Move values only used in scraping function into inside-boundary condition

* Error out if scraping from EB in RZ

* Spack Development: macOS & GNUmake (BLAST-WarpX#2545)

- Add macOS hints for OpenMP
- Add hints for running GNUmake regression tests locally

* Regression Tests: OMP on (BLAST-WarpX#2548)

We generally run only with one OpenMP threads at the moment, but
disabling OpenMP altogether causes an extra compile, which slows down
CI.

* Tests: numthreads to 1 (BLAST-WarpX#2546)

* Tests: numthreads to 1

We already hack this option to read `numthreads = 1` already for
benchmarks, thus we remove the confusing other values now.

* Prepare for CI: Do not Overwrite `numthreads`

* std::ifstream: Defensive Patterns (BLAST-WarpX#2547)

Add failure handling if inputs in `std::ifstream`s cannot be opened
or have problems seek-ing through them.

This should catch I/O errors early.

* openPMD: 0.14.3 (BLAST-WarpX#2551)

Automatically copy and compile openPMD-api 0.14.3, but still supporting the 0.14.2+ range (BLAST-WarpX#2150).

The 0.14.3 release solves ABI incompatibilities in C++14/17 mixed builds, among other issues (mainly read).

* Add Ar and Xe to pre-defined particle types. (BLAST-WarpX#2549)

* Added Ar and Xe to pre-defined particle types

* Added Boltzmann's constant to warpx parser

* Updated documentation

* 2D EM solver with EB (BLAST-WarpX#2401)

* adding the FieldProbe

* adding missing file

* updating makefile

* fixing host-device problem

* Revert "fixing host-device problem"

This reverts commit 801e6fc.

* fixing host-device problem

* making some variables const

* adding a few comments

* Adding the FieldProbe to the documentation

* making the probe mpi-safe

* added field probe to reduced diag test

* added field probe to reduced diag analysis

* using cell-centered fields in probe diag

* removed a few typos

* Interpolating to the point instead oof cell center

* bug fix

* improved a comment

* updated documentation

* Undone an outdated change

* improving some variable names

* improving the box extraction

* making the interpolation order an input parameter

* fix a typo

* setting the field values to zero if the point is not in the domain

* skipping the communication if probe proc is IO prcessor

* Fixed typo in documentation

Co-authored-by: Neïl Zaim <[email protected]>

* Updating an header

* Added a comment on the probe position

* tidying up the analysis script

* fixed a comment

* removing an unused include

* improving the parsing of parameters

* fixing some comments

* making some variables const

* changed some ParticleReal into Real

* using better tags in MPI communication

* Making field probe work in 2D

* making a variable const

* initializing y_probe only in 3D

* tidying up a line which is common to 2D and 3D

* making a variable constexpr

* adding a _rt

* checking that the probe location is in one of the processors

* removing a useless if condition

* Fixing the initialization in 2D

* Avoiding scrape particles in 2D (it segfaults)

* Adding a test for 2D EB

* Fixed the areas initialization

* Initializing to zero some multifabs

* Modified the ECT solver to make it work in 2D

* Modified the cell extensions to make them work in 2D

* Improved 2D cube test

* Added 2D rotated cube test

* Adding the 2d analysis script and CI

* Removed an unused import from the analysis script

* Ignoring some unused variables

* Fixing the number of dimensions in the 2d test

* Added missing analysis for ECT

* Enabled again 2d particles scraping

* Fixing the test_name with the general logic

* Fixing the test_name with the general logic

* Removed some commented code

* Modified several preprocessor directives to check consistency EB-dimension

* Added missing semicolons

* Fixed a preprocessor directivew

* Fix typo: WARPX_DIM_XZ

* Improving some comments

Co-authored-by: Axel Huebl <[email protected]>

* Adding some more consistency checks

* Adding some more consistency checks

* Fixed a typo

Co-authored-by: Neïl Zaim <[email protected]>
Co-authored-by: Axel Huebl <[email protected]>

* Fix Instability in PML with PSATD (BLAST-WarpX#2558)

* Fix Instability in PML with PSATD

Damping in PML should be applied before the communications between the regular grids and PML, and between PML grids take place, otherwise the ghost cells are filled with lagged information, which results in an instability. Closes BLAST-WarpX#2525.

* Update checksum of the pml_psatd_dive_divb_cleaning test

* Bugfix in load balancing routine (BLAST-WarpX#2555)

* add remake of phi_fp during load balancing RemakeLevel

* added phi_cp remake to RemakeLevel function

* revert changes from previous commit

* I/O performance hints for Summit (BLAST-WarpX#2495)

* Fix conflict with upstream

* Apply suggestions from code review

* Remove space in the end of lines

* Include suggestions from PR review

* Generalize ROMIO Hints in Batch Scripts

* Fix Comment

* Fix Comment

* Remove duplication

* Formatting

Co-authored-by: Axel Huebl <[email protected]>

* AMReX/PICSAR: Weekly Update (BLAST-WarpX#2559)

* AMReX: Weekly Update

* PICSAR: Weekly Update

* Docs: Fix .rst Label in PML, rm .tex (BLAST-WarpX#2537)

Fix an auto-converted label in a `.rst` file for the manual.
Remove the `PML.tex` file.

* Add 2D circle EB test (BLAST-WarpX#2538)

* Added embedded_circle test

* Add embedded_circle test files

* Removed diag files

* removed PICMI input file

* Update to use default regression analysis

* Added line breaks for spacing

* Added description

* Fixed benchmark file

* Added load balancing to test

* Commented out load_balancing portion of test.
This will be added back in once load balancing is fixed.

Co-authored-by: Axel Huebl <[email protected]>

* Fixes to the EB init (BLAST-WarpX#2565)

* Avoid code duplications in ECT face extension (BLAST-WarpX#2557)

* Refactoring the nborrow functions

* Refactoring the one cell extension

* Refactoring the eight cells extension

* Enabling 2D

* Bug fix

* Some more improvements

* Fixing templates

* Switching the order of templates and AMREX_GPU_DEVICE

* Adding the needed AMREX_GPU_DEVICE in WarpX.H

* Fixing GPU related issues

* Fixed a for loop bound

* Making the new functions free

* Suggestion from review

* Suggestion from review

Co-authored-by: Axel Huebl <[email protected]>

* Suggestion from review

Co-authored-by: Axel Huebl <[email protected]>

* Improve loops over dimensions for 2D

Co-authored-by: Axel Huebl <[email protected]>

* Enhanced inline documentation of EB related data (BLAST-WarpX#2562)

* Enhanced inline documentation of EB related data

* Added ECT to the glossary

* Made the EB documentation doxygen-compatible

* Clean up input files for tests with MCC (BLAST-WarpX#2552)

* Added embedded_circle test

* Add embedded_circle test files

* Removed diag files

* removed PICMI input file

* Update to use default regression analysis

* Added line breaks for spacing

Co-authored-by: Axel Huebl <[email protected]>

* Added description

* Added Ar and Xe to pre-defined particle types

* added Boltzmann's constant to pre-defined constants and cleaned up the MCC CI test input

* Added Boltzmann's constant to warpx parser

* cleaned up embedded circle CI test input

* Remove duplicate entry.

Co-authored-by: kzhu-ME <[email protected]>
Co-authored-by: Kevin Z. Zhu <[email protected]>
Co-authored-by: Axel Huebl <[email protected]>

* Cell Center Macroscopic Properties (BLAST-WarpX#2530)

* Cell Center Macroscopic Properties

* Commit Suggestions from PR Review

* Fix Error for 2D

Co-authored-by: Edoardo Zoni <[email protected]>

* added superLU solver example to docs (BLAST-WarpX#2567)

* Fix: GNUmake Python Link -g (BLAST-WarpX#2568)

On CPU links of the GNUmake Python lib, we forgot our `-g`, which we
add to all build and optimization types. THis is part of the
`LINKFLAGS` variable.

* Apply PEC to Split PML Fields (BLAST-WarpX#2541)

* WarpXMovingWindow.cpp: Add `amrex::` Prefix (BLAST-WarpX#2579)

* Bug fixes and cleanup in load balancing (BLAST-WarpX#2563)

* added helper function to rebuild MultiFabs and iMultiFabs during load balancing and included rebuilding of EB multifabs

* added redistribute call for the particle boundary buffer during load balancing

* consistently use DistribtionMap rather than dmap in ElectrostaticSolver.cpp

* applied suggested changes from code review by Phil Miller

* removed default argument for redistribute in RemakeMultiFab

* removed RemakeMultiFab() as a member of WarpX

* Only remake EB multifabs if they are used

Co-authored-by: Lorenzo Giacomel <[email protected]>

* adapted existing particle scraping test (PICMI version) to also cover the redistribution of particle buffers from load balancing

* added redeclaring of m_borrowing

* Move redeclaring of m_borrow inside if statement for ECT solver algorihtm

Co-authored-by: Lorenzo Giacomel <[email protected]>

* added calls to MarkCells and ComputeFaceExtensions

* fixed issue causing CI test to fail and copied conditionals from WarpXInitData.cpp to recompute EB quantities

* Guard cells communication for EB data when re-gridding (#105)

* Add 2D circle EB test (BLAST-WarpX#2538)

* Added embedded_circle test

* Add embedded_circle test files

* Removed diag files

* removed PICMI input file

* Update to use default regression analysis

* Added line breaks for spacing

* Added description

* Fixed benchmark file

* Added load balancing to test

* Commented out load_balancing portion of test.
This will be added back in once load balancing is fixed.

Co-authored-by: Axel Huebl <[email protected]>

* Added guard cells communication for EB data in regridding

Co-authored-by: Kevin Z. Zhu <[email protected]>
Co-authored-by: Axel Huebl <[email protected]>

* moved all EB grid data calculations to a new function InitializeEBGridData() which is now called by both WarpX::InitLevelData and WarpX::RemakeLevel

* Fix typo in doc string.

Co-authored-by: Phil Miller <[email protected]>

Co-authored-by: Lorenzo Giacomel <[email protected]>
Co-authored-by: Kevin Z. Zhu <[email protected]>
Co-authored-by: Axel Huebl <[email protected]>
Co-authored-by: Phil Miller <[email protected]>

* WarpXComm.cpp: Use Explicit Types, not `auto` (BLAST-WarpX#2578)

* WarpXComm.cpp: Use Explicit Types, not `auto`

* Use MultiFab* const instead of MultiFab* const&

* Add WARPX_PROFILE calls to each python callback. (BLAST-WarpX#2573)

When python callbacks take some time, this is useful as otherwise many
callbacks are lumped together in WarpX::Evolve::step.

* LaserInjectionFromTXYEFile Test: Use MPI (BLAST-WarpX#2577)

* WarpX tests: All MPI

Enable MPI for the one regression test that does not use it.
Still uses one rank there.

This saves a compile per CI run.

* add numprocs

* Bug fix in postprocessing yt data and other small changes (#104)

* fixed bug in post-processing of field diagnostic yt data

* fixed issue causing post processing of field diagnostics unit test to fail

* added helper function to rebuild MultiFabs and iMultiFabs during load balancing and included rebuilding of EB multifabs

* added redistribute call for the particle boundary buffer during load balancing

* consistently use DistribtionMap rather than dmap in ElectrostaticSolver.cpp

* applied suggested changes from code review by Phil Miller

* removed default argument for redistribute in RemakeMultiFab

* added load balance intervals as an optional input parameter to diode_setup.py

* removed RemakeMultiFab() as a member of WarpX

* Only remake EB multifabs if they are used

Co-authored-by: Lorenzo Giacomel <[email protected]>

* changed plot_contours grid setting parameter from b to visible in accordance with matplotlib 3.5 changes

* Revert change to WarpXRegrid.cpp

* Suggested change from PS during code review

Co-authored-by: Peter Scherpelz <[email protected]>

Co-authored-by: Lorenzo Giacomel <[email protected]>
Co-authored-by: Peter Scherpelz <[email protected]>

Co-authored-by: Edoardo Zoni <[email protected]>
Co-authored-by: Axel Huebl <[email protected]>
Co-authored-by: Luca Fedeli <[email protected]>
Co-authored-by: Remi Lehe <[email protected]>
Co-authored-by: Phil Miller <[email protected]>
Co-authored-by: Kevin Z. Zhu <[email protected]>
Co-authored-by: Lorenzo Giacomel <[email protected]>
Co-authored-by: Neïl Zaim <[email protected]>
Co-authored-by: Weiqun Zhang <[email protected]>
Co-authored-by: Jean Luca Bez <[email protected]>
Co-authored-by: kzhu-ME <[email protected]>
Co-authored-by: Revathi  Jambunathan <[email protected]>
Co-authored-by: Phil Miller <[email protected]>
Co-authored-by: Peter Scherpelz <[email protected]>

Author: 13 people

.azure-pipelines.yml

@@ -58,7 +58,7 @@ jobs:
       sudo curl -L -o /usr/local/bin/cmake-easyinstall https://git.io/JvLxY
       sudo chmod a+x /usr/local/bin/cmake-easyinstall
       if [ "${WARPX_CI_OPENPMD:-FALSE}" == "TRUE" ]; then
-        cmake-easyinstall --prefix=/usr/local git+https://github.com/openPMD/[email protected].2 \
+        cmake-easyinstall --prefix=/usr/local git+https://github.com/openPMD/[email protected].3 \
           -DopenPMD_USE_PYTHON=OFF -DBUILD_TESTING=OFF -DBUILD_EXAMPLES=OFF -DBUILD_CLI_TOOLS=OFF
         python -m pip install --upgrade openpmd-api
       fi

.github/workflows/cuda.yml

@@ -24,7 +24,7 @@ jobs:
       run: |
         .github/workflows/dependencies/nvcc11.sh
         export CEI_SUDO="sudo"
-        cmake-easyinstall --prefix=/usr/local git+https://github.com/openPMD/[email protected].2 -DopenPMD_USE_PYTHON=OFF -DBUILD_TESTING=OFF -DBUILD_EXAMPLES=OFF -DBUILD_CLI_TOOLS=OFF
+        cmake-easyinstall --prefix=/usr/local git+https://github.com/openPMD/[email protected].3 -DopenPMD_USE_PYTHON=OFF -DBUILD_TESTING=OFF -DBUILD_EXAMPLES=OFF -DBUILD_CLI_TOOLS=OFF
     - name: build WarpX
       run: |
         export PATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:${PATH}

.github/workflows/dependencies/icc.sh

@@ -39,7 +39,7 @@ export CEI_SUDO="sudo"
 CXX=$(which icpc) CC=$(which icc) \
   cmake-easyinstall               \
   --prefix=/usr/local             \
-  git+https://github.com/openPMD/[email protected].2 \
+  git+https://github.com/openPMD/[email protected].3 \
   -DopenPMD_USE_PYTHON=OFF \
   -DBUILD_TESTING=OFF      \
   -DBUILD_EXAMPLES=OFF     \

Docs/source/glossary.rst

@@ -18,6 +18,7 @@ Abbreviations
 * **CI:** continuous integration, automated tests that we perform before a proposed code-change is accepted; see PR
 * **CPU:** `central processing unit <https://en.wikipedia.org/wiki/Central_processing_unit>`__, we usual mean a socket or generally the host-side of a computer (compared to the accelerator, e.g. GPU)
 * **DOE:** `The United States Department of Energy <https://en.wikipedia.org/wiki/United_States_Department_of_Energy>`__, the largest sponsor of national laboratory research in the United States of America
+* **ECT:** `Enlarged Cell Technique <https://ieeexplore.ieee.org/document/4463918>`__, an electromagnetic solver with accurate resolution of perfectly conducting embedded boundaries
 * **EB:** embedded boundary, boundary conditions inside the simulation box, e.g. following material surfaces
 * **EM:** electromagnetic, e.g. EM PIC
 * **ES:** electrostatic, e.g. ES PIC

Docs/source/install/cmake.rst

@@ -108,7 +108,7 @@ CMake Option                  Default & Values                               Des
 ``WarpX_amrex_internal``      **ON**/OFF                                     Needs a pre-installed AMReX library if set to ``OFF``
 ``WarpX_openpmd_src``         *None*                                         Path to openPMD-api source directory (preferred if set)
 ``WarpX_openpmd_repo``        ``https://github.com/openPMD/openPMD-api.git`` Repository URI to pull and build openPMD-api from
-``WarpX_openpmd_branch``      ``0.14.2``                                     Repository branch for ``WarpX_openpmd_repo``
+``WarpX_openpmd_branch``      ``0.14.3``                                     Repository branch for ``WarpX_openpmd_repo``
 ``WarpX_openpmd_internal``    **ON**/OFF                                     Needs a pre-installed openPMD-api library if set to ``OFF``
 ``WarpX_picsar_src``          *None*                                         Path to PICSAR source directory (preferred if set)
 ``WarpX_picsar_repo``         ``https://github.com/ECP-WarpX/picsar.git``    Repository URI to pull and build PICSAR from

Docs/source/install/hpc/summit.rst

@@ -234,6 +234,52 @@ parameters provided good performance:
 * **Sixteen `64x64x64` grids per MPI rank** (with default tiling in WarpX, this
   results in ~49 tiles per OpenMP thread)
 
+.. _building-summit-io-performance:
+
+I/O Performance Tuning
+----------------------
+
+.. _building-summit-large-blocks:
+
+GPFS Large Block I/O
+^^^^^^^^^^^^^^^^^^^^
+
+Setting ``IBM_largeblock_io`` to ``true`` disables data shipping, saving overhead when writing/reading large contiguous I/O chunks.
+
+.. code-block:: bash
+
+   export IBM_largeblock_io=true
+
+.. _building-summit-romio-hints:
+
+ROMIO MPI-IO Hints
+^^^^^^^^^^^^^^^^^^
+
+You might notice some parallel HDF5 performance improvements on Summit by setting the appropriate ROMIO hints for MPI-IO operations.
+
+.. code-block:: bash
+
+   export OMPI_MCA_io=romio321
+   export ROMIO_HINTS=./romio-hints
+
+You can generate the ``romio-hints`` by issuing the following command. Remember to change the number of ``cb_nodes`` to match the number of compute nodes you are using (example here: ``64``).
+
+.. code-block:: bash
+
+   cat > romio-hints << EOL
+   romio_cb_write enable
+   romio_ds_write enable
+   cb_buffer_size 16777216
+   cb_nodes 64
+   EOL
+
+The ``romio-hints`` file contains pairs of key-value hints to enable and tune collective
+buffering of MPI-IO operations. As Summit's Alpine file system uses a 16MB block size,
+you should set the collective buffer size to 16GB and tune the number of aggregators
+(``cb_nodes``) to the number of compute nodes you are using, i.e., one aggregator per node.
+
+Further details are available at `Summit's documentation page <https://docs.olcf.ornl.gov/systems/summit_user_guide.html#slow-performance-using-parallel-hdf5-resolved-march-12-2019>`__.
+
 .. _building-summit-issues:
 
 Known System Issues

Docs/source/latex_theory/Makefile

@@ -1,13 +1,11 @@
 SRC_FILES = theory.tex \
-            AMR/AMR.tex \
-	    PML/PML.tex
+            AMR/AMR.tex
 
 all: $(SRC_FILES) clean
 	pandoc intro.tex --mathjax --wrap=preserve --bibliography allbibs.bib -o intro.rst
 	pandoc AMR/AMR.tex --mathjax --wrap=preserve --bibliography allbibs.bib -o amr.rst
 	pandoc Boosted_frame/Boosted_frame.tex --mathjax --wrap=preserve --bibliography allbibs.bib -o boosted_frame.rst
 	pandoc input_output/input_output.tex --mathjax --wrap=preserve --bibliography allbibs.bib -o input_output.rst
-	pandoc PML/PML.tex --mathjax --wrap=preserve --bibliography allbibs.bib -o PML.rst
 	mv *.rst ../theory
 	cd ../../../../picsar/Doxygen/pages/latex_theory/; pandoc theory.tex --mathjax --wrap=preserve --bibliography allbibs.bib -o picsar_theory.rst
 	mv ../../../../picsar/Doxygen/pages/latex_theory/picsar_theory.rst ../theory

Docs/source/latex_theory/PML/PML.tex

@@ -32,7 +32,7 @@
 \input{AMR/AMR.tex}
 
 %%%%%\input{Particle_pushers/Vay_pusher.tex}
-\input{PML/PML.tex}
+%\input{PML/PML.tex}
 
 \input{Boosted_frame/Boosted_frame.tex}