Skip to content

Commit f691736

Browse files
vicenteboleavicentebolea
authored
Merge pull request #4184 from vicentebolea/backports-from-master
Backports from master
2 parents c2a9464 + 260166d commit f691736

File tree

37 files changed

+2243
-703
lines changed

37 files changed

+2243
-703
lines changed

‎bindings/C/adios2/c/adios2_c_io.h‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -329,8 +329,8 @@ adios2_error adios2_remove_all_attributes(adios2_io *io);
329329
* MPI Collective function as it calls MPI_Comm_dup
330330
* @param io engine owner
331331
* @param name unique engine identifier
332-
* @param mode adios2_mode_write, adios2_mode_read, adios2_mode_append, and
333-
* adios2_mode_readRandomAccess
332+
* @param mode adios2_mode_write, adios2_mode_read, adios2_mode_append
333+
* and adios2_mode_readRandomAccess
334334
* @return success: handler, failure: NULL
335335
*/
336336
adios2_engine *adios2_open(adios2_io *io, const char *name, const adios2_mode mode);
@@ -341,7 +341,7 @@ adios2_engine *adios2_open(adios2_io *io, const char *name, const adios2_mode mo
341341
* MPI Collective function as it calls MPI_Comm_dup
342342
* @param io engine owner
343343
* @param name unique engine identifier
344-
* @param mode adios2_mode_write, adios2_mode_read, adios2_mode_append, and
344+
* @param mode adios2_mode_write, adios2_mode_read, adios2_mode_append and
345345
* adios2_mode_readRandomAccess
346346
* @param comm communicator other than adios' handler comm. MPI only.
347347
* @return success: handler, failure: NULL

‎bindings/CXX11/adios2/cxx11/Variable.h‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -214,7 +214,7 @@ class Variable
214214
* variable.Count() = {Ny,Nx}, then memoryCount = {Ny+2,Nx+2}
215215
* </pre>
216216
*/
217-
void SetMemorySelection(const adios2::Box<adios2::Dims> &memorySelection);
217+
void SetMemorySelection(const adios2::Box<adios2::Dims> &memorySelection = {{}, {}});
218218

219219
/**
220220
* Sets a step selection modifying current startStep, countStep

‎docs/user_guide/source/components/anatomy.rst‎

Lines changed: 36 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -114,7 +114,10 @@ named `adios2::Mode::ReadRandomAccess`. `adios2::Mode::Read` mode allows data ac
114114
current timestep. `ReadRandomAccess` can only be used with file engines and involves loading all the file metadata at
115115
once. So it can be more memory intensive than `adios2::Mode::Read` mode, but allows reading data from any timestep using
116116
`SetStepSelection()`. If you use `adios2::Mode::ReadRandomAccess` mode, be sure to allocate enough memory to hold
117-
multiple steps of the variable content.
117+
multiple steps of the variable content. Note that ADIOS streaming
118+
engines (like SST, DataMan, etc.) do not support `ReadRandomAccess`
119+
mode. Also newer file Engines like BP5 to not allow
120+
`BeginStep/EndStep` calls in `ReadRandomAccess` mode.
118121

119122
.. code:: C++
120123

@@ -134,3 +137,35 @@ multiple steps of the variable content.
134137
| |--> IO goes out of scope
135138
|
136139
|--> ADIOS goes out of scope or adios2_finalize()
140+
141+
Previously we explored how to read using the input mode `adios2::Mode::Read`. Nonetheless, ADIOS has another input mode
142+
named `adios2::Mode::ReadRandomAccess`. `adios2::Mode::Read` mode allows data access only timestep by timestep using
143+
`BeginStep/EndStep`, but generally it is more memory efficient as ADIOS is only required to load metadata for the
144+
current timestep. `ReadRandomAccess` can only be used with file engines and involves loading all the file metadata at
145+
once. So it can be more memory intensive than `adios2::Mode::Read` mode, but allows reading data from any timestep using
146+
`SetStepSelection()`. If you use `adios2::Mode::ReadRandomAccess` mode, be sure to allocate enough memory to hold
147+
multiple steps of the variable content. Note that ADIOS streaming
148+
engines (like SST, DataMan, etc.) do not support `ReadRandomAccess`
149+
mode. Also newer file Engines like BP5 to not allow
150+
`BeginStep/EndStep` calls in `ReadRandomAccess` mode.
151+
152+
.. code:: C++
153+
154+
ADIOS adios("config.xml", MPI_COMM_WORLD);
155+
|
156+
| IO io = adios.DeclareIO(...);
157+
| |
158+
| | Engine e = io.Open("InputFileName.bp", adios2::Mode::ReadRandomAccess);
159+
| | |
160+
| | | Variable var = io.InquireVariable(...)
161+
| | | | var.SetStepSelection()
162+
| | | | e.Get(var, datapointer);
163+
| | | |
164+
| | |
165+
| | e.Close();
166+
| |
167+
| |--> IO goes out of scope
168+
|
169+
|--> ADIOS goes out of scope or adios2_finalize()
170+
171+

‎docs/user_guide/source/components/io.rst‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -196,7 +196,7 @@ A particular ``Engine`` type is set to the current ``IO`` component with the ``I
196196
Engine polymorphism is handled internally by the ``IO`` class, which allows subclassing future derived ``Engine`` types without changing the basic API.
197197

198198
``Engine`` objects are created in various modes.
199-
The available modes are ``adios2::Mode::Read``, ``adios2::Mode::Write``, ``adios2::Mode::Append``, ``adios2::Mode::Sync``, ``adios2::Mode::Deferred``, and ``adios2::Mode::Undefined``.
199+
The available modes are ``adios2::Mode::Read``, ``adios2::Mode::ReadRandomAccess``, ``adios2::Mode::Write``, ``adios2::Mode::Append``, ``adios2::Mode::Sync``, ``adios2::Mode::Deferred``, and ``adios2::Mode::Undefined``.
200200

201201

202202
.. code-block:: c++

‎docs/user_guide/source/engines/bp5.rst‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -130,6 +130,14 @@ This engine allows the user to fine tune the buffering operations through the fo
130130

131131
#. **Threads**: Read side: Specify how many threads one process can use to speed up reading. The default value is *0*, to let the engine estimate the number of threads based on how many processes are running on the compute node and how many hardware threads are available on the compute node but it will use maximum 16 threads. Value *1* forces the engine to read everything within the main thread of the process. Other values specify the exact number of threads the engine can use. Although multithreaded reading works in a single *Get(adios2::Mode::Sync)* call if the read selection spans multiple data blocks in the file, the best parallelization is achieved by using deferred mode and reading everything in *PerformGets()/EndStep()*.
132132

133+
#. **FlattenSteps**: This is a writer-side parameter specifies that the
134+
reader should interpret multiple writer-created timesteps as a
135+
single timestep, essentially flattening all Put()s into a single step.
136+
137+
#. **IgnoreFlattenSteps**: This is a reader-side parameter that
138+
tells the reader to ignore any FlattenSteps parameter supplied
139+
to the writer.
140+
133141
============================== ===================== ===========================================================
134142
**Key** **Value Format** **Default** and Examples
135143
============================== ===================== ===========================================================
@@ -156,6 +164,8 @@ This engine allows the user to fine tune the buffering operations through the fo
156164
StatsLevel integer, 0 or 1 **1**, 0
157165
MaxOpenFilesAtOnce integer >= 0 **UINT_MAX**, 1024, 1
158166
Threads integer >= 0 **0**, 1, 32
167+
FlattenSteps boolean **off**, on, true, false
168+
IgnoreFlattenSteps boolean **off**, on, true, false
159169
============================== ===================== ===========================================================
160170

161171

‎docs/user_guide/source/setting_up/source/cmake.rst‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ To build ADIOS v2.x, clone the repository and invoke the canonical CMake build s
88
99
$ git clone https://github.com/ornladios/ADIOS2.git ADIOS2
1010
$ mkdir adios2-build && cd adios2-build
11-
$ cmake ../ADIOS2 cmake -DADIOS2_BUILD_EXAMPLES=ON
11+
$ cmake ../ADIOS2 -DADIOS2_BUILD_EXAMPLES=ON
1212
-- The C compiler identification is GNU 9.4.0
1313
-- The CXX compiler identification is GNU 9.4.0
1414
...
@@ -111,10 +111,11 @@ Optionally, run the tests (need to configure with ``-DBUILD_TESTING=ON`` cmake f
111111
Total Test time (real) = 95.95 sec
112112
113113
114-
And finally, use the standard invocation to install:
114+
And finally, use the standard invocation to install (setting the install path beforehand):
115115

116116
.. code-block:: bash
117117
118+
$ cmake ../ADIOS2 -DCMAKE_INSTALL_PREFIX=/path/to/where/adios/will/be/installed
118119
$ make install
119120
120121

‎docs/user_guide/source/tutorials/helloWorld.rst‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -86,14 +86,15 @@ Start editing the skeleton file `ADIOS2/examples/hello/helloWorld/hello-world_tu
8686

8787
adios2::Engine reader = io.Open("hello-world-cpp.bp", adios2::Mode::Read);
8888
std::string greeting;
89+
reader.BeginStep();
8990
reader.Get(varGreeting, greeting);
9091
reader.EndStep();
9192
reader.Close();
9293
return greeting;
9394

9495
.. note::
9596

96-
The ``BeginStep`` and ``EndStep`` calls are required when **reading** one step and multiple steps. We will see in
97+
In Mode::Read, the ``BeginStep`` and ``EndStep`` calls are required when **reading** one step and multiple steps. We will see in
9798
another tutorial how to read multiple steps. It's important to note that the ``BeginStep`` should be called **before**
9899
all ``Inquire*`` / ``Available*`` function calls.
99100

‎source/adios2/common/ADIOSMacros.h‎

Lines changed: 20 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@
3333
</pre>
3434
*/
3535
#define ADIOS2_FOREACH_ATTRIBUTE_PRIMITIVE_STDTYPE_1ARG(MACRO) \
36+
MACRO(char) \
3637
MACRO(int8_t) \
3738
MACRO(int16_t) \
3839
MACRO(int32_t) \
@@ -43,24 +44,24 @@
4344
MACRO(uint64_t) \
4445
MACRO(float) \
4546
MACRO(double) \
46-
MACRO(long double) \
47-
MACRO(char)
47+
MACRO(long double)
4848

4949
#define ADIOS2_FOREACH_PRIMITIVE_STDTYPE_1ARG(MACRO) \
5050
ADIOS2_FOREACH_ATTRIBUTE_PRIMITIVE_STDTYPE_1ARG(MACRO) \
5151
MACRO(std::complex<float>) \
5252
MACRO(std::complex<double>)
5353

5454
#define ADIOS2_FOREACH_ATTRIBUTE_STDTYPE_1ARG(MACRO) \
55-
ADIOS2_FOREACH_PRIMITIVE_STDTYPE_1ARG(MACRO) \
56-
MACRO(std::string)
55+
MACRO(std::string) \
56+
ADIOS2_FOREACH_PRIMITIVE_STDTYPE_1ARG(MACRO)
5757

5858
#define ADIOS2_FOREACH_STDTYPE_1ARG(MACRO) \
59-
ADIOS2_FOREACH_PRIMITIVE_STDTYPE_1ARG(MACRO) \
60-
MACRO(std::string)
59+
MACRO(std::string) \
60+
ADIOS2_FOREACH_PRIMITIVE_STDTYPE_1ARG(MACRO)
6161

6262
#define ADIOS2_FOREACH_TYPE_1ARG(MACRO) \
6363
MACRO(std::string) \
64+
MACRO(char) \
6465
MACRO(signed char) \
6566
MACRO(unsigned char) \
6667
MACRO(short) \
@@ -75,10 +76,10 @@
7576
MACRO(double) \
7677
MACRO(long double) \
7778
MACRO(std::complex<float>) \
78-
MACRO(std::complex<double>) \
79-
MACRO(char)
79+
MACRO(std::complex<double>)
8080

8181
#define ADIOS2_FOREACH_PRIMITIVE_TYPE_1ARG(MACRO) \
82+
MACRO(char) \
8283
MACRO(signed char) \
8384
MACRO(unsigned char) \
8485
MACRO(short) \
@@ -93,18 +94,17 @@
9394
MACRO(double) \
9495
MACRO(long double) \
9596
MACRO(std::complex<float>) \
96-
MACRO(std::complex<double>) \
97-
MACRO(char)
97+
MACRO(std::complex<double>)
9898

9999
#define ADIOS2_FOREACH_COMPLEX_PRIMITIVE_TYPE_1ARG(MACRO) \
100100
MACRO(float) \
101101
MACRO(double) \
102102
MACRO(long double)
103103

104104
#define ADIOS2_FOREACH_CHAR_TYPE_1ARG(MACRO) \
105+
MACRO(char) \
105106
MACRO(signed char) \
106-
MACRO(unsigned char) \
107-
MACRO(char)
107+
MACRO(unsigned char)
108108

109109
#define ADIOS2_FOREACH_NUMERIC_TYPE_1ARG(MACRO) \
110110
MACRO(short) \
@@ -123,6 +123,7 @@
123123

124124
#define ADIOS2_FOREACH_ATTRIBUTE_TYPE_1ARG(MACRO) \
125125
MACRO(std::string) \
126+
MACRO(char) \
126127
MACRO(signed char) \
127128
MACRO(unsigned char) \
128129
MACRO(short) \
@@ -137,10 +138,10 @@
137138
MACRO(double) \
138139
MACRO(long double) \
139140
MACRO(std::complex<float>) \
140-
MACRO(std::complex<double>) \
141-
MACRO(char)
141+
MACRO(std::complex<double>)
142142

143143
#define ADIOS2_FOREACH_ATTRIBUTE_PRIMITIVE_TYPE_1ARG(MACRO) \
144+
MACRO(char) \
144145
MACRO(signed char) \
145146
MACRO(unsigned char) \
146147
MACRO(short) \
@@ -155,8 +156,7 @@
155156
MACRO(double) \
156157
MACRO(long double) \
157158
MACRO(std::complex<float>) \
158-
MACRO(std::complex<double>) \
159-
MACRO(char)
159+
MACRO(std::complex<double>)
160160

161161
/**
162162
<pre>
@@ -185,6 +185,7 @@
185185
#define ADIOS2_FOREACH_ATTRIBUTE_STDTYPE_2ARGS(MACRO) \
186186
MACRO(std::string, string) \
187187
MACRO(int8_t, int8) \
188+
MACRO(char, char) \
188189
MACRO(uint8_t, uint8) \
189190
MACRO(int16_t, int16) \
190191
MACRO(uint16_t, uint16) \
@@ -196,11 +197,11 @@
196197
MACRO(double, double) \
197198
MACRO(long double, ldouble) \
198199
MACRO(std::complex<float>, cfloat) \
199-
MACRO(std::complex<double>, cdouble) \
200-
MACRO(char, char)
200+
MACRO(std::complex<double>, cdouble)
201201

202202
#define ADIOS2_FOREACH_PRIMITVE_STDTYPE_2ARGS(MACRO) \
203203
MACRO(int8_t, int8) \
204+
MACRO(char, char) \
204205
MACRO(uint8_t, uint8) \
205206
MACRO(int16_t, int16) \
206207
MACRO(uint16_t, uint16) \
@@ -212,8 +213,7 @@
212213
MACRO(double, double) \
213214
MACRO(long double, ldouble) \
214215
MACRO(std::complex<float>, cfloat) \
215-
MACRO(std::complex<double>, cdouble) \
216-
MACRO(char, char)
216+
MACRO(std::complex<double>, cdouble)
217217

218218
#define ADIOS2_FOREACH_MINMAX_STDTYPE_2ARGS(MACRO) \
219219
MACRO(int8_t, int8) \

‎source/adios2/core/VariableBase.cpp‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -221,6 +221,13 @@ void VariableBase::SetMemorySelection(const Box<Dims> &memorySelection)
221221
const Dims &memoryStart = memorySelection.first;
222222
const Dims &memoryCount = memorySelection.second;
223223

224+
if (memoryStart.empty() && memoryCount.empty())
225+
{
226+
m_MemoryStart.clear();
227+
m_MemoryCount.clear();
228+
return;
229+
}
230+
224231
if (m_SingleValue)
225232
{
226233
helper::Throw<std::invalid_argument>("Core", "VariableBase", "SetMemorySelection",

0 commit comments

Comments
 (0)