ADIOS2 Append Mode (#1007)

* error::Internal

* Frontend implementation of appending

Also use switch statement in many frontend places for the Access enum.
With 4 variants, if statements become increasingly unreadable and it's
good to have compiler warnings if a case is forgotten. Try to avoid
default branches.

Also refactor `autoDetectPadding()` into a separate function since it is
now needed in two places.

* Backend implementation

1) Make backends aware of Append mode
2) In ADIOS1, fix use of namespaces, only use #include statements
   outside of namespaces

* Documentation and testing

* Extend tests: also use variable-based encoding

* Allow overwriting old datasets in HDF5

* Test that RW-mode in group-based mode still works

* Document extended meaning of createFile task

* Apply suggestions from code review

Co-authored-by: Axel Huebl <[email protected]>

* Remove special test paths for ADIOS2

* C++17 and other CI fixes

* Add missing throw

* Apply suggestions from code review

Co-authored-by: Axel Huebl <[email protected]>

* Use H5Lexists to be more efficient in lookup

Co-authored-by: Axel Huebl <[email protected]>

Author: franzpoeschel

docs/source/usage/workflow.rst

@@ -1,5 +1,39 @@
 .. _workflow:
 
+Access modes
+============
+
+The openPMD-api distinguishes between a number of different access modes:
+
+* **Create mode**: Used for creating a new Series from scratch.
+  Any file possibly existing in the specified location will be overwritten.
+* **Read-only mode**: Used for reading from an existing Series.
+  No modifications will be made.
+* **Read/Write mode**: Creates a new Series if not existing, otherwise opens an existing Series for reading and writing.
+  New datasets and iterations will be inserted as needed.
+  Not fully supported by all backends:
+
+  * ADIOS1: Automatically coerced to *Create* mode if the file does not exist yet and to *Read-only* mode if it exists.
+  * ADIOS2: Automatically coerced to *Create* mode if the file does not exist yet and to *Read-only* mode if it exists.
+  Since this happens on a per-file level, this mode allows to read from existing iterations and write to new iterations at the same time in file-based iteration encoding.
+* **Append mode**: Restricted mode for appending new iterations to an existing Series that is supported by all backends at least in file-based iteration encoding, and by all but ADIOS1 in other encodings.
+  The API is equivalent to that of the *Create* mode, meaning that no reading is supported whatsoever.
+  If the Series does not exist yet, this behaves equivalently to the *Create* mode.
+  Existing iterations will not be deleted, newly-written iterations will be inserted.
+
+  **Warning:** When writing an iteration that already exists, the behavior is implementation-defined and depends on the chosen backend and iteration encoding:
+
+  * The new iteration might fully replace the old one.
+  * The new iteration might be merged into the old one.
+  * (To be removed in a future update) The old and new iteration might coexist in the resulting dataset.
+
+  We suggest to fully define iterations when using Append mode (i.e. as if using Create mode) to avoid implementation-specific behavior.
+  Appending to an openPMD Series is only supported on a per-iteration level.
+
+  **Warning:** There is no reading involved in using Append mode.
+  It is a user's responsibility to ensure that the appended dataset and the appended-to dataset are compatible with each other.
+  The results of using incompatible backend configurations are undefined.
+
 Workflow
 ========
 

include/openPMD/Error.hpp

@@ -69,5 +69,16 @@ namespace error
 
         BackendConfigSchema(std::vector<std::string>, std::string what);
     };
+
+    /**
+     * @brief Internal errors that should not happen. Please report.
+     *
+     * Example: A nullpointer is observed somewhere.
+     */
+    class Internal : public Error
+    {
+    public:
+        Internal(std::string const &what);
+    };
 } // namespace error
 } // namespace openPMD

include/openPMD/IO/AbstractIOHandler.hpp

@@ -121,6 +121,23 @@ namespace internal
  */
 class AbstractIOHandler
 {
+    friend class Series;
+
+private:
+    void setIterationEncoding(IterationEncoding encoding)
+    {
+        /*
+         * In file-based iteration encoding, the APPEND mode is handled entirely
+         * by the frontend, the backend should just treat it as CREATE mode
+         */
+        if (encoding == IterationEncoding::fileBased &&
+            m_backendAccess == Access::APPEND)
+        {
+            // do we really want to have those as const members..?
+            *const_cast<Access *>(&m_backendAccess) = Access::CREATE;
+        }
+    }
+
 public:
 #if openPMD_HAVE_MPI
     AbstractIOHandler(std::string path, Access at, MPI_Comm)
@@ -153,6 +170,7 @@ class AbstractIOHandler
     virtual std::string backendName() const = 0;
 
     std::string const directory;
+    // why do these need to be separate?
     Access const m_backendAccess;
     Access const m_frontendAccess;
     std::queue<IOTask> m_work;

include/openPMD/IO/AbstractIOHandlerImpl.hpp

@@ -266,14 +266,17 @@ class AbstractIOHandlerImpl
      * file.
      *
      * The operation should fail if m_handler->m_frontendAccess is
-     * Access::READ_ONLY. The new file should be located in
-     * m_handler->directory. The new file should have the filename
-     * parameters.name. The filename should include the correct corresponding
-     * filename extension. Any existing file should be overwritten if
-     * m_handler->m_frontendAccess is Access::CREATE. The Writables file
-     * position should correspond to the root group "/" of the hierarchy. The
-     * Writable should be marked written when the operation completes
-     * successfully.
+     * Access::READ_ONLY. If m_handler->m_frontendAccess is Access::APPEND, a
+     * possibly existing file should not be overwritten. Instead, written
+     * updates should then either occur in-place or in form of new IO steps.
+     * Support for reading is not necessary in Append mode.
+     * The new file should be located in m_handler->directory.
+     * The new file should have the filename parameters.name.
+     * The filename should include the correct corresponding filename extension.
+     * Any existing file should be overwritten if m_handler->m_frontendAccess is
+     * Access::CREATE. The Writables file position should correspond to the root
+     * group "/" of the hierarchy. The Writable should be marked written when
+     * the operation completes successfully.
      */
     virtual void
     createFile(Writable *, Parameter<Operation::CREATE_FILE> const &) = 0;

include/openPMD/IO/Access.hpp

@@ -28,7 +28,8 @@ enum class Access
 {
     READ_ONLY, //!< open series as read-only, fails if series is not found
     READ_WRITE, //!< open existing series as writable
-    CREATE //!< create new series and truncate existing (files)
+    CREATE, //!< create new series and truncate existing (files)
+    APPEND //!< write new iterations to an existing series without reading
 }; // Access
 
 // deprecated name (used prior to 0.12.0)

src/Error.cpp

@@ -45,5 +45,12 @@ namespace error
               concatVector(errorLocation_in) + "': " + std::move(what))
         , errorLocation(std::move(errorLocation_in))
     {}
+
+    Internal::Internal(std::string const &what)
+        : Error(
+              "Internal error: " + what +
+              "\nThis is a bug. Please report at ' "
+              "https://github.com/openPMD/openPMD-api/issues'.")
+    {}
 } // namespace error
 } // namespace openPMD

src/IO/ADIOS/ADIOS2IOHandler.cpp

@@ -315,9 +315,6 @@ void ADIOS2IOHandlerImpl::createFile(
         m_iterationEncoding = parameters.encoding;
         associateWithFile(writable, shared_name);
         this->m_dirty.emplace(shared_name);
-        getFileData(shared_name, IfFileNotOpen::OpenImplicitly).m_mode =
-            adios2::Mode::Write; // WORKAROUND
-        // ADIOS2 does not yet implement ReadWrite Mode
 
         writable->written = true;
         writable->abstractFilePosition = std::make_shared<ADIOS2FilePosition>();
@@ -1074,21 +1071,16 @@ adios2::Mode ADIOS2IOHandlerImpl::adios2AccessMode(std::string const &fullPath)
         if (auxiliary::directory_exists(fullPath) ||
             auxiliary::file_exists(fullPath))
         {
-            std::cerr << "ADIOS2 does currently not yet implement ReadWrite "
-                         "(Append) mode. "
-                      << "Replacing with Read mode." << std::endl;
             return adios2::Mode::Read;
         }
         else
         {
-            std::cerr << "ADIOS2 does currently not yet implement ReadWrite "
-                         "(Append) mode. "
-                      << "Replacing with Write mode." << std::endl;
             return adios2::Mode::Write;
         }
-    default:
-        return adios2::Mode::Undefined;
+    case Access::APPEND:
+        return adios2::Mode::Append;
     }
+    throw std::runtime_error("Unreachable!");
 }
 
 json::TracingJSON ADIOS2IOHandlerImpl::nullvalue = {
@@ -2235,6 +2227,7 @@ namespace detail
                         delayOpeningTheFirstStep = true;
                         break;
                     case adios2::Mode::Write:
+                    case adios2::Mode::Append:
                         /*
                          * File engines, write mode:
                          * Default for old layout is no steps.
@@ -2442,6 +2435,7 @@ namespace detail
         {
             switch (m_mode)
             {
+            case adios2::Mode::Append:
             case adios2::Mode::Write: {
                 // usesSteps attribute only written upon ::advance()
                 // this makes sure that the attribute is only put in case
@@ -2686,17 +2680,14 @@ namespace detail
                 switch (ba.m_mode)
                 {
                 case adios2::Mode::Write:
+                case adios2::Mode::Append:
                     eng.PerformPuts();
                     break;
                 case adios2::Mode::Read:
                     eng.PerformGets();
                     break;
-                case adios2::Mode::Append:
-                    // TODO order?
-                    eng.PerformGets();
-                    eng.PerformPuts();
-                    break;
                 default:
+                    throw error::Internal("[ADIOS2] Unexpected access mode.");
                     break;
                 }
             },

src/IO/ADIOS/CommonADIOS1IOHandler.cpp

@@ -20,6 +20,7 @@
  */
 
 #include "openPMD/IO/ADIOS/CommonADIOS1IOHandler.hpp"
+#include "openPMD/Error.hpp"
 
 #if openPMD_HAVE_ADIOS1
 
@@ -406,6 +407,15 @@ void CommonADIOS1IOHandlerImpl<ChildClass>::createFile(
         if (!auxiliary::ends_with(name, ".bp"))
             name += ".bp";
 
+        if (m_handler->m_backendAccess == Access::APPEND &&
+            auxiliary::file_exists(name))
+        {
+            throw error::OperationUnsupportedInBackend(
+                "ADIOS1",
+                "Appending to existing file on disk (use Access::CREATE to "
+                "overwrite)");
+        }
+
         writable->written = true;
         writable->abstractFilePosition =
             std::make_shared<ADIOS1FilePosition>("/");

src/IO/HDF5/HDF5IOHandler.cpp

@@ -237,13 +237,41 @@ void HDF5IOHandlerImpl::createFile(
         std::string name = m_handler->directory + parameters.name;
         if (!auxiliary::ends_with(name, ".h5"))
             name += ".h5";
-        unsigned flags;
-        if (m_handler->m_backendAccess == Access::CREATE)
+        unsigned flags{};
+        switch (m_handler->m_backendAccess)
+        {
+        case Access::CREATE:
             flags = H5F_ACC_TRUNC;
-        else
+            break;
+        case Access::APPEND:
+            if (auxiliary::file_exists(name))
+            {
+                flags = H5F_ACC_RDWR;
+            }
+            else
+            {
+                flags = H5F_ACC_TRUNC;
+            }
+            break;
+        case Access::READ_WRITE:
             flags = H5F_ACC_EXCL;
-        hid_t id =
-            H5Fcreate(name.c_str(), flags, H5P_DEFAULT, m_fileAccessProperty);
+            break;
+        case Access::READ_ONLY:
+            // condition has been checked above
+            throw std::runtime_error(
+                "[HDF5] Control flow error in createFile backend access mode.");
+        }
+
+        hid_t id{};
+        if (flags == H5F_ACC_RDWR)
+        {
+            id = H5Fopen(name.c_str(), flags, m_fileAccessProperty);
+        }
+        else
+        {
+            id = H5Fcreate(
+                name.c_str(), flags, H5P_DEFAULT, m_fileAccessProperty);
+        }
         VERIFY(id >= 0, "[HDF5] Internal error: Failed to create HDF5 file");
 
         writable->written = true;
@@ -409,6 +437,36 @@ void HDF5IOHandlerImpl::createDataset(
             "[HDF5] Internal error: Failed to open HDF5 group during dataset "
             "creation");
 
+        if (m_handler->m_backendAccess == Access::APPEND)
+        {
+            // The dataset might already exist in the file from a previous run
+            // We delete it, otherwise we could not create it again with
+            // possibly different parameters.
+            if (htri_t link_id = H5Lexists(node_id, name.c_str(), H5P_DEFAULT);
+                link_id > 0)
+            {
+                // This only unlinks, but does not delete the dataset
+                // Deleting the actual dataset physically is now up to HDF5:
+                // > when removing an object with H5Ldelete, the HDF5 library
+                // > should be able to detect and recycle the file space when no
+                // > other reference to the deleted object exists
+                // https://github.com/openPMD/openPMD-api/pull/1007#discussion_r867223316
+                herr_t status = H5Ldelete(node_id, name.c_str(), H5P_DEFAULT);
+                VERIFY(
+                    status == 0,
+                    "[HDF5] Internal error: Failed to delete old dataset '" +
+                        name + "' from group for overwriting.");
+            }
+            else if (link_id < 0)
+            {
+                throw std::runtime_error(
+                    "[HDF5] Internal error: Failed to check for link existence "
+                    "of '" +
+                    name + "' inside group for overwriting.");
+            }
+            // else: link_id == 0: Link does not exist, nothing to do
+        }
+
         Datatype d = parameters.dtype;
         if (d == Datatype::UNDEFINED)
         {
@@ -702,7 +760,14 @@ void HDF5IOHandlerImpl::openFile(
     Access at = m_handler->m_backendAccess;
     if (at == Access::READ_ONLY)
         flags = H5F_ACC_RDONLY;
-    else if (at == Access::READ_WRITE || at == Access::CREATE)
+    /*
+     * Within the HDF5 backend, APPEND and READ_WRITE mode are
+     * equivalent, but the openPMD frontend exposes no reading
+     * functionality in APPEND mode.
+     */
+    else if (
+        at == Access::READ_WRITE || at == Access::CREATE ||
+        at == Access::APPEND)
         flags = H5F_ACC_RDWR;
     else
         throw std::runtime_error("[HDF5] Unknown file Access");

src/IO/JSON/JSONIOHandlerImpl.cpp

@@ -117,7 +117,7 @@ void JSONIOHandlerImpl::createFile(
             file.invalidate();
         }
 
-        std::string const dir(m_handler->directory);
+        std::string const &dir(m_handler->directory);
         if (!auxiliary::directory_exists(dir))
         {
             auto success = auxiliary::create_directories(dir);
@@ -126,8 +126,14 @@ void JSONIOHandlerImpl::createFile(
 
         associateWithFile(writable, shared_name);
         this->m_dirty.emplace(shared_name);
-        // make sure to overwrite!
-        this->m_jsonVals[shared_name] = std::make_shared<nlohmann::json>();
+
+        if (m_handler->m_backendAccess != Access::APPEND)
+        {
+            // make sure to overwrite!
+            this->m_jsonVals[shared_name] = std::make_shared<nlohmann::json>();
+        }
+        // else: the JSON value is not available in m_jsonVals and will be
+        // read from the file later on before overwriting
 
         writable->written = true;
         writable->abstractFilePosition = std::make_shared<JSONFilePosition>();
@@ -910,6 +916,14 @@ JSONIOHandlerImpl::getFilehandle(File fileName, Access access)
     {
     case Access::CREATE:
     case Access::READ_WRITE:
+    case Access::APPEND:
+        /*
+         * Always truncate when writing, we alway write entire JSON
+         * datasets, never partial ones.
+         * Within the JSON backend, APPEND and READ_WRITE mode are
+         * equivalent, but the openPMD frontend exposes no reading
+         * functionality in APPEND mode.
+         */
         fs->open(path, std::ios_base::out | std::ios_base::trunc);
         break;
     case Access::READ_ONLY: