From 00532a7873f1ec3eff9aa4238f303597bd039b6b Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Wed, 3 Nov 2021 18:13:39 -0700 Subject: [PATCH] Release 0.14.3 backports (#1135) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Read: time/dt also in long double (#1096) * Python: time/dt round-trip Test writing and reading time and dt on an iteration via properties. * Fix: Iteration read of long double time Support reading of `dt` and `time` attributes if they are of type `long double`. (openPMD standard: all `floatX` supported) * Executables: CXX_STANDARD/EXTENSIONS (#1102) Set `CXX_EXTENSIONS OFF` and `CXX_STANDARD_REQUIRED ON` for created executables. This mitigates issues with NVCC 11.0 and C++17 builds seen as added `-std=gnu++17` flags that lead to ``` nvcc fatal : Value 'gnu++17' is not defined for option 'std' ``` when using `nvcc` as CXX compiler directly. * Doc: More Locations -DPython_EXECUTABLE (#1104) Mention the `-DPython_EXECUTABLE` twice more in build examples. * NVCC + C++17 (#1103) * NVCC + C++17 Work-around a build issue with NVCC in C++17 builds. ``` include/openPMD/backend/Attributable.hpp(437): error #289: no instance of constructor "openPMD::Attribute::Attribute" matches the argument list argument types are: (std::__cxx11::string) detected during instantiation of "__nv_bool openPMD::AttributableInterface::setAttribute(const std::__cxx11::string &, T) [with T=std::__cxx11::string]" ``` from ``` inline bool AttributableInterface::setAttribute( std::string const & key, char const value[] ) { return this->setAttribute(key, std::string(value)); } ``` Seen with: - NVCC 11.0.2 + GCC 8.3.0 - NVCC 11.0.2 + GCC 7.5.0 * NVCC 11.0.2 C++17 work-around: Add Comment * Lazy parsing: Make findable in docs and use in openpmd-ls (#1111) * Use deferred iteration parsing in openpmd-ls * Make lazy/deferred parsing searchable * Add a way to search for usesteps key * HDF5: Document HDF5_USE_FILE_LOCKING (#1106) Document a HDF5 read work-around that we currently need on OLCF Jupyter (https://jupyter.olcf.ornl.gov), due to a mounting issue of GPFS in the Jupyter serice (OLCFHELP-3685). From the HDF5 1.10.1 Release Notes: ``` Other New Features and Enhancements =================================== Library ------- - Added a mechanism for disabling the SWMR file locking scheme. The file locking calls used in HDF5 1.10.0 (including patch1) will fail when the underlying file system does not support file locking or where locks have been disabled. To disable all file locking operations, an environment variable named HDF5_USE_FILE_LOCKING can be set to the five-character string 'FALSE'. This does not fundamentally change HDF5 library operation (aside from initial file open/create, SWMR is lock-free), but users will have to be more careful about opening files to avoid problematic access patterns (i.e.: multiple writers) that the file locking was designed to prevent. Additionally, the error message that is emitted when file lock operations set errno to ENOSYS (typical when file locking has been disabled) has been updated to describe the problem and potential resolution better. (DER, 2016/10/26, HDFFV-9918) ``` This also exists as a compilation option for HDF5 in CMake, where it defaults to ``TRUE`` by default, which is also what distributions/ package managers ship. Disabling from Bash: ```bash export HDF5_USE_FILE_LOCKING=FALSE ``` Disabling from Python: ```py import os os.environ['HDF5_USE_FILE_LOCKING'] = "FALSE" ``` * Avoid object slicing when deriving from Series class (#1107) * Make Series class final * Use private constructor to avoid object slicing * Doc: OMPI_MCA_io Control (#1114) Document OpenMPI MPI-I/O backend control. We have documented this long in #446. * openPMD.hpp: Include auxiliary StringManip (#1124) Include this, handy functions. * CXX Std: Remember Impl. (#1128) We use `` or `` in our public API interface for datatypes, depending on the C++ standard. This pull request makes sure that the same implementation is used in downstream code, even if the C++ standard is switched. This avoids ABI issues when, e.g., using a C++14 built openPMD-api in a C++17 downstream code. * Spack: No More `load -r` (#1125) The `-r` argument was removed from `spack load` and is now implied. * Fix AppVeyor: Python Executable (#1127) * GH Action: Add MSVC & ClangCL on Win * Fix AppVeyor: Python Executable * Avoid mismatching system Python and Conda Python * Conda: Fix Numpy * CMake: Skip Pipe Test Written in a too special way, we cannot assume SH is always present * Test 8b (Bench Read Parallel): Support Variable encoding, Fix Bugs (#1131) * added support to read variable encoding, plus fixed some bugs * fixed style * Update examples/8b_benchmark_read_parallel.cpp remove commented out code Co-authored-by: Axel Huebl * Update examples/8b_benchmark_read_parallel.cpp Co-authored-by: Axel Huebl * Update examples/8b_benchmark_read_parallel.cpp Co-authored-by: Axel Huebl * Update examples/8b_benchmark_read_parallel.cpp Co-authored-by: Axel Huebl * Update examples/8b_benchmark_read_parallel.cpp Co-authored-by: Axel Huebl * Update examples/8b_benchmark_read_parallel.cpp Co-authored-by: Axel Huebl * Update examples/8b_benchmark_read_parallel.cpp Co-authored-by: Axel Huebl * Update examples/8b_benchmark_read_parallel.cpp Co-authored-by: Axel Huebl * Update examples/8b_benchmark_read_parallel.cpp Co-authored-by: Axel Huebl * Update examples/8b_benchmark_read_parallel.cpp Co-authored-by: Axel Huebl * Update examples/8b_benchmark_read_parallel.cpp Co-authored-by: Axel Huebl * removed commented line * updated 8b env option Co-authored-by: Axel Huebl * HDF5 I/O optimizations (#1129) * Include HDF5 optimization options * Fix code style check * Fix validations and include checks * Fix style check * Remove unecessary strict check * Update documentation with HDF5 tuning options * Update contributions * Fix Guards for H5Pset_all_coll_metadata* * MPI Guard: H5Pset_all_coll_metadata* * Remove duplicated variable Co-authored-by: Axel Huebl * Include known issues section for HDF5 (#1132) * Update known issues with HDF5 and collective metadata operations * Fix rst link and tiny typo * Add targeted bugfix releases. Co-authored-by: Axel Huebl * Include check for paged allocation (#1133) * Include check for paged allocation * Update ParallelHDF5IOHandler.cpp * libfabric 1.6+: Document SST Work-Arounds (#1134) * libfabric 1.6+: Document SST Work-Arounds Document work-arounds for libfabric 1.6+ on Cray systems when using data staging / streaming with ADIOS2 SST. Co-authored-by: Franz Pöschel * Fix: Read Inconsistent Zero Pads (#1118) * [Draft] Fix: Read Inconsistent Zero Pads Some codes mess up the zero-padding in `fileBased` encoding, e.g., when specifying padding to 5 digits but creating >100'000 output steps. Files like those cannot yet be parsed and fell back to no padding, which fails to open the file: ``` openpmd_00000.h5 openpmd_02000.h5 openpmd_101000.h5 openpmd_01000.h5 openpmd_100000.h5 openpmd_104000.h5 ``` Error: ``` RuntimeError: [HDF5] Failed to open HDF5 file diags/diag1/openpmd_0.h5 ``` * Revert previous changes except for test Parse iteration numbers that are longer than their padding Read inconsistent zero padding * Overflow Padding: Read Test * Warn if the prefix does end in a digit * Fix: Don't let oversize numbers accidentally bump the padding * Update test * Issue warnings on misleading patterns also when writing * Minor Style Update Co-authored-by: Franz Pöschel * Release: 0.14.3 Co-authored-by: Franz Pöschel Co-authored-by: guj Co-authored-by: Jean Luca Bez Co-authored-by: Jean Luca Bez --- .appveyor.yml | 33 ++-- .github/workflows/windows.yml | 57 +++++++ .rodare.json | 6 + CHANGELOG.rst | 43 ++++++ CITATION.cff | 2 +- CMakeLists.txt | 80 +++++++--- README.md | 6 +- cmake/try_variant.cpp | 51 ++++++ docs/source/backends/adios2.rst | 20 ++- docs/source/backends/hdf5.rst | 73 +++++++-- docs/source/conf.py | 4 +- docs/source/details/backendconfig.rst | 4 +- docs/source/index.rst | 2 +- docs/source/install/install.rst | 4 +- docs/source/usage/benchmarks.rst | 8 +- examples/8b_benchmark_read_parallel.cpp | 145 +++++++++++------- include/openPMD/IO/HDF5/HDF5IOHandlerImpl.hpp | 3 + include/openPMD/Series.hpp | 3 + include/openPMD/auxiliary/VariantSrc.hpp | 6 +- include/openPMD/backend/Attributable.hpp | 7 +- include/openPMD/cli/ls.hpp | 3 +- include/openPMD/config.hpp.in | 4 + include/openPMD/openPMD.hpp | 1 + include/openPMD/version.hpp | 2 +- setup.py | 2 +- share/openPMD/download_samples.ps1 | 2 + src/IO/HDF5/HDF5IOHandler.cpp | 129 ++++++++++++++-- src/IO/HDF5/ParallelHDF5IOHandler.cpp | 49 +++++- src/Iteration.cpp | 4 + src/Series.cpp | 134 ++++++++-------- test/SerialIOTest.cpp | 97 +++++++++--- test/python/unittest/API/APITest.py | 17 +- 32 files changed, 771 insertions(+), 230 deletions(-) create mode 100644 .github/workflows/windows.yml create mode 100644 cmake/try_variant.cpp diff --git a/.appveyor.yml b/.appveyor.yml index c10a31646f..55a5a78f70 100644 --- a/.appveyor.yml +++ b/.appveyor.yml @@ -1,33 +1,19 @@ environment: matrix: - - TARGET_ARCH: x86 - CONDA_PY: 36 - CONDA_INSTALL_LOCN: C:\\Miniconda36 - APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2017 - platform: x86 - SHARED: ON - - TARGET_ARCH: x64 - CONDA_PY: 37 + CONDA_PY: 3.7 CONDA_INSTALL_LOCN: C:\\Miniconda37-x64 APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2017 platform: x64 - SHARED: ON + SHARED: OFF - TARGET_ARCH: x86 - CONDA_PY: 37 + CONDA_PY: 3.7 CONDA_INSTALL_LOCN: C:\\Miniconda37 APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2017 platform: x86 SHARED: ON - - TARGET_ARCH: x64 - CONDA_PY: 36 - CONDA_INSTALL_LOCN: C:\\Miniconda36-x64 - APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2017 - platform: x64 - SHARED: OFF - configuration: # - Debug - Release @@ -91,7 +77,6 @@ install: # Add path, activate `conda` and update conda. - cmd: call %CONDA_INSTALL_LOCN%\Scripts\activate.bat - - cmd: conda update --yes --quiet conda - cmd: set PYTHONUNBUFFERED=1 @@ -101,9 +86,9 @@ install: - cmd: conda config --append channels conda-forge # Configure the VM. - - cmd: conda install -n root --quiet --yes numpy>=1.15 cmake hdf5 - # ADIOS2 build only for 64bit Windows and Python 3.6+ - - cmd: if "%TARGET_ARCH%"=="x64" if %CONDA_PY% GEQ 36 conda install -n root --quiet --yes adios2 + - cmd: conda install -n root --quiet --yes numpy cmake hdf5 python=%CONDA_PY% + # ADIOS2 build only for 64bit Windows + - cmd: if "%TARGET_ARCH%"=="x64" conda install -n root --quiet --yes adios2 python=%CONDA_PY% before_build: - cmd: cd C:\projects\openpmd-api @@ -121,12 +106,12 @@ before_build: # - cmd: if "%TARGET_ARCH%"=="x64" "C:\Program Files (x86)\Microsoft Visual Studio 15.9\VC\vcvarsall.bat" amd64 # CMake configure - - cmd: cmake -G "%OPENPMD_CMAKE_GENERATOR%" -DCMAKE_BUILD_TYPE=%CONFIGURATION% -DBUILD_SHARED_LIBS=%SHARED% -DBUILD_TESTING=ON -DCMAKE_INSTALL_PREFIX="%CONDA_INSTALL_LOCN%" -DCMAKE_INSTALL_BINDIR="Library\bin" ".." + - cmd: cmake -G "%OPENPMD_CMAKE_GENERATOR%" -DCMAKE_BUILD_TYPE=%CONFIGURATION% -DBUILD_SHARED_LIBS=%SHARED% -DBUILD_TESTING=ON -DopenPMD_USE_PYTHON=ON -DPython_EXECUTABLE="%CONDA_INSTALL_LOCN%\python.exe" -DCMAKE_INSTALL_PREFIX="%CONDA_INSTALL_LOCN%" -DCMAKE_INSTALL_BINDIR="Library\bin" ".." build_script: - - cmd: cmake --build . --config %CONFIGURATION% + - cmd: cmake --build . --config %CONFIGURATION% -j 2 - cmd: cmake --build . --config %CONFIGURATION% --target install test_script: - cmd: ctest -V -C %CONFIGURATION% - - cmd: python -c "import openpmd_api; print(openpmd_api.__version__); print(openpmd_api.variants)" + - cmd: call %CONDA_INSTALL_LOCN%\python.exe -c "import openpmd_api; print(openpmd_api.__version__); print(openpmd_api.variants)" diff --git a/.github/workflows/windows.yml b/.github/workflows/windows.yml new file mode 100644 index 0000000000..e3eb4c671a --- /dev/null +++ b/.github/workflows/windows.yml @@ -0,0 +1,57 @@ +name: 🪟 Windows + +on: [push, pull_request] + +concurrency: + group: ${{ github.ref }}-${{ github.head_ref }}-windows + cancel-in-progress: true + +jobs: + build_win_msvc: + name: MSVC w/o MPI + runs-on: windows-latest + if: github.event.pull_request.draft == false + steps: + - uses: actions/checkout@v2 + - name: Build & Install + run: | + python3.exe -m pip install --upgrade pip + python3.exe -m pip install --upgrade numpy + + pwsh "share\openPMD\download_samples.ps1" build + cmake -S . -B build ` + -DCMAKE_BUILD_TYPE=Debug ` + -DopenPMD_USE_MPI=OFF + cmake --build build --config Debug --parallel 2 + cmake --build build --config Debug --target install + +# add before install, and fix Python path: +# ctest --test-dir build -C Debug --output-on-failure + + + build_win_clang: + name: Clang w/o MPI + runs-on: windows-2019 + if: github.event.pull_request.draft == false + steps: + - uses: actions/checkout@v2 + - uses: seanmiddleditch/gha-setup-ninja@master + - name: Build & Install + shell: cmd + run: | + python3.exe -m pip install --upgrade pip + python3.exe -m pip install --upgrade numpy + + call "C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\vc\Auxiliary\build\vcvarsall.bat" x64 + pwsh "share\openPMD\download_samples.ps1" build + cmake -S . -B build ^ + -G "Ninja" ^ + -DCMAKE_C_COMPILER=clang-cl ^ + -DCMAKE_CXX_COMPILER=clang-cl ^ + -DCMAKE_BUILD_TYPE=Release ^ + -DopenPMD_USE_MPI=OFF + cmake --build build --config Release --parallel 2 + cmake --build build --config Debug --target install + +# add before install, and fix Python path: +# ctest --test-dir build -C Debug --output-on-failure diff --git a/.rodare.json b/.rodare.json index 7687580c49..cd2777030f 100644 --- a/.rodare.json +++ b/.rodare.json @@ -108,6 +108,12 @@ "name": "Schnetter, Erik", "orcid": "0000-0002-4518-9017", "type": "Other" + }, + { + "affiliation": "Lawrence Berkeley National Laboratory", + "name": "Bez, Jean Luca", + "orcid": "0000-0002-3915-1135", + "type": "Other" } ], "title": "C++ & Python API for Scientific I/O with openPMD", diff --git a/CHANGELOG.rst b/CHANGELOG.rst index 135f8033d5..8542ef49ea 100644 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -3,6 +3,49 @@ Changelog ========= +0.14.2 +------ +**Date:** 2021-11-03 + +Read Bugs, C++17 Mixing and HDF5 Performance + +This release makes reads more robust by fixing small API, file-based parsing and test bugs. +Building the library in C++14 and using it in C++17 will not result in incompatible ABIs anymore. +HDF5 1.10.1+ performance was improved significantly. + +Changes to "0.14.2" +^^^^^^^^^^^^^^^^^^^ + +Bug Fixes +""""""""" + +- read: + + - allow inconsistent zero pads #1118 + - time/dt also in long double #1096 +- test 8b - bench read parallel: + + - support variable encoding #1131 + - block located at top left corner was mistaken to read a block in the center #1131 +- CI (AppVeyor): Python executable #1127 +- C++17 mixing: remember ```` implementation #1128 +- support NVCC + C++17 #1103 +- avoid object slicing when deriving from ``Series`` class #1107 +- executables: ``CXX_STANDARD``/``EXTENSIONS`` #1102 + +Other +""""" + +- HDF5 I/O optimizations #1129 #1132 #1133 +- libfabric 1.6+: Document SST Work-Arounds #1134 +- OpenMPI: Document ``OMPI_MCA_io`` Control #1114 +- HDF5: Document ``HDF5_USE_FILE_LOCKING`` #1106 +- Lazy parsing: Make findable in docs and use in ``openpmd-ls`` #1111 +- Docs: More Locations ``-DPython_EXECUTABLE`` #1104 +- Spack: No More ``load -r`` #1125 +- ``openPMD.hpp``: include auxiliary ``StringManip`` #1124 + + 0.14.2 ------ **Date:** 2021-08-17 diff --git a/CITATION.cff b/CITATION.cff index 190a3cfcf5..911d132fda 100644 --- a/CITATION.cff +++ b/CITATION.cff @@ -25,7 +25,7 @@ contact: orcid: https://orcid.org/0000-0003-1943-7141 email: axelhuebl@lbl.gov title: "openPMD-api: C++ & Python API for Scientific I/O with openPMD" -version: 0.14.2 +version: 0.14.3 repository-code: https://github.com/openPMD/openPMD-api doi: 10.14278/rodare.27 license: LGPL-3.0-or-later diff --git a/CMakeLists.txt b/CMakeLists.txt index c7909e5a61..851eaa4b4d 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -2,7 +2,7 @@ # cmake_minimum_required(VERSION 3.15.0) -project(openPMD VERSION 0.14.2) # LANGUAGES CXX +project(openPMD VERSION 0.14.3) # LANGUAGES CXX # the openPMD "markup"/"schema" standard version set(openPMD_STANDARD_VERSION 1.1.0) @@ -453,26 +453,36 @@ target_include_directories(openPMD PUBLIC $ ) +# is this a C++17 compile? If yes, skip MPark.Variant +try_compile(openPMD_HAS_CXX17 + ${openPMD_BINARY_DIR}/try_variant + ${openPMD_SOURCE_DIR}/cmake/try_variant.cpp + CXX_STANDARD_REQUIRED ON + CXX_EXTENSIONS OFF +) +message(STATUS " supported (C++17 or newer): ${openPMD_HAS_CXX17}") + # C++11 std::variant (C++17 stdlib preview) -# TODO not needed with C++17 compiler -add_library(openPMD::thirdparty::mpark_variant INTERFACE IMPORTED) -set(openPMD_PC_EXTRA_INCLUDE "") -if(openPMD_USE_INTERNAL_VARIANT) - target_include_directories(openPMD::thirdparty::mpark_variant SYSTEM INTERFACE - $ - ) - message(STATUS "MPark.Variant: Using INTERNAL version '1.4.0'") -else() - find_package(mpark_variant 1.3.0 REQUIRED) # TODO: we want 1.4.1+ - target_link_libraries(openPMD::thirdparty::mpark_variant - INTERFACE mpark_variant) - get_target_property(EXTERNAL_MPARK_INCLUDE mpark_variant INTERFACE_INCLUDE_DIRECTORIES) - if(openPMD_HAVE_PKGCONFIG AND EXTERNAL_MPARK_INCLUDE) - set(openPMD_PC_EXTRA_INCLUDE "-I${EXTERNAL_MPARK_INCLUDE}") +if(NOT openPMD_HAS_CXX17) + add_library(openPMD::thirdparty::mpark_variant INTERFACE IMPORTED) + set(openPMD_PC_EXTRA_INCLUDE "") + if(openPMD_USE_INTERNAL_VARIANT) + target_include_directories(openPMD::thirdparty::mpark_variant SYSTEM INTERFACE + $ + ) + message(STATUS "MPark.Variant: Using INTERNAL version '1.4.0'") + else() + find_package(mpark_variant 1.3.0 REQUIRED) # TODO: we want 1.4.1+ + target_link_libraries(openPMD::thirdparty::mpark_variant + INTERFACE mpark_variant) + get_target_property(EXTERNAL_MPARK_INCLUDE mpark_variant INTERFACE_INCLUDE_DIRECTORIES) + if(openPMD_HAVE_PKGCONFIG AND EXTERNAL_MPARK_INCLUDE) + set(openPMD_PC_EXTRA_INCLUDE "-I${EXTERNAL_MPARK_INCLUDE}") + endif() + message(STATUS "MPark.Variant: Found version '${mpark_variant_VERSION}'") endif() - message(STATUS "MPark.Variant: Found version '${mpark_variant_VERSION}'") + target_link_libraries(openPMD PUBLIC openPMD::thirdparty::mpark_variant) endif() -target_link_libraries(openPMD PUBLIC openPMD::thirdparty::mpark_variant) # Catch2 for unit tests if(openPMD_BUILD_TESTING) @@ -518,8 +528,10 @@ if(openPMD_HAVE_ADIOS1) ) target_compile_options(openPMD.ADIOS1.Serial PUBLIC ${_msvc_options}) target_compile_options(openPMD.ADIOS1.Parallel PUBLIC ${_msvc_options}) - target_link_libraries(openPMD.ADIOS1.Serial PUBLIC openPMD::thirdparty::mpark_variant) - target_link_libraries(openPMD.ADIOS1.Parallel PUBLIC openPMD::thirdparty::mpark_variant) + if(NOT openPMD_HAS_CXX17) + target_link_libraries(openPMD.ADIOS1.Serial PUBLIC openPMD::thirdparty::mpark_variant) + target_link_libraries(openPMD.ADIOS1.Parallel PUBLIC openPMD::thirdparty::mpark_variant) + endif() target_include_directories(openPMD.ADIOS1.Serial PRIVATE ${openPMD_SOURCE_DIR}/include ${openPMD_BINARY_DIR}/include) @@ -811,6 +823,11 @@ if(openPMD_BUILD_TESTING) else() target_link_libraries(${testname}Tests PRIVATE CatchMain) endif() + + set_target_properties(${testname}Tests PROPERTIES + CXX_EXTENSIONS OFF + CXX_STANDARD_REQUIRED ON + ) endforeach() endif() @@ -818,6 +835,10 @@ if(openPMD_BUILD_CLI_TOOLS) foreach(toolname ${openPMD_CLI_TOOL_NAMES}) add_executable(openpmd-${toolname} src/cli/${toolname}.cpp) target_link_libraries(openpmd-${toolname} PRIVATE openPMD) + set_target_properties(openpmd-${toolname} PROPERTIES + CXX_EXTENSIONS OFF + CXX_STANDARD_REQUIRED ON + ) endforeach() endif() @@ -827,10 +848,18 @@ if(openPMD_BUILD_EXAMPLES) if(openPMD_HAVE_MPI) add_executable(${examplename} examples/${examplename}.cpp) target_link_libraries(${examplename} PRIVATE openPMD) + set_target_properties(${examplename} PROPERTIES + CXX_EXTENSIONS OFF + CXX_STANDARD_REQUIRED ON + ) endif() else() add_executable(${examplename} examples/${examplename}.cpp) target_link_libraries(${examplename} PRIVATE openPMD) + set_target_properties(${examplename} PROPERTIES + CXX_EXTENSIONS OFF + CXX_STANDARD_REQUIRED ON + ) endif() endforeach() endif() @@ -983,7 +1012,7 @@ if(openPMD_INSTALL) ) # install third-party libraries # TODO not needed with C++17 compiler - if(openPMD_USE_INTERNAL_VARIANT) + if(NOT openPMD_HAS_CXX17 AND openPMD_USE_INTERNAL_VARIANT) install(DIRECTORY "${openPMD_SOURCE_DIR}/share/openPMD/thirdParty/variant/include/mpark" DESTINATION ${CMAKE_INSTALL_INCLUDEDIR} ) @@ -1216,7 +1245,8 @@ if(openPMD_BUILD_TESTING) endforeach() # openpmd-pipe (python) test - if( openPMD_HAVE_HDF5 + if( NOT WIN32 + AND openPMD_HAVE_HDF5 AND (openPMD_HAVE_ADIOS2 OR openPMD_HAVE_ADIOS1) AND EXAMPLE_DATA_FOUND ) @@ -1345,7 +1375,11 @@ if(openPMD_INSTALL) endif() message("") message(" Additionally, install following third party libraries:") - message(" MPark.Variant: ${openPMD_USE_INTERNAL_VARIANT}") + if(openPMD_HAS_CXX17) + message(" MPark.Variant: OFF") + else() + message(" MPark.Variant: ${openPMD_USE_INTERNAL_VARIANT}") + endif() else() message(" Installation: OFF") endif() diff --git a/README.md b/README.md index 2999a9f664..f9d684d451 100644 --- a/README.md +++ b/README.md @@ -148,7 +148,7 @@ Choose *one* of the install methods below to get started: ```bash # optional: +python +adios1 -adios2 -hdf5 -mpi spack install openpmd-api -spack load -r openpmd-api +spack load openpmd-api ``` ### [Conda](https://conda.io) @@ -232,7 +232,7 @@ cd openPMD-api-build # for options append: # -DopenPMD_USE_...=... # e.g. for python support add: -# -DopenPMD_USE_PYTHON=ON +# -DopenPMD_USE_PYTHON=ON -DPython_EXECUTABLE=$(which python3) cmake ../openPMD-api cmake --build . @@ -412,6 +412,8 @@ Further thanks go to improvements and contributions from: Dask guidance & reviews * [Erik Schnetter (PITP)](https://github.com/eschnett): C++ API bug fixes +* [Jean Luca Bez (LBNL)](https://github.com/jeanbez): + HDF5 performance tuning ### Grants diff --git a/cmake/try_variant.cpp b/cmake/try_variant.cpp new file mode 100644 index 0000000000..51b8c094f8 --- /dev/null +++ b/cmake/try_variant.cpp @@ -0,0 +1,51 @@ +/* Copyright 2021 Axel Huebl + * + * This file is part of openPMD-api. + * + * openPMD-api is free software: you can redistribute it and/or modify + * it under the terms of of either the GNU General Public License or + * the GNU Lesser General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * openPMD-api is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License and the GNU Lesser General Public License + * for more details. + * + * You should have received a copy of the GNU General Public License + * and the GNU Lesser General Public License along with openPMD-api. + * If not, see . + */ +#pragma once + +#include +#include +#if __cplusplus >= 201703L +# include // IWYU pragma: export +#else +# error "Not a C++17 implementation" +#endif + + +int main() +{ + std::variant< int, float > v; + v = 42; + int i = std::get< int >(v); + assert(42 == i); + assert(42 == std::get< 0 >(v)); + + try + { + std::get< float >(v); + } + catch( std::bad_variant_access const & ex ) + { + std::cout << ex.what() << std::endl; + } + + v = 13.2f; + assert(13.2f == std::get<0>(v)); +} diff --git a/docs/source/backends/adios2.rst b/docs/source/backends/adios2.rst index 9c53d61190..72cafa020a 100644 --- a/docs/source/backends/adios2.rst +++ b/docs/source/backends/adios2.rst @@ -25,7 +25,7 @@ In order to activate steps, it is imperative to use the :ref:`Streaming API `_) that disallows random-accessing steps in file-based engines. With this ADIOS2 release, files written with steps may only be read using the streaming API. -In order to keep compatibility with older codes reading ADIOS2 files, step-based processing must currently be opted in to via use of the :ref:`JSON parameter` ``adios2.engine.usesteps = true`` when using a file-based engine such as BP3 or BP4. +In order to keep compatibility with older codes reading ADIOS2 files, step-based processing must currently be opted in to via use of the :ref:`JSON parameter` ``adios2.engine.usesteps = true`` when using a file-based engine such as BP3 or BP4 (usesteps). Upon reading a file, the ADIOS2 backend will automatically recognize whether it has been written with or without steps, ignoring the JSON option mentioned above. Steps are mandatory for streaming-based engines and trying to switch them off will result in a runtime error. @@ -155,6 +155,24 @@ Ignore the 30GB initialization phases. .. image:: ./memory_groupbased_nosteps.png :alt: Memory usage of group-based iteration without using steps + +Known Issues +------------ + +.. warning:: + + Nov 1st, 2021 (`ADIOS2 2887 `__): + The fabric selection in ADIOS2 has was designed for libfabric 1.6. + With newer versions of libfabric, the following workaround is needed to guide the selection of a functional fabric for RDMA support: + + The following environment variables can be set as work-arounds on Cray systems, when working with ADIOS2 SST: + + .. code-block:: bash + + export FABRIC_IFACE=mlx5_0 # ADIOS SST: select interface (1 NIC on Summit) + export FI_OFI_RXM_USE_SRX=1 # libfabric: use shared receive context from MSG provider + + Selected References ------------------- diff --git a/docs/source/backends/hdf5.rst b/docs/source/backends/hdf5.rst index 08f73c742f..5d6c21c424 100644 --- a/docs/source/backends/hdf5.rst +++ b/docs/source/backends/hdf5.rst @@ -21,14 +21,22 @@ Backend-Specific Controls The following environment variables control HDF5 I/O behavior at runtime. -===================================== ========= ==================================================================================== -environment variable default description -===================================== ========= ==================================================================================== -``OPENPMD_HDF5_INDEPENDENT`` ``ON`` Sets the MPI-parallel transfer mode to collective (``OFF``) or independent (``ON``). -``OPENPMD_HDF5_ALIGNMENT`` ``1`` Tuning parameter for parallel I/O, choose an alignment which is a multiple of the disk block size. -``OPENPMD_HDF5_CHUNKS`` ``auto`` Defaults for ``H5Pset_chunk``: ``"auto"`` (heuristic) or ``"none"`` (no chunking). -``H5_COLL_API_SANITY_CHECK`` unset Set to ``1`` to perform an ``MPI_Barrier`` inside each meta-data operation. -===================================== ========= ==================================================================================== +======================================== ============ =========================================================================================================== +Environment variable Default Description +======================================== ============ =========================================================================================================== +``OPENPMD_HDF5_INDEPENDENT`` ``ON`` Sets the MPI-parallel transfer mode to collective (``OFF``) or independent (``ON``). +``OPENPMD_HDF5_ALIGNMENT`` ``1`` Tuning parameter for parallel I/O, choose an alignment which is a multiple of the disk block size. +``OPENPMD_HDF5_THRESHOLD`` ``0`` Tuning parameter for parallel I/O, where ``0`` aligns all requests and other values act as a threshold. +``OPENPMD_HDF5_CHUNKS`` ``auto`` Defaults for ``H5Pset_chunk``: ``"auto"`` (heuristic) or ``"none"`` (no chunking). +``OPENPMD_HDF5_COLLECTIVE_METADATA`` ``ON`` Sets the MPI-parallel transfer mode for metadata operations to collective (``ON``) or independent (``OFF``). +``OPENPMD_HDF5_PAGED_ALLOCATION`` ``ON`` Tuning parameter for parallel I/O in HDF5 to enable paged allocation. +``OPENPMD_HDF5_PAGED_ALLOCATION_SIZE`` ``33554432`` Size of the page, in bytes, if HDF5 paged allocation optimization is enabled. +``OPENPMD_HDF5_DEFER_METADATA`` ``ON`` Tuning parameter for parallel I/O in HDF5 to enable deferred HDF5 metadata operations. +``OPENPMD_HDF5_DEFER_METADATA_SIZE`` ``ON`` Size of the buffer, in bytes, if HDF5 deferred metadata optimization is enabled. +``H5_COLL_API_SANITY_CHECK`` unset Debug: Set to ``1`` to perform an ``MPI_Barrier`` inside each meta-data operation. +``HDF5_USE_FILE_LOCKING`` ``TRUE`` Work-around: Set to ``FALSE`` in case you are on an HPC or network file system that hang in open for reads. +``OMPI_MCA_io`` unset Work-around: Disable OpenMPI's I/O implementation for older releases by setting this to ``^ompio``. +======================================== ============ =========================================================================================================== ``OPENPMD_HDF5_INDEPENDENT``: by default, we implement MPI-parallel data ``storeChunk`` (write) and ``loadChunk`` (read) calls as `none-collective MPI operations `_. Attribute writes are always collective in parallel HDF5. @@ -36,18 +44,63 @@ Although we choose the default to be non-collective (independent) for ease of us For independent parallel I/O, potentially prefer using a modern version of the MPICH implementation (especially, use ROMIO instead of OpenMPI's ompio implementation). Please refer to the `HDF5 manual, function H5Pset_dxpl_mpio `_ for more details. -``OPENPMD_HDF5_ALIGNMENT`` This sets the alignment in Bytes for writes via the ``H5Pset_alignment`` function. +``OPENPMD_HDF5_ALIGNMENT``: this sets the alignment in Bytes for writes via the ``H5Pset_alignment`` function. According to the `HDF5 documentation `_: *For MPI IO and other parallel systems, choose an alignment which is a multiple of the disk block size.* On Lustre filesystems, according to the `NERSC documentation `_, it is advised to set this to the Lustre stripe size. In addition, ORNL Summit GPFS users are recommended to set the alignment value to 16777216(16MB). -``OPENPMD_HDF5_CHUNKS`` This sets defaults for data chunking via `H5Pset_chunk `__. +``OPENPMD_HDF5_THRESHOLD``: this sets the threshold for the alignment of HDF5 operations via the ``H5Pset_alignment`` function. +Setting it to ``0`` will force all requests to be aligned. +Any file object greater than or equal in size to threshold bytes will be aligned on an address which is a multiple of ``OPENPMD_HDF5_ALIGNMENT``. + +``OPENPMD_HDF5_CHUNKS``: this sets defaults for data chunking via `H5Pset_chunk `__. Chunking generally improves performance and only needs to be disabled in corner-cases, e.g. when heavily relying on independent, parallel I/O that non-collectively declares data records. +``OPENPMD_HDF5_COLLECTIVE_METADATA``: this is an option to enable collective MPI calls for HDF5 metadata operations via `H5Pset_all_coll_metadata_ops `__ and `H5Pset_coll_metadata_write `__. +By default, this optimization is enabled as it has proven to provide performance improvements. +This option is only available from HDF5 1.10.0 onwards. For previous version it will fallback to independent MPI calls. + +``OPENPMD_HDF5_PAGED_ALLOCATION``: this option enables paged allocation for HDF5 operations via `H5Pset_file_space_strategy `__. +The page size can be controlled by the ``OPENPMD_HDF5_PAGED_ALLOCATION_SIZE`` option. + +``OPENPMD_HDF5_PAGED_ALLOCATION_SIZE``: this option configures the size of the page if ``OPENPMD_HDF5_PAGED_ALLOCATION`` optimization is enabled via `H5Pset_file_space_page_size `__. +Values are expressed in bytes. Default is set to 32MB. + +``OPENPMD_HDF5_DEFER_METADATA``: this option enables deffered HDF5 metadata operations. +The metadata buffer size can be controlled by the ``OPENPMD_HDF5_DEFER_METADATA_SIZE`` option. + +``OPENPMD_HDF5_DEFER_METADATA_SIZE``: this option configures the size of the buffer if ``OPENPMD_HDF5_DEFER_METADATA`` optimization is enabled via `H5Pset_mdc_config `__. +Values are expressed in bytes. Default is set to 32MB. + ``H5_COLL_API_SANITY_CHECK``: this is a HDF5 control option for debugging parallel I/O logic (API calls). Debugging a parallel program with that option enabled can help to spot bugs such as collective MPI-calls that are not called by all participating MPI ranks. Do not use in production, this will slow parallel I/O operations down. +``HDF5_USE_FILE_LOCKING``: this is a HDF5 1.10.1+ control option that disables HDF5 internal file locking operations (see `HDF5 1.10.1 release notes `__). +This mechanism is mainly used to ensure that a file that is still being written to cannot (yet) be opened by either a reader or another writer. +On some HPC and Jupyter systems, parallel/network file systems like GPFS are mounted in a way that interferes with this internal, HDF5 access consistency check. +As a result, read-only operations like ``h5ls some_file.h5`` or openPMD ``Series`` open can hang indefinitely. +If you are sure that the file was written completely and is closed by the writer, e.g., because a simulation finished that created HDF5 outputs, then you can set this environment variable to ``FALSE`` to work-around the problem. +You should also report this problem to your system support, so they can fix the file system mount options or disable locking by default in the provided HDF5 installation. + +``OMPI_MCA_io``: this is an OpenMPI control variable. +OpenMPI implements its own MPI-I/O implementation backend *OMPIO*, starting with `OpenMPI 2.x `__ . +This backend is known to cause problems in older releases that might still be in use on some systems. +Specifically, `we found and reported a silent data corruption issue `__ that was fixed only in `OpenMPI versions 3.0.4, 3.1.4, 4.0.1 `__ and newer. +There are also problems in OMPIO with writes larger than 2GB, which have only been fixed in `OpenMPI version 3.0.5, 3.1.5, 4.0.3 `__ and newer. +Using ``export OMPI_MCA_io=^ompio`` before ``mpiexec``/``mpirun``/``srun``/``jsrun`` will disable OMPIO and instead fall back to the older *ROMIO* MPI-I/O backend in OpenMPI. + + +Known Issues +------------ + +.. warning:: + + Jul 23th, 2021 (`HDFFV-11260 `__): + Collective HDF5 metadata reads became broken in 1.10.5. + Consider using 1.10.4 if you plan to enable the collective HDF5 metadata operations optimization in openPMD (``OPENPMD_HDF5_COLLECTIVE_METADATA=ON``). + Enabling this feature with a newer version will make HDF5 fall back to the individual metadata operations. + HDF5 plans to fix the issue in the upcoming 1.10.8+ and 1.12.2+ releases, but visit the issue tracker above to see the status of the bug fix. Selected References ------------------- diff --git a/docs/source/conf.py b/docs/source/conf.py index 317b8b9e97..530ee1397b 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -83,9 +83,9 @@ # built documents. # # The short X.Y version. -version = u'0.14.2' +version = u'0.14.3' # The full version, including alpha/beta/rc tags. -release = u'0.14.2' +release = u'0.14.3' # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. diff --git a/docs/source/details/backendconfig.rst b/docs/source/details/backendconfig.rst index 7aea3a24ab..7d89976731 100644 --- a/docs/source/details/backendconfig.rst +++ b/docs/source/details/backendconfig.rst @@ -36,7 +36,7 @@ For a consistent user interface, backends shall follow the following rules: Backend-independent JSON configuration -------------------------------------- -The key ``defer_iteration_parsing`` can be used to optimize the process of opening an openPMD Series. +The key ``defer_iteration_parsing`` can be used to optimize the process of opening an openPMD Series (deferred/lazy parsing). By default, a Series is parsed eagerly, i.e. opening a Series implies reading all available iterations. Especially when a Series has many iterations, this can be a costly operation and users may wish to defer parsing of iterations to a later point adding ``{"defer_iteration_parsing": true}`` to their JSON configuration. @@ -71,7 +71,7 @@ Explanation of the single keys: Please refer to the `official ADIOS2 documentation `_ for a list of available engines. * ``adios2.engine.parameters``: An associative array of string-formatted engine parameters, passed directly through to ``adios2::IO::SetParameters``. Please refer to the official ADIOS2 documentation for the allowable engine parameters. -* ``adios2.engine.usesteps``: Described more closely in the documentation for the :ref:`ADIOS2 backend`. +* ``adios2.engine.usesteps``: Described more closely in the documentation for the :ref:`ADIOS2 backend` (usesteps). * ``adios2.dataset.operators``: This key contains a list of ADIOS2 `operators `_, used to enable compression or dataset transformations. Each object in the list has two keys: diff --git a/docs/source/index.rst b/docs/source/index.rst index 2b3f951cf3..975b5fbc1c 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -42,7 +42,7 @@ openPMD-api version supported openPMD standard versions ======================== =================================== ``2.0.0+`` ``2.0.0+`` (not released yet) ``1.0.0+`` ``1.0.1-1.1.0`` (not released yet) -``0.13.1-0.14.2`` (beta) ``1.0.0-1.1.0`` +``0.13.1-0.14.3`` (beta) ``1.0.0-1.1.0`` ``0.1.0-0.12.0`` (alpha) ``1.0.0-1.1.0`` ======================== =================================== diff --git a/docs/source/install/install.rst b/docs/source/install/install.rst index e67c2ec43c..3b5f54dcd1 100644 --- a/docs/source/install/install.rst +++ b/docs/source/install/install.rst @@ -36,7 +36,7 @@ A package for openPMD-api is available via the `Spack `_ packa # optional: +python +adios1 -adios2 -hdf5 -mpi spack install openpmd-api - spack load -r openpmd-api + spack load openpmd-api .. _install-conda: @@ -146,7 +146,7 @@ Linux & OSX # for options append: # -DopenPMD_USE_...=... # e.g. for python support add: - # -DopenPMD_USE_PYTHON=ON + # -DopenPMD_USE_PYTHON=ON -DPython_EXECUTABLE=$(which python3) cmake ../openPMD-api cmake --build . diff --git a/docs/source/usage/benchmarks.rst b/docs/source/usage/benchmarks.rst index 27f0853c4f..148ceb2253 100644 --- a/docs/source/usage/benchmarks.rst +++ b/docs/source/usage/benchmarks.rst @@ -96,7 +96,13 @@ The options are: a file prefix, and a read pattern For example, if the files are in the format of ``/path/8a_parallel_3Db_%07T.bp`` -the input will be: ``/path/8a_parallel_3Db `` +the input can be simply: ``/path/8a_parallel_3Db `` + +otherwise, please use the full name of the file. + +While openPMD-api supports more than one file types, this benchmark intents to read just one type. +By default, ADIOS2 file is assumed. If it is not the desired file type, one can hint with an environment variable, e.g. +``export OPENPMD_BENCHMARK_USE_BACKEND=HDF5`` The Read options intent to measure overall processing time in the following categories: diff --git a/examples/8b_benchmark_read_parallel.cpp b/examples/8b_benchmark_read_parallel.cpp index 4def72d913..0e199613bd 100644 --- a/examples/8b_benchmark_read_parallel.cpp +++ b/examples/8b_benchmark_read_parallel.cpp @@ -128,6 +128,7 @@ class Timer //MemoryProfiler (rank, tag); } ~Timer() { + MPI_Barrier(MPI_COMM_WORLD); std::string tt = "~"+m_Tag; //MemoryProfiler (m_Rank, tt.c_str()); m_End = std::chrono::system_clock::now(); @@ -188,9 +189,13 @@ std::vector getBackends() { #if openPMD_HAVE_ADIOS2 if( auxiliary::getEnvString( "OPENPMD_BP_BACKEND", "NOT_SET" ) != "ADIOS1" ) res.emplace_back(".bp"); + if( auxiliary::getEnvString( "OPENPMD_BENCHMARK_USE_BACKEND", "NOT_SET" ) == "ADIOS" ) + return res; #endif #if openPMD_HAVE_HDF5 + if( auxiliary::getEnvString( "OPENPMD_BENCHMARK_USE_BACKEND", "NOT_SET" ) == "HDF5" ) + res.clear(); res.emplace_back(".h5"); #endif return res; @@ -211,30 +216,22 @@ class TestInput * Run the read tests * assumes both GroupBased and fileBased series of this prefix exist. * @ param prefix file prefix + * e.g. abc.bp (for group/variable based encoding) + * abc (for file based encoding) * */ void run(const std::string& prefix) { - { // file based - std::ostringstream s; - s < 1) ); bool atBottomRight = ( (m_Pattern % 10 == 2) && (fractionOnDim > 1) ); + bool overlay = ( (m_Pattern % 10 == 3) && (fractionOnDim > 1) ); bool rankZeroOnly = ( alongDim == 4); bool diagnalBlocks = ( alongDim > meshExtent.size() ) && !rankZeroOnly; @@ -404,6 +408,8 @@ class TestInput s <<" topleft "; else if (atBottomRight) s <<" bottomRight "; + else if (overlay) + s << " near center "; } else if ( diagnalBlocks ) s <<" blockStyle = diagnal"; else @@ -423,12 +429,14 @@ class TestInput if ( rankZeroOnly ) { - if ( atCenter ) + if ( atTopLeft ) off[i] = 0; // top corner - else if ( atTopLeft ) + else if ( atBottomRight ) off[i] = (meshExtent[i]-blob); // bottom corner - else if (atBottomRight) + else if (atCenter) off[i] = (fractionOnDim/2) * blob; // middle corner + else if (overlay) + off[i] = (fractionOnDim/2) * blob - blob/3; // near middle corner } else { @@ -456,7 +464,7 @@ class TestInput auto slice_data = rho.loadChunk(off, ext); series.flush(); - std::cout<<"Rank: "<= 100 ) return; @@ -592,7 +600,7 @@ class TestInput return; whichDim -= 5; - MeshRecordComponent bx = series.iterations[ts].meshes["B"]["x"]; + MeshRecordComponent bx = iter.meshes["B"]["x"]; Extent meshExtent = bx.getExtent(); if ( bx.getExtent().size() != 3) { @@ -601,15 +609,15 @@ class TestInput return; } - MeshRecordComponent by = series.iterations[ts].meshes["B"]["y"]; - MeshRecordComponent bz = series.iterations[ts].meshes["B"]["z"]; + MeshRecordComponent by = iter.meshes["B"]["y"]; + MeshRecordComponent bz = iter.meshes["B"]["z"]; Offset off(meshExtent.size(),0); Extent ext(meshExtent.size(),1); std::ostringstream s; - s<<" Eletrict Field slice: "; + s<<" Electric Field slice: "; if (!getSlice(meshExtent, whichDim, rankZeroOnly, off, ext, s)) return; @@ -626,52 +634,70 @@ class TestInput * Read an iteration step, mesh & particles * * @param Series openPMD series + * @param iter iteration (actual iteration step may not equal to ts) * @param ts timestep * */ void - readStep( Series& series, int ts ) + readStep( Series& series, IndexedIteration& iter, int ts ) { std::string comp_name = openPMD::MeshRecordComponent::SCALAR; - MeshRecordComponent rho = series.iterations[ts].meshes["rho"][comp_name]; - + MeshRecordComponent rho = iter.meshes["rho"][comp_name]; Extent meshExtent = rho.getExtent(); if ( 0 == m_MPIRank ) { - std::cout<<"... rho meshExtent : ts="< rho meshExtent : ts=" << ts << " ["; for (unsigned long i : meshExtent) std::cout< currPatterns; + if (m_Pattern > 0) + currPatterns.push_back(m_Pattern); + else + currPatterns.insert(currPatterns.end(), { 1, 5, 15, 25, 55, 65, 75, 440, 441, 442, 443, 7 }); - sliceField(series, ts); + for(int i : currPatterns) { + m_Pattern = i; + sliceMe(series, rho); + block(series, rho); + fullscan(series, rho); - // read particles - if ( m_Pattern == 7 ) - { - openPMD::ParticleSpecies electrons = - series.iterations[ts].particles["ion"]; - RecordComponent charge = electrons["charge"][RecordComponent::SCALAR]; - sliceParticles(series, charge); - } + sliceField(series, iter); + + sliceParticles(series, iter); + } + if (currPatterns.size() > 1) + m_Pattern = 0; } /* - * Read a slice of particles + * Read a slice of id of the first particle * * @param series openPMD Series - * @param charge Particle record + * @param iter current iteration * */ - void sliceParticles(Series& series, RecordComponent& charge) + void sliceParticles(Series& series, IndexedIteration& iter) { - Extent pExtent = charge.getExtent(); + // read id of the first particle found + if ( m_Pattern != 7 ) + return; + + if ( 0 == iter.particles.size() ) + { + if ( 0 == m_MPIRank ) + std::cerr << " No Particles found. Skipping particle slicing. " << std::endl; + return; + } + + openPMD::ParticleSpecies p = iter.particles.begin()->second; + RecordComponent idVal = p["id"][RecordComponent::SCALAR]; + + Extent pExtent = idVal.getExtent(); auto blob = pExtent[0]/(10*m_MPISize); if (0 == blob) @@ -689,7 +715,7 @@ class TestInput Offset colOff = {m_MPIRank*blob}; Extent colExt = {blob}; - auto col_data = charge.loadChunk(colOff, colExt); + auto col_data = idVal.loadChunk(colOff, colExt); series.flush(); } @@ -725,6 +751,7 @@ main( int argc, char *argv[] ) return 0; } + { Timer g( " Main ", input.m_MPIRank ); std::string prefix = argv[1]; @@ -759,7 +786,7 @@ main( int argc, char *argv[] ) input.m_Backend = which; input.run(prefix); } - + } // Timer g MPI_Finalize(); return 0; diff --git a/include/openPMD/IO/HDF5/HDF5IOHandlerImpl.hpp b/include/openPMD/IO/HDF5/HDF5IOHandlerImpl.hpp index 5ec2c4af6e..815c57516e 100644 --- a/include/openPMD/IO/HDF5/HDF5IOHandlerImpl.hpp +++ b/include/openPMD/IO/HDF5/HDF5IOHandlerImpl.hpp @@ -70,6 +70,9 @@ namespace openPMD hid_t m_datasetTransferProperty; hid_t m_fileAccessProperty; + hid_t m_fileCreateProperty; + + hbool_t m_hdf5_collective_metadata = 1; // h5py compatible types for bool and complex hid_t m_H5T_BOOL_ENUM; diff --git a/include/openPMD/Series.hpp b/include/openPMD/Series.hpp index 2469d5c29c..dac26922f4 100644 --- a/include/openPMD/Series.hpp +++ b/include/openPMD/Series.hpp @@ -478,6 +478,9 @@ class Series : public SeriesInterface private: std::shared_ptr< internal::SeriesInternal > m_series; + // constructor from private parts + Series( std::shared_ptr< internal::SeriesInternal > ); + public: explicit Series(); diff --git a/include/openPMD/auxiliary/VariantSrc.hpp b/include/openPMD/auxiliary/VariantSrc.hpp index b6e8de49b0..5b6eda1dfc 100644 --- a/include/openPMD/auxiliary/VariantSrc.hpp +++ b/include/openPMD/auxiliary/VariantSrc.hpp @@ -1,4 +1,4 @@ -/* Copyright 2002-2021 Axel Huebl +/* Copyright 2020-2021 Axel Huebl * * This file is part of openPMD-api. * @@ -20,7 +20,9 @@ */ #pragma once -#if __cplusplus >= 201703L +#include "openPMD/config.hpp" + +#if openPMD_HAS_CXX17 # include // IWYU pragma: export namespace variantSrc = std; #else diff --git a/include/openPMD/backend/Attributable.hpp b/include/openPMD/backend/Attributable.hpp index a1e0c05a95..157d732345 100644 --- a/include/openPMD/backend/Attributable.hpp +++ b/include/openPMD/backend/Attributable.hpp @@ -434,7 +434,12 @@ AttributableInterface::setAttribute( std::string const & key, T value ) && !attri.m_attributes.key_comp()(key, it->first) ) { // key already exists in map, just replace the value - it->second = Attribute(value); + + // note: due to a C++17 issue with NVCC 11.0.2 we write the + // T value to variant conversion explicitly + // https://github.com/openPMD/openPMD-api/pull/1103 + //it->second = Attribute(std::move(value)); + it->second = Attribute(Attribute::resource(std::move(value))); return true; } else { diff --git a/include/openPMD/cli/ls.hpp b/include/openPMD/cli/ls.hpp index 60f3100d25..cefe36682d 100644 --- a/include/openPMD/cli/ls.hpp +++ b/include/openPMD/cli/ls.hpp @@ -98,7 +98,8 @@ namespace ls try { auto s = Series( argv[1], - Access::READ_ONLY + Access::READ_ONLY, + R"({"defer_iteration_parsing": true})" ); helper::listSeries(s, true, std::cout); diff --git a/include/openPMD/config.hpp.in b/include/openPMD/config.hpp.in index 53bc3c08af..63be511355 100644 --- a/include/openPMD/config.hpp.in +++ b/include/openPMD/config.hpp.in @@ -20,6 +20,10 @@ */ #pragma once +#ifndef openPMD_HAS_CXX17 +# cmakedefine01 openPMD_HAS_CXX17 +#endif + #ifndef openPMD_HAVE_MPI # cmakedefine01 openPMD_HAVE_MPI #endif diff --git a/include/openPMD/openPMD.hpp b/include/openPMD/openPMD.hpp index b5625e3191..d385709628 100644 --- a/include/openPMD/openPMD.hpp +++ b/include/openPMD/openPMD.hpp @@ -57,6 +57,7 @@ namespace openPMD {} #include "openPMD/auxiliary/Option.hpp" #include "openPMD/auxiliary/OutOfRangeMsg.hpp" #include "openPMD/auxiliary/ShareRaw.hpp" +#include "openPMD/auxiliary/StringManip.hpp" #include "openPMD/auxiliary/Variant.hpp" #include "openPMD/helper/list_series.hpp" diff --git a/include/openPMD/version.hpp b/include/openPMD/version.hpp index 89e2c1498f..dd264b0682 100644 --- a/include/openPMD/version.hpp +++ b/include/openPMD/version.hpp @@ -29,7 +29,7 @@ */ #define OPENPMDAPI_VERSION_MAJOR 0 #define OPENPMDAPI_VERSION_MINOR 14 -#define OPENPMDAPI_VERSION_PATCH 2 +#define OPENPMDAPI_VERSION_PATCH 3 #define OPENPMDAPI_VERSION_LABEL "" /** @} */ diff --git a/setup.py b/setup.py index 59497492f2..63b6256166 100644 --- a/setup.py +++ b/setup.py @@ -156,7 +156,7 @@ def build_extension(self, ext): setup( name='openPMD-api', # note PEP-440 syntax: x.y.zaN but x.y.z.devN - version='0.14.2', + version='0.14.3', author='Axel Huebl, Franz Poeschel, Fabian Koller, Junmin Gu', author_email='axelhuebl@lbl.gov, f.poeschel@hzdr.de', maintainer='Axel Huebl', diff --git a/share/openPMD/download_samples.ps1 b/share/openPMD/download_samples.ps1 index 3e2f2eae83..5fc8987c4d 100755 --- a/share/openPMD/download_samples.ps1 +++ b/share/openPMD/download_samples.ps1 @@ -5,6 +5,8 @@ Param( ) $orgdir = $(Get-Location | Foreach-Object { $_.Path }) + +$null = New-Item -Type Directory -Force $bdir cd $bdir New-item -ItemType directory -Name samples\git-sample\thetaMode\ diff --git a/src/IO/HDF5/HDF5IOHandler.cpp b/src/IO/HDF5/HDF5IOHandler.cpp index f024cff6a6..09ddc63fd4 100644 --- a/src/IO/HDF5/HDF5IOHandler.cpp +++ b/src/IO/HDF5/HDF5IOHandler.cpp @@ -117,6 +117,14 @@ HDF5IOHandlerImpl::HDF5IOHandlerImpl( << shadow << std::endl; } } + +#if H5_VERSION_GE(1,10,0) && openPMD_HAVE_MPI + auto const hdf5_collective_metadata = auxiliary::getEnvString( "OPENPMD_HDF5_COLLECTIVE_METADATA", "ON" ); + if( hdf5_collective_metadata == "ON" ) + m_hdf5_collective_metadata = 1; + else + m_hdf5_collective_metadata = 0; +#endif } HDF5IOHandlerImpl::~HDF5IOHandlerImpl() @@ -202,6 +210,16 @@ HDF5IOHandlerImpl::createPath(Writable* writable, if( m_handler->m_backendAccess == Access::READ_ONLY ) throw std::runtime_error("[HDF5] Creating a path in a file opened as read only is not possible."); + hid_t gapl = H5Pcreate(H5P_GROUP_ACCESS); +#if H5_VERSION_GE(1,10,0) && openPMD_HAVE_MPI + if( m_hdf5_collective_metadata ) + { + H5Pset_all_coll_metadata_ops(gapl, true); + } +#endif + + herr_t status; + if( !writable->written ) { /* Sanitize path */ @@ -220,7 +238,7 @@ HDF5IOHandlerImpl::createPath(Writable* writable, File file = getFile( position ).get(); hid_t node_id = H5Gopen(file.id, concrete_h5_file_position(position).c_str(), - H5P_DEFAULT); + gapl); VERIFY(node_id >= 0, "[HDF5] Internal error: Failed to open HDF5 group during path creation"); /* Create the path in the file */ @@ -243,7 +261,6 @@ HDF5IOHandlerImpl::createPath(Writable* writable, } /* Close the groups */ - herr_t status; while( !groups.empty() ) { status = H5Gclose(groups.top()); @@ -256,6 +273,9 @@ HDF5IOHandlerImpl::createPath(Writable* writable, m_fileNames[writable] = file.name; } + + status = H5Pclose(gapl); + VERIFY(status == 0, "[HDF5] Internal error: Failed to close HDF5 property during path creation"); } void @@ -303,12 +323,20 @@ HDF5IOHandlerImpl::createDataset(Writable* writable, } } + hid_t gapl = H5Pcreate(H5P_GROUP_ACCESS); +#if H5_VERSION_GE(1,10,0) && openPMD_HAVE_MPI + if( m_hdf5_collective_metadata ) + { + H5Pset_all_coll_metadata_ops(gapl, true); + } +#endif + /* Open H5Object to write into */ auto res = getFile( writable ); File file = res ? res.get() : getFile( writable->parent ).get(); hid_t node_id = H5Gopen(file.id, concrete_h5_file_position(writable).c_str(), - H5P_DEFAULT); + gapl); VERIFY(node_id >= 0, "[HDF5] Internal error: Failed to open HDF5 group during dataset creation"); Datatype d = parameters.dtype; @@ -409,6 +437,8 @@ HDF5IOHandlerImpl::createDataset(Writable* writable, VERIFY(status == 0, "[HDF5] Internal error: Failed to close HDF5 dataset space during dataset creation"); status = H5Gclose(node_id); VERIFY(status == 0, "[HDF5] Internal error: Failed to close HDF5 group during dataset creation"); + status = H5Pclose(gapl); + VERIFY(status == 0, "[HDF5] Internal error: Failed to close HDF5 property during dataset creation"); writable->written = true; writable->abstractFilePosition = std::make_shared< HDF5FilePosition >(name); @@ -600,9 +630,18 @@ HDF5IOHandlerImpl::openPath( { File file = getFile(writable->parent).get(); hid_t node_id, path_id; + + hid_t gapl = H5Pcreate(H5P_GROUP_ACCESS); +#if H5_VERSION_GE(1,10,0) && openPMD_HAVE_MPI + if( m_hdf5_collective_metadata ) + { + H5Pset_all_coll_metadata_ops(gapl, true); + } +#endif + node_id = H5Gopen(file.id, concrete_h5_file_position(writable->parent).c_str(), - H5P_DEFAULT); + gapl); VERIFY(node_id >= 0, "[HDF5] Internal error: Failed to open HDF5 group during path opening"); /* Sanitize path */ @@ -615,7 +654,7 @@ HDF5IOHandlerImpl::openPath( path += '/'; path_id = H5Gopen(node_id, path.c_str(), - H5P_DEFAULT); + gapl); VERIFY(path_id >= 0, "[HDF5] Internal error: Failed to open HDF5 group during path opening"); herr_t status; @@ -626,6 +665,8 @@ HDF5IOHandlerImpl::openPath( herr_t status; status = H5Gclose(node_id); VERIFY(status == 0, "[HDF5] Internal error: Failed to close HDF5 group during path opening"); + status = H5Pclose(gapl); + VERIFY(status == 0, "[HDF5] Internal error: Failed to close HDF5 property during path opening"); writable->written = true; writable->abstractFilePosition = std::make_shared< HDF5FilePosition >(path); @@ -640,9 +681,18 @@ HDF5IOHandlerImpl::openDataset(Writable* writable, { File file = getFile( writable->parent ).get(); hid_t node_id, dataset_id; + + hid_t gapl = H5Pcreate(H5P_GROUP_ACCESS); +#if H5_VERSION_GE(1,10,0) && openPMD_HAVE_MPI + if( m_hdf5_collective_metadata ) + { + H5Pset_all_coll_metadata_ops(gapl, true); + } +#endif + node_id = H5Gopen(file.id, concrete_h5_file_position(writable->parent).c_str(), - H5P_DEFAULT); + gapl); VERIFY(node_id >= 0, "[HDF5] Internal error: Failed to open HDF5 group during dataset opening"); /* Sanitize name */ @@ -731,6 +781,8 @@ HDF5IOHandlerImpl::openDataset(Writable* writable, VERIFY(status == 0, "[HDF5] Internal error: Failed to close HDF5 dataset during dataset opening"); status = H5Gclose(node_id); VERIFY(status == 0, "[HDF5] Internal error: Failed to close HDF5 group during dataset opening"); + status = H5Pclose(gapl); + VERIFY(status == 0, "[HDF5] Internal error: Failed to close HDF5 property during dataset opening"); writable->written = true; writable->abstractFilePosition = std::make_shared< HDF5FilePosition >(name); @@ -989,9 +1041,18 @@ HDF5IOHandlerImpl::writeAttribute(Writable* writable, auto res = getFile( writable ); File file = res ? res.get() : getFile( writable->parent ).get(); hid_t node_id, attribute_id; + + hid_t fapl = H5Pcreate(H5P_LINK_ACCESS); +#if H5_VERSION_GE(1,10,0) && openPMD_HAVE_MPI + if( m_hdf5_collective_metadata ) + { + H5Pset_all_coll_metadata_ops(fapl, true); + } +#endif + node_id = H5Oopen(file.id, concrete_h5_file_position(writable).c_str(), - H5P_DEFAULT); + fapl); VERIFY(node_id >= 0, "[HDF5] Internal error: Failed to open HDF5 object during attribute write"); Attribute const att(parameters.resource); Datatype dtype = parameters.dtype; @@ -1248,6 +1309,8 @@ HDF5IOHandlerImpl::writeAttribute(Writable* writable, VERIFY(status == 0, "[HDF5] Internal error: Failed to close attribute " + name + " at " + concrete_h5_file_position(writable) + " during attribute write"); status = H5Oclose(node_id); VERIFY(status == 0, "[HDF5] Internal error: Failed to close " + concrete_h5_file_position(writable) + " during attribute write"); + status = H5Pclose(fapl); + VERIFY(status == 0, "[HDF5] Internal error: Failed to close HDF5 property during attribute write"); m_fileNames[writable] = file.name; } @@ -1353,9 +1416,18 @@ HDF5IOHandlerImpl::readAttribute(Writable* writable, hid_t obj_id, attr_id; herr_t status; + + hid_t fapl = H5Pcreate(H5P_LINK_ACCESS); +#if H5_VERSION_GE(1,10,0) && openPMD_HAVE_MPI + if( m_hdf5_collective_metadata ) + { + H5Pset_all_coll_metadata_ops(fapl, true); + } +#endif + obj_id = H5Oopen(file.id, concrete_h5_file_position(writable).c_str(), - H5P_DEFAULT); + fapl); VERIFY(obj_id >= 0, std::string("[HDF5] Internal error: Failed to open HDF5 object '") + concrete_h5_file_position(writable).c_str() + "' during attribute read"); std::string const & attr_name = parameters.name; @@ -1769,6 +1841,8 @@ HDF5IOHandlerImpl::readAttribute(Writable* writable, VERIFY(status == 0, "[HDF5] Internal error: Failed to close attribute " + attr_name + " at " + concrete_h5_file_position(writable) + " during attribute read"); status = H5Oclose(obj_id); VERIFY(status == 0, "[HDF5] Internal error: Failed to close " + concrete_h5_file_position(writable) + " during attribute read"); + status = H5Pclose(fapl); + VERIFY(status == 0, "[HDF5] Internal error: Failed to close HDF5 attribute during attribute read"); } void @@ -1780,9 +1854,18 @@ HDF5IOHandlerImpl::listPaths(Writable* writable, auto res = getFile( writable ); File file = res ? res.get() : getFile( writable->parent ).get(); + + hid_t gapl = H5Pcreate(H5P_GROUP_ACCESS); +#if H5_VERSION_GE(1,10,0) && openPMD_HAVE_MPI + if( m_hdf5_collective_metadata ) + { + H5Pset_all_coll_metadata_ops(gapl, true); + } +#endif + hid_t node_id = H5Gopen(file.id, concrete_h5_file_position(writable).c_str(), - H5P_DEFAULT); + gapl); VERIFY(node_id >= 0, "[HDF5] Internal error: Failed to open HDF5 group during path listing"); H5G_info_t group_info; @@ -1803,6 +1886,8 @@ HDF5IOHandlerImpl::listPaths(Writable* writable, status = H5Gclose(node_id); VERIFY(status == 0, "[HDF5] Internal error: Failed to close HDF5 group " + concrete_h5_file_position(writable) + " during path listing"); + status = H5Pclose(gapl); + VERIFY(status == 0, "[HDF5] Internal error: Failed to close HDF5 property during path listing"); } void @@ -1814,9 +1899,18 @@ HDF5IOHandlerImpl::listDatasets(Writable* writable, auto res = getFile( writable ); File file = res ? res.get() : getFile( writable->parent ).get(); + + hid_t gapl = H5Pcreate(H5P_GROUP_ACCESS); +#if H5_VERSION_GE(1,10,0) && openPMD_HAVE_MPI + if( m_hdf5_collective_metadata ) + { + H5Pset_all_coll_metadata_ops(gapl, true); + } +#endif + hid_t node_id = H5Gopen(file.id, concrete_h5_file_position(writable).c_str(), - H5P_DEFAULT); + gapl); VERIFY(node_id >= 0, "[HDF5] Internal error: Failed to open HDF5 group during dataset listing"); H5G_info_t group_info; @@ -1837,6 +1931,8 @@ HDF5IOHandlerImpl::listDatasets(Writable* writable, status = H5Gclose(node_id); VERIFY(status == 0, "[HDF5] Internal error: Failed to close HDF5 group " + concrete_h5_file_position(writable) + " during dataset listing"); + status = H5Pclose(gapl); + VERIFY(status == 0, "[HDF5] Internal error: Failed to close HDF5 property during dataset listing"); } void HDF5IOHandlerImpl::listAttributes(Writable* writable, @@ -1848,9 +1944,18 @@ void HDF5IOHandlerImpl::listAttributes(Writable* writable, auto res = getFile( writable ); File file = res ? res.get() : getFile( writable->parent ).get(); hid_t node_id; + + hid_t fapl = H5Pcreate(H5P_LINK_ACCESS); +#if H5_VERSION_GE(1,10,0) && openPMD_HAVE_MPI + if( m_hdf5_collective_metadata ) + { + H5Pset_all_coll_metadata_ops(fapl, true); + } +#endif + node_id = H5Oopen(file.id, concrete_h5_file_position(writable).c_str(), - H5P_DEFAULT); + fapl); VERIFY(node_id >= 0, "[HDF5] Internal error: Failed to open HDF5 group during attribute listing"); herr_t status; @@ -1888,6 +1993,8 @@ void HDF5IOHandlerImpl::listAttributes(Writable* writable, status = H5Oclose(node_id); VERIFY(status == 0, "[HDF5] Internal error: Failed to close HDF5 object during attribute listing"); + status = H5Pclose(fapl); + VERIFY(status == 0, "[HDF5] Internal error: Failed to close HDF5 property during dataset listing"); } auxiliary::Option< HDF5IOHandlerImpl::File > diff --git a/src/IO/HDF5/ParallelHDF5IOHandler.cpp b/src/IO/HDF5/ParallelHDF5IOHandler.cpp index 84ad9dc90d..19c9621393 100644 --- a/src/IO/HDF5/ParallelHDF5IOHandler.cpp +++ b/src/IO/HDF5/ParallelHDF5IOHandler.cpp @@ -27,6 +27,7 @@ #endif #include +#include namespace openPMD @@ -60,6 +61,39 @@ ParallelHDF5IOHandlerImpl::ParallelHDF5IOHandlerImpl( { m_datasetTransferProperty = H5Pcreate(H5P_DATASET_XFER); m_fileAccessProperty = H5Pcreate(H5P_FILE_ACCESS); + m_fileCreateProperty = H5Pcreate(H5P_FILE_CREATE); + +#if H5_VERSION_GE(1,10,1) + auto const hdf5_spaced_allocation = auxiliary::getEnvString( "OPENPMD_HDF5_PAGED_ALLOCATION", "ON" ); + if( hdf5_spaced_allocation == "ON" ) { + auto const strPageSize = auxiliary::getEnvString( "OPENPMD_HDF5_PAGED_ALLOCATION_SIZE", "33554432" ); + std::stringstream tstream(strPageSize); + hsize_t page_size; + tstream >> page_size; + + H5Pset_file_space_strategy(m_fileCreateProperty, H5F_FSPACE_STRATEGY_PAGE, 0, (hsize_t)0); + H5Pset_file_space_page_size(m_fileCreateProperty, page_size); + } +#endif + + auto const hdf5_defer_metadata = auxiliary::getEnvString( "OPENPMD_HDF5_DEFER_METADATA", "ON" ); + if( hdf5_defer_metadata == "ON" ) { + auto const strMetaSize = auxiliary::getEnvString( "OPENPMD_HDF5_DEFER_METADATA_SIZE", "33554432" ); + std::stringstream tstream(strMetaSize); + hsize_t meta_size; + tstream >> meta_size; + + H5AC_cache_config_t cache_config; + cache_config.version = H5AC__CURR_CACHE_CONFIG_VERSION; + H5Pget_mdc_config(m_fileAccessProperty, &cache_config); + cache_config.set_initial_size = 1; + cache_config.initial_size = meta_size; + cache_config.evictions_enabled = 0; + cache_config.incr_mode = H5C_incr__off; + cache_config.flash_incr_mode = H5C_flash_incr__off; + cache_config.decr_mode = H5C_decr__off; + H5Pset_mdc_config(m_fileAccessProperty, &cache_config); + } H5FD_mpio_xfer_t xfer_mode = H5FD_MPIO_COLLECTIVE; auto const hdf5_collective = auxiliary::getEnvString( "OPENPMD_HDF5_INDEPENDENT", "ON" ); @@ -73,13 +107,26 @@ ParallelHDF5IOHandlerImpl::ParallelHDF5IOHandlerImpl( herr_t status; status = H5Pset_dxpl_mpio(m_datasetTransferProperty, xfer_mode); +#if H5_VERSION_GE(1,10,0) + status = H5Pset_all_coll_metadata_ops(m_fileAccessProperty, m_hdf5_collective_metadata); + VERIFY(status >= 0, "[HDF5] Internal error: Failed to set metadata read HDF5 file access property"); + + status = H5Pset_coll_metadata_write(m_fileAccessProperty, m_hdf5_collective_metadata); + VERIFY(status >= 0, "[HDF5] Internal error: Failed to set metadata write HDF5 file access property"); +#endif + auto const strByte = auxiliary::getEnvString( "OPENPMD_HDF5_ALIGNMENT", "1" ); std::stringstream sstream(strByte); hsize_t bytes; sstream >> bytes; + auto const strThreshold = auxiliary::getEnvString( "OPENPMD_HDF5_THRESHOLD", "0" ); + std::stringstream tstream(strThreshold); + hsize_t threshold; + tstream >> threshold; + if ( bytes > 1 ) - H5Pset_alignment(m_fileAccessProperty, 0, bytes); + H5Pset_alignment(m_fileAccessProperty, threshold, bytes); VERIFY(status >= 0, "[HDF5] Internal error: Failed to set HDF5 dataset transfer property"); status = H5Pset_fapl_mpio(m_fileAccessProperty, m_mpiComm, m_mpiInfo); diff --git a/src/Iteration.cpp b/src/Iteration.cpp index f984f1a3d4..5b932b59cd 100644 --- a/src/Iteration.cpp +++ b/src/Iteration.cpp @@ -391,6 +391,8 @@ void Iteration::read_impl( std::string const & groupPath ) setDt(Attribute(*aRead.resource).get< float >()); else if( *aRead.dtype == DT::DOUBLE ) setDt(Attribute(*aRead.resource).get< double >()); + else if( *aRead.dtype == DT::LONG_DOUBLE ) + setDt(Attribute(*aRead.resource).get< long double >()); else throw std::runtime_error("Unexpected Attribute datatype for 'dt'"); @@ -401,6 +403,8 @@ void Iteration::read_impl( std::string const & groupPath ) setTime(Attribute(*aRead.resource).get< float >()); else if( *aRead.dtype == DT::DOUBLE ) setTime(Attribute(*aRead.resource).get< double >()); + else if( *aRead.dtype == DT::LONG_DOUBLE ) + setTime(Attribute(*aRead.resource).get< long double >()); else throw std::runtime_error("Unexpected Attribute datatype for 'time'"); diff --git a/src/Series.cpp b/src/Series.cpp index 073d55968d..e3491adf5a 100644 --- a/src/Series.cpp +++ b/src/Series.cpp @@ -29,6 +29,7 @@ #include "openPMD/Series.hpp" #include "openPMD/version.hpp" +#include #include #include #include @@ -75,8 +76,11 @@ namespace * bool is True if file could be of type f and matches the iterationEncoding. False otherwise. * int is the amount of padding present in the iteration number %T. Is 0 if bool is False. */ - std::function - matcher(std::string const &prefix, int padding, std::string const &postfix, Format f); + std::function matcher( + std::string const &prefix, + int padding, + std::string const &postfix, + Format f); } // namespace [anonymous] struct SeriesInterface::ParsedInput @@ -435,6 +439,21 @@ void SeriesInterface::init( series.m_filenamePostfix = input->filenamePostfix; series.m_filenamePadding = input->filenamePadding; + if( series.m_iterationEncoding == IterationEncoding::fileBased && + !series.m_filenamePrefix.empty() && + std::isdigit( static_cast< unsigned char >( + *series.m_filenamePrefix.rbegin() ) ) ) + { + std::cerr << R"END( +[Warning] In file-based iteration encoding, it is strongly recommended to avoid +digits as the last characters of the filename prefix. +For instance, a robust pattern is to prepend the expansion pattern +of the filename with an underscore '_'. +Example: 'data_%T.json' or 'simOutput_%06T.h5' +Given file pattern: ')END" + << series.m_name << "'" << std::endl; + } + if(IOHandler()->m_frontendAccess == Access::READ_ONLY || IOHandler()->m_frontendAccess == Access::READ_WRITE ) { /* Allow creation of values in Containers and setting of Attributes @@ -1475,6 +1494,15 @@ SeriesInternal::~SeriesInternal() } } // namespace internal +Series::Series( std::shared_ptr< internal::SeriesInternal > series_in ) + : SeriesInterface{ + series_in.get(), + static_cast< internal::AttributableData * >( series_in.get() ) } + , m_series{ std::move( series_in ) } + , iterations{ m_series->iterations } +{ +} + Series::Series() : SeriesInterface{ nullptr, nullptr }, iterations{} { } @@ -1517,7 +1545,9 @@ Series::operator bool() const ReadIterations Series::readIterations() { - return { *this }; + // Use private constructor instead of copy constructor to avoid + // object slicing + return { this->m_series }; } WriteIterations @@ -1549,71 +1579,53 @@ namespace } std::function - buildMatcher(std::string const ®exPattern) { + buildMatcher(std::string const ®exPattern, int padding) { std::regex pattern(regexPattern); - return [pattern](std::string const &filename) -> Match { + return [pattern, padding](std::string const &filename) -> Match { std::smatch regexMatches; bool match = std::regex_match(filename, regexMatches, pattern); - int padding = match ? regexMatches[1].length() : 0; - return {match, padding, match ? std::stoull(regexMatches[1]) : 0}; - }; + int processedPadding = padding != 0 + ? padding + : ( match ? regexMatches[ 1 ].length() : 0 ); + return { + match, + processedPadding, + match ? std::stoull( regexMatches[ 1 ] ) : 0 }; }; } - std::function - matcher(std::string const &prefix, int padding, std::string const &postfix, Format f) { - switch (f) { - case Format::HDF5: { - std::string nameReg = "^" + prefix + "([[:digit:]]"; - if (padding != 0) - nameReg += "{" + std::to_string(padding) + "}"; - else - nameReg += "+"; - nameReg += +")" + postfix + ".h5$"; - return buildMatcher(nameReg); - } - case Format::ADIOS1: - case Format::ADIOS2: { - std::string nameReg = "^" + prefix + "([[:digit:]]"; - if (padding != 0) - nameReg += "{" + std::to_string(padding) + "}"; - else - nameReg += "+"; - nameReg += +")" + postfix + ".bp$"; - return buildMatcher(nameReg); - } - case Format::ADIOS2_SST: - { - std::string nameReg = "^" + prefix + "([[:digit:]]"; - if( padding != 0 ) - nameReg += "{" + std::to_string(padding) + "}"; - else - nameReg += "+"; - nameReg += + ")" + postfix + ".sst$"; - return buildMatcher(nameReg); - } - case Format::ADIOS2_SSC: - { - std::string nameReg = "^" + prefix + "([[:digit:]]"; - if( padding != 0 ) - nameReg += "{" + std::to_string(padding) + "}"; - else - nameReg += "+"; - nameReg += + ")" + postfix + ".ssc$"; - return buildMatcher(nameReg); - } - case Format::JSON: { - std::string nameReg = "^" + prefix + "([[:digit:]]"; - if (padding != 0) - nameReg += "{" + std::to_string(padding) + "}"; - else - nameReg += "+"; - nameReg += +")" + postfix + ".json$"; - return buildMatcher(nameReg); - } - default: - return [](std::string const &) -> Match { return {false, 0, 0}; }; + std::function matcher( + std::string const &prefix, + int padding, + std::string const &postfix, + Format f) + { + std::string filenameSuffix = suffix( f ); + if( filenameSuffix.empty() ) + { + return [](std::string const &) -> Match { return {false, 0, 0}; }; + } + + std::string nameReg = "^" + prefix; + if (padding != 0) + { + // The part after the question mark: + // The number must be at least `padding` digits long + // The part before the question mark: + // It may be longer than that only if the first digit is not zero + // The outer pair of parentheses is for later extraction of the + // iteration number via std::stoull(regexMatches[1]) + nameReg += "(([1-9][[:digit:]]*)?([[:digit:]]"; + nameReg += "{" + std::to_string(padding) + "}))"; + } + else + { + // No padding specified, any number of digits is ok. + nameReg += "([[:digit:]]"; + nameReg += "+)"; } + nameReg += postfix + filenameSuffix + "$"; + return buildMatcher(nameReg, padding); } } // namespace [anonymous] } // namespace openPMD diff --git a/test/SerialIOTest.cpp b/test/SerialIOTest.cpp index 35ee6351ae..1627e1e9b7 100644 --- a/test/SerialIOTest.cpp +++ b/test/SerialIOTest.cpp @@ -90,7 +90,7 @@ TEST_CASE( "adios2_char_portability", "[serial][adios2]" ) writeAttribute( "/openPMD", std::string( "1.1.0" ) ); writeAttribute( "/openPMDextension", uint32_t( 0 ) ); writeAttribute( "/software", std::string( "openPMD-api" ) ); - writeAttribute( "/softwareVersion", std::string( "0.14.2" ) ); + writeAttribute( "/softwareVersion", std::string( "0.14.3" ) ); IO.DefineAttribute< uint64_t >( "__openPMD_internal/openPMD2_adios2_schema", 20210209 ); @@ -1481,7 +1481,7 @@ void fileBased_write_test(const std::string & backend) auxiliary::remove_directory("../samples/subdir"); { - Series o = Series("../samples/subdir/serial_fileBased_write%08T." + backend, Access::CREATE); + Series o = Series("../samples/subdir/serial_fileBased_write%03T." + backend, Access::CREATE); ParticleSpecies& e_1 = o.iterations[1].particles["e"]; @@ -1556,12 +1556,12 @@ void fileBased_write_test(const std::string & backend) o.flush(); o.iterations[5].setTime(static_cast< double >(5)); } - REQUIRE((auxiliary::file_exists("../samples/subdir/serial_fileBased_write00000001." + backend) - || auxiliary::directory_exists("../samples/subdir/serial_fileBased_write00000001." + backend))); - REQUIRE((auxiliary::file_exists("../samples/subdir/serial_fileBased_write00000002." + backend) - || auxiliary::directory_exists("../samples/subdir/serial_fileBased_write00000002." + backend))); - REQUIRE((auxiliary::file_exists("../samples/subdir/serial_fileBased_write00000003." + backend) - || auxiliary::directory_exists("../samples/subdir/serial_fileBased_write00000003." + backend))); + REQUIRE((auxiliary::file_exists("../samples/subdir/serial_fileBased_write001." + backend) + || auxiliary::directory_exists("../samples/subdir/serial_fileBased_write001." + backend))); + REQUIRE((auxiliary::file_exists("../samples/subdir/serial_fileBased_write002." + backend) + || auxiliary::directory_exists("../samples/subdir/serial_fileBased_write002." + backend))); + REQUIRE((auxiliary::file_exists("../samples/subdir/serial_fileBased_write003." + backend) + || auxiliary::directory_exists("../samples/subdir/serial_fileBased_write003." + backend))); { Series o = Series("../samples/subdir/serial_fileBased_write%T." + backend, Access::READ_ONLY); @@ -1574,12 +1574,12 @@ void fileBased_write_test(const std::string & backend) REQUIRE(o.iterations.count(5) == 1); #if openPMD_USE_INVASIVE_TESTS - REQUIRE(o.get().m_filenamePadding == 8); + REQUIRE(o.get().m_filenamePadding == 3); #endif REQUIRE(o.basePath() == "/data/%T/"); REQUIRE(o.iterationEncoding() == IterationEncoding::fileBased); - REQUIRE(o.iterationFormat() == "serial_fileBased_write%08T"); + REQUIRE(o.iterationFormat() == "serial_fileBased_write%03T"); REQUIRE(o.openPMD() == "1.1.0"); REQUIRE(o.openPMDextension() == 1u); REQUIRE(o.particlesPath() == "particles/"); @@ -1646,15 +1646,37 @@ void fileBased_write_test(const std::string & backend) o.iterations[ 6 ] .particles[ "e" ][ "position" ][ "x" ] .makeConstant< double >( 1.0 ); + + // additional iteration with over-running iteration padding but similar content + // padding: 000 + uint64_t const overlong_it = 123456; + o.iterations[ overlong_it ]; + // write something to trigger opening of the file + o.iterations[ overlong_it ].particles[ "e" ][ "position" ][ "x" ].resetDataset( + { Datatype::DOUBLE, { 12 } } ); + o.iterations[ overlong_it ] + .particles[ "e" ][ "position" ][ "x" ] + .makeConstant< double >( 1.0 ); + + o.iterations[ overlong_it ].setTime(static_cast< double >(overlong_it)); + REQUIRE(o.iterations.size() == 7); } - REQUIRE((auxiliary::file_exists("../samples/subdir/serial_fileBased_write00000004." + backend) - || auxiliary::directory_exists("../samples/subdir/serial_fileBased_write00000004." + backend))); + REQUIRE((auxiliary::file_exists("../samples/subdir/serial_fileBased_write004." + backend) + || auxiliary::directory_exists("../samples/subdir/serial_fileBased_write004." + backend))); + REQUIRE((auxiliary::file_exists("../samples/subdir/serial_fileBased_write123456." + backend) + || auxiliary::directory_exists("../samples/subdir/serial_fileBased_write123456." + backend))); - // additional iteration with different iteration padding but similar content + // additional iteration with shorter iteration padding but similar content { Series o = Series("../samples/subdir/serial_fileBased_write%01T." + backend, Access::READ_WRITE); - REQUIRE(o.iterations.empty()); + REQUIRE(o.iterations.size() == 1); + /* + * 123456 has no padding, it's just a very long number. + * So even when opening the series with a padding of 1, + * that iteration will be opened. + */ + REQUIRE(o.iterations.count(123456) == 1); auto& it = o.iterations[10]; ParticleSpecies& e = it.particles["e"]; @@ -1664,8 +1686,9 @@ void fileBased_write_test(const std::string & backend) e["positionOffset"]["x"].makeConstant(1.23); fileBased_add_EDpic(e, 42); + it.setTime(static_cast< double >(10)); - REQUIRE(o.iterations.size() == 1); + REQUIRE(o.iterations.size() == 2); } REQUIRE((auxiliary::file_exists("../samples/subdir/serial_fileBased_write10." + backend) || auxiliary::directory_exists("../samples/subdir/serial_fileBased_write10." + backend))); @@ -1673,23 +1696,55 @@ void fileBased_write_test(const std::string & backend) // read back with auto-detection and non-fixed padding { Series s = Series("../samples/subdir/serial_fileBased_write%T." + backend, Access::READ_ONLY); - REQUIRE(s.iterations.size() == 7); + REQUIRE(s.iterations.size() == 8); + REQUIRE(s.iterations.contains(4)); + REQUIRE(s.iterations.contains(10)); + REQUIRE(s.iterations.contains(123456)); + + REQUIRE(s.iterations[3].time< double >() == 3.0); + REQUIRE(s.iterations[4].time< double >() == 4.0); + REQUIRE(s.iterations[5].time< double >() == 5.0); + REQUIRE(s.iterations[10].time< double >() == 10.0); + REQUIRE(s.iterations[123456].time< double >() == double(123456)); } - // write with auto-detection and in-consistent padding + // write with auto-detection and in-consistent padding from step 10 { REQUIRE_THROWS_WITH(Series("../samples/subdir/serial_fileBased_write%T." + backend, Access::READ_WRITE), Catch::Equals("Cannot write to a series with inconsistent iteration padding. Please specify '%0T' or open as read-only.")); } - // read back with auto-detection and fixed padding + // read back with fixed padding + { + Series s = Series("../samples/subdir/serial_fileBased_write%03T." + backend, Access::READ_ONLY); + REQUIRE(s.iterations.size() == 7); + REQUIRE(s.iterations.contains(4)); + REQUIRE(!s.iterations.contains(10)); + REQUIRE(s.iterations.contains(123456)); + + REQUIRE(s.iterations[3].time< double >() == 3.0); + REQUIRE(s.iterations[4].time< double >() == 4.0); + REQUIRE(s.iterations[5].time< double >() == 5.0); + } + + // read back with auto-detection (allow relaxed/overflow padding) { - Series s = Series("../samples/subdir/serial_fileBased_write%08T." + backend, Access::READ_ONLY); - REQUIRE(s.iterations.size() == 6); + Series s = Series("../samples/subdir/serial_fileBased_write%T." + backend, Access::READ_ONLY); + REQUIRE(s.iterations.size() == 8); + REQUIRE(s.iterations.contains(4)); + REQUIRE(s.iterations.contains(10)); + REQUIRE(s.iterations.contains(123456)); + + REQUIRE(s.iterations[3].time< double >() == 3.0); + REQUIRE(s.iterations[4].time< double >() == 4.0); + REQUIRE(s.iterations[5].time< double >() == 5.0); + REQUIRE(s.iterations[10].time< double >() == 10.0); + REQUIRE(s.iterations[123456].time< double >() == + static_cast< double >(123456)); } { - Series list{ "../samples/subdir/serial_fileBased_write%08T." + backend, Access::READ_ONLY }; + Series list{ "../samples/subdir/serial_fileBased_write%03T." + backend, Access::READ_ONLY }; helper::listSeries( list ); } } diff --git a/test/python/unittest/API/APITest.py b/test/python/unittest/API/APITest.py index 879d50aef2..5f55f4731c 100644 --- a/test/python/unittest/API/APITest.py +++ b/test/python/unittest/API/APITest.py @@ -578,7 +578,12 @@ def makeDataRoundTrip(self, file_ending): io.Access.create ) - ms = series.iterations[0].meshes + it = series.iterations[0] + + it.time = np.single(1.23) + it.dt = np.longdouble(1.2) + + ms = it.meshes SCALAR = io.Mesh_Record_Component.SCALAR DS = io.Dataset @@ -610,7 +615,15 @@ def makeDataRoundTrip(self, file_ending): io.Access.read_only ) - ms = series.iterations[0].meshes + it = series.iterations[0] + + np.testing.assert_almost_equal(it.time, 1.23) + np.testing.assert_almost_equal(it.dt, 1.2) + # TODO + # self.assertTrue(it.time.dtype == np.dtype('single')) + # self.assertTrue(it.dt.dtype == np.dtype('longdouble')) + + ms = it.meshes o = [1, 2, 3] e = [1, 1, 1]