Skip to content

4.11 Release Notes September, 2021

Jump to bottom
Mark (he/his) C. Miller edited this page Sep 11, 2021 · 34 revisions

Items footnoted with 1, below, are supported only on the HDF5 driver.

New API functions

  • New Silo C functions...
    • DBFlush()1 - ensure a file is updated with all previous operations.
    • DBMkDirP() - like unix mkdir -p.
    • DBMkSymlink(), DBGetSymlink(), DBIsSymlink()1 - symbolic links within and across Silo files.
    • DBReadVarVals() - optimized partial I/O for random set(s) of points from a Silo object.
    • DBCp() - fully general copy including recursion within and across files like unix cp -lrs.
    • DBLs() - like unix ls with various filters often more convenient than DBGetToc()
    • DBSet<Some Property>File(), DBGet<Some Property>File() - file scope properties.
    • DBSetAllowLongStrComponents(), DBGetAllowLongStrComponents() - allow long string components of objects.
    • DBIsEmptyMultiXXX() - check multiblock objects for whether they are empty objects.
    • DBGetVarByteLengthFile()1 - determine size of variable in file which can be different than in memory.
    • DBFreeStringArray() - convenience method for Silo's semi-colon separated string lists/arrays.
    • DBCalcDenseArraysFromMaterial(), DBCalcMaterialFromDenseArrays() - convenience methods for working with dense material arrays.
    • DBSprintf() - convenient circular string cash
  • New Silo Fortran functions...
    • DBINQBLEN() - same as DBGetVarByteLength()
    • DBINQDTYP() - same as DBGetVarType() - datatypes
    • DBINQVTYP() - same as DBInqVarType() - Silo object types
    • DBINQMTYP() - same as DBInqMeshType()
    • DBERRLVL() - same as DBErrLvl()
  • New PDB (Lite) functions. At requests from various users, these PDB methods were all copied from PACT/PDB into Silo/PDB and Lite-ified...
    • SC_isfile()
    • PD_set_buffer_size()
    • PD_get_error()
    • PD_alloc_entry()
    • PD_query_entry()
    • PD_get_entry_info()
    • PD_free_entry_info()
    • PD_rel_entry_info()
    • PD_ln() - attempts to use this to implement DBMkSymlink on PDB driver have yet to pass testing.
    • PD_defstr_alt()
    • PD_identify_version
    • PD_id_file()
    • PD_isfile()

Feature Enhancements

  • Memory Files1
    • Silo has long had the Core file driver. This has made it possible for Silo clients to bring a whole file into memory in a single I/O request to disk and operate on it there or to create a file in memory and write it in a single I/O request to disk. But, this functionality was limited in that a Silo caller could never manipulate the memory used by the Core file driver.
    • Memory files are an extension of Silo's core file driver that permit Silo callers to pass a memory buffer to Silo to treat like a file in memory and to obtain a buffer from Silo for a file it created in memory.
    • This feature of Silo uses HDF5's file image functionality which was added to HDF5 by a prior LLNL contract with The HDF Group.
    • An example of its use is illustrated in the Silo test code bcastopen.c which implements...
    DBfile *DBOpenByBcast(char const *filename, MPI_Comm comm, int rank_of_root)
    • which opens a file on all MPI ranks by opening on the one rank identified as root and then broadcasting it to all other ranks which then turn around and open the memory buffer they recieve.
    • Unfortunately, the current implimentation of memory files in Silo 4.11 does not provide zero-copy functionality which is sort of essential for any Silo file that is expected to be a significant portion of a processor's memory. We expect to remedy this in a patch release, 4.11.1, as soon as various issues in Silo's use of HDF5 and/or the HDF5 library itself essential to providing this functionality are resolved.
  • Python 3
    • Thanks to guidance from Cyrus Harrison, Silo's python module now supports both Python 2 and Python 3.
    • In addition, some problem sized memory leaks in the python module were fixed and a number of miscelaneous bug fixes have improved its utility greatly.
  • Symbolic Links1
    • Silo now supports user-defined symbolic links with DBMkSymlink().
    • Silo consumers can operate mostly agnostic to symbolic links links.
    • Symbolic links can even be external to the Silo file and they will mostly work.
    • There are some limited (not fully understood) circumstances where external symbolic links will not work.
  • Generalized object copying with DBCp()
    • A fully general facsimile of Unix cp command is now available in Silo, DBCp().
    • DBCp() works both within and across silo files.
    • DBCp() is preferred over DBCpDir().
    • This was implemented at the Silo API level so it works on both PDB and HDF5 files.
    • In particular, opening a Silo file with one driver and copying to another driver is an easy way to convert between formats.
  • File level properties
    • Previously, Silo properties such as checksumming and allowing overwrites were global to the whole library.
    • Now, Silo supports file-scope properties which inheret from currently set global properties.
    • There are methods such as DBSetEnableChecksums(int enable) (global) and DBSetEnableChecksumsFile(int enable) (file).
  • ZFP Compression1
    • Support for Peter Lindstrom's ZFP compression library was added.
    • To use it, see the documentation on DBSetCompression() or DBSetCompressionFile()
    • This uses the H5Z-ZFP compression filter.
    • Both the ZFP library and H5Z-ZFP filter are built into Silo by default (with appropriate name mangling to avoid collision with the real ZFP and H5Z-ZFP.
    • This can be disabled by --disable-zfp during configuration.
  • Performance Improvements1
    • Adjustments in HDF5 metadata caching and complete disablement of raw data (blocks) caching in HDF5 were added to improve performance.
    • Newer functionality in HDF5 to evict metadata objects when they are closed is now activated in Silo.
  • HDF5 lib portability
    • Silo has been tested on HDF5 1.8, 1.10 and 1.12 series of HDF5 libraries.
    • For performance reasons, it is recommended that Silo be used on HDF5-1.12 series whenever practical.
  • Transformations for Sparse and Dense material objects
    • Silo's DBmaterial object (sparse) is a rather complicated data structure but also highly optimized to store only non-clean material volume fractions.
    • New material data structure methods were added to Silo to convert between Silo's optimized DBmaterial object and fully dense material data arrays.
    • See documentation on DBCalcDenseArraysFromMaterial() and DBCalcMaterialFromDenseArrays()
  • Upgraded Autoconf and Qt configuration
    • Silo's autoconf artifacts were upgraded to Autoconf version 2.69
    • The Qt configuration logic for silex (ax_have_qt.m4) was upgraded to the latest available, September 18th, 2019
    • In general, if qmake is in your path, the Qt configuration should proceed smoothly even on macOS.
  • Partial I/O Queries1
    • Optimized support for teasing handfuls of values (often just a single value) from variable(s) in Silo files was added.
    • See DBReadVarVals() documentation for details.
    • The optimization currently works only on the HDF5 driver and is aimed at issuing I/O requests to HDF5 that read only the portions of the dataset actually needed to service the query.
  • Embedded strings in nameschemes can themselves be other nameschemes

Bug Fixes

  • Some problem-sized memory leaks in the Python module were corrected.
  • Display of string arrays and string data in silex was improved.
  • FPZIP compilation with symbol names DOMAIN and RANGE for gcc >6 was fixed.
  • Use of HDF5 mpimposix file driver, which prevented compilation with newer HDF5 libraries, was removed.
  • Some memory corruption problems in PDB lite were fixed.
  • Display of some multi-level arrays in Silo's browser was corrected.
  • Sub-objects were not being properly returned in DBGetJsonObject().
  • Leaks in objects with region_pnames member were fixed.
  • Default hash table size in PDB lite was changed to extra-large (521021 entries). This action probably should have been taken years ago.
  • Error checking and messaging in Python module was improved substantially.
  • Multiple issues with writing objects from Python module were corrected.
  • dir() and help() in Python module now return something useful in both Python 2 and 3.
  • Objects missing from table-of-contents returned in Python were added.
  • Constants missing from Silo's python module were added.
  • Missing support for long long type data was added to Silex.

Caveats

  • This release does not include updates for Windows.
  • Silo's old redmine issue tracker was shut down. Currently, you can report issues here. We plan to move the Silo code base from a Subversion repo at LLNL to GitHub and when that happens, the Silo project along with these issues will be moved and housed within the LLNL organization on GitHub.
  • Since the 4.10.2 release there have been a number of point releases for specific users encountering specific issues on LLNL systems. These were variously versioned 4.10.X and when X=3 perhaps with a -post%d appended. This new version, 4.11, includes the work that went into those various point releases and should be sufficient to replace any/all of them. Nonetheless, 4.11 is deemed NOT ABI compatible with any of those point releases.
  • Installations for LLNL users on RZ/CZ are found in /usr/gapps/silo/4.11/<$SYSTYPE> for clang, gcc, xlc (blueos) and icc (toss3) compilers. These new 4.11 installs will be installed with HDF5-1.8.16 for reasons of compatability. However, installations for HDF5-1.12 will also be made available and are likely preferred due to improved time and space performance. Due to quota limitations, old 4.6 and 4.7 silo installations had to be moved. The Silo.so Python 2 and 3 modules are in libso and libso3 dirs respectively.
  • Installations on SCF will be forthcomming before end of September.
  • Due to challenges in obtaining a (local) FrameMaker licenese, updates to the documentation will be delayed also to end of September. However, the new chapter on the Python interface is temporarily available here and, for those interested, the experimental Json interface here
  • We anticipate the need for a 4.11.1 release before end of CY21 to address, minimally, zero-copy memory files as well as other aforementioned limitations. We will ensure any such release will be binary compatible with 4.11.
  • For reasons described above, the current link for LLNL-managed installations will remain at 4.10.2.
  • The 4.11 standard and bsd release tarballs are in process of being added to the Silo downloads page. In the meantime, LLNL users can find the release tarballs in /usr/gapps/silo/4.11. In addition, those same tarballs are temporarily available here
Clone this wiki locally