4.11 Release Notes September, 2021

Feature Enhancements

Items footnoted with 1, below, are supported only on the HDF5 driver.

• Memory Files¹
  • Silo has long had the Core file driver. This has made it possible for Silo clients to bring a whole file into memory in a single I/O request to disk and operate on it there or to create a file in memory and write it in a single I/O request to disk. But, this functionality was limited in that a Silo caller could never manipulate the memory used by the Core file driver.
  • Memory files are an extension of Silo's core file driver that permit Silo callers to pass a memory buffer to Silo to treat like a file in memory and to obtain a buffer from Silo for a file it created in memory.
  • This feature of Silo uses HDF5's file image functionality which was added to HDF5 by a prior LLNL contract with The HDF Group.
  • An example of its use is illustrated in the Silo test code bcastopen.c which implements...

  DBfile *DBOpenByBcast(char const *filename, MPI_Comm comm, int rank_of_root)
  

  • which opens a file on all MPI ranks by opening on the one rank identified as root and then broadcasting it to all other ranks which then turn around and open the memory buffer they recieve.
  • Unfortunately, the current implimentation of memory files in Silo 4.11 does not provide zero-copy functionality which is sort of essential for any Silo file that is expected to be a significant portion of a processor's memory. We expect to remedy this in a patch release, 4.11.1, as soon as various issues in Silo's use of HDF5 and/or the HDF5 library itself essential to providing this functionality are resolved.
• Python 3
  • Thanks to guidance from Cyrus Harrison, Silo's python module now supports both Python 2 and Python 3.
  • In addition, some problem sized memory leaks in the python module were fixed and a number of miscelaneous bug fixes have improved its utility greatly.
• Symbolic Links¹
  • Silo now supports user-defined symbolic links with DBMkSymlink().
  • Silo consumers can operate mostly agnostic to symbolic links links.
  • Symbolic links can even be external to the Silo file and they will mostly work.
  • There are some limited (not fully understood) circumstances where external symbolic links will not work.
• DBMkDirP()
  • A generalization of DBMkDir() was added to create whole directory paths.
  • This is similar in behavior to Unix' mkdir -p.
• Generalized object copying with DBCp()
  • A fully general facsimile of Unix cp command is now available in Silo, DBCp().
  • DBCp() works both within and across silo files.
  • DBCp() is preferred over DBCpDir().
  • This was implemented at the Silo API level so it works on both PDB and HDF5 files.
  • In particular, opening a Silo file with one driver and copying to another driver is an easy way to convert between formats.
• ZFP Compression¹
  • Support for Peter Lindstrom's ZFP compression library was added.
  • This uses the H5Z-ZFP compression filter.
  • Both the ZFP library and H5Z-ZFP filter are built into Silo by default (with appropriate name mangling to avoid collision with the real ZFP and H5Z-ZFP.
  • This can be disabled by --disable-zfp during configuration.
• File level properties
  • Previously, Silo properties such as checksumming and allowing overwrites were global to the whole library.
  • Now, Silo supports file-scope properties which inheret from currently set global properties.
  • There are methods such as DBSetEnableChecksums(int enable) (global) and DBSetEnableChecksumsFile(int enable) (file).
• Performance Improvements¹
  • Adjustments in HDF5 metadata caching and complete disablement of raw data (blocks) caching in HDF5 were added to improve performance.
  • Newer functionality in HDF5 to evict metadata objects when they are closed is now activated in Silo.
• HDF5 lib portability
  • Silo has been tested on HDF5 1.8, 1.10 and 1.12 series of HDF5 libraries.
  • For performance reasons, it is recommended that Silo be used on HDF5-1.12 series whenever practical.
• Numerous functions added to PDB (Lite)
  • A number of missing functions for PDB (Lite) were added to simplify things for PDB callers.
• Transformations for Sparse and Dense material objects
  • Silo's DBmaterial object (sparse) is a rather complicated data structure but also highly optimized to store only non-clean material volume fractions.
  • New material data structure methods were added to Silo to convert between Silo's optimized DBmaterial object and fully dense material data arrays.
  • These are called DBCalcDenseArraysFromMaterial() and DBCalcMaterialFromDenseArrays()
• Upgraded Autoconf and Qt configuration
  • Silo's autoconf artifacts were upgraded to Autoconf version 2.69
  • The Qt configuration logic for silex (ax_have_qt.m4) was upgraded to the latest available, September 18th, 2019
  • In general, if qmake is in your path, the Qt configuration should proceed smoothly even on macOS.
• Partial I/O Queries¹
  • Optimized upport for teasing handfuls of values (often just a single value) from variable(s) in Silo files was added.
  • See DBReadVarVals() documentation for details.
  • The optimization currently works only on the HDF5 driver and is aimed at issuing I/O requests to HDF5 that read only the portions of the dataset actually needed to service the query.

Bug Fixes

• Some problem-sized memory leaks in the Python module were corrected

• Display of string arrays and string data in silex was improved

• FPZIP compilation with gcc >6 was fixed

• Use of HDF5 mpimposix file driver, which prevented compilation with newer HDF5 libraries, was removed.

• Some memory corruption problems in PDB lite were fixed.

• Silo's old redmine issue tracker was shut down. Currently, you can report issues here.

• Silo's python module was enhanc