Silo_File_Version_Management_and_Backward_Compatability

Silo File Version Management and Backward Compatability

Can a file produced with a newer version of the Silo library be read by an older version of the library? This question gets at the heart of what it means for Silo to maintain backward compatability in its file format.

This question is both similar to but also entirely different from the issue of API (Application Programming Interface) and ABI (Application Binary Interface) version compatability. API and ABI backward compatability have to do with compile-time, link-time and run-time behavior of the interface between a library and its clients. File format compatability has to do with compatability in presistent data produced by a library and its clients.

As new releases of the Silo library are made, the changes typically fall into two broad categories; additions and changes. Additions involve the introduction of new objects, functions and/or options. Changes involve adjustments to existing objects, functions and/or options.

Additions

For the most part, additions in Silo have historically been handled in such a way that backward compatability is never threatened. At the same time, it is also understood that there can be no expectation that an older version of the library be able to handle those objects in a file that represent additions. So, where additions are concerned, backward compatability means primarily that any additions that might exist in the file don’t get in the way of the library’s ability to deal with everything else in the file. While there may be some minimal ability of an older library to handle additions, this is not even required for backward compatability. For example, suppose between releases A and B of Silo, a new object type, Foo, is added. The existance of a Foo object in a file produced with releasae B of Silo will not prevent a workflow using release A from being able to read everything but objects of type Foo.

Some Mathematical Rigor

Characterizing backward compatability can become complicated. It involves combinations of changes in applications using Silo, the Silo library itself and any TPLs (third party libraries) Silo may use (e.g. HDF5). It also depends on what objects and features a given application is using either directly in Silo or indirectly through one of Silo’s TPLs. Finally, backward compatability may also hinge on whether applications are producers, consumers or both of Silo data.

Here, we’ll try to use the simplest scenario in which all of the key issues can be explained.

• So, suppose there are two applications, X and U and later versions of these same applications, Y and V, respectively.
• Also, suppose there are two versions of the Silo library, A and later version B.
• Applications X and U, both using release A of Silo, routinely exchange data. We denote this as X/A <→ U/A.
• Next, consider all possible objects, functions and options in a release, say A of Silo. Denote this as A(*)= { a in A | a is an object, function or option }.
• Lets suppose applicaton X uses a subset of these, denoted by A(X)= { a in A | X uses a }. Likewise, for application U, we have A(U)= { a in A | U uses a }.
• Going from release A to release B of Silo involved both additions and changes, and some changes are not backward compatable (maybe correct mathematical term is invertible?).
  • The additions imply that B(*) contains A(*).
  • The changes imply that there exists objects, functions or options in B that are != same in A.
• Now, what happens when application X upgrades to the newer Silo? Suppose that release B of Silo is different from A in that it includes both additions and changes.

• Does X/A <→ X/B work?
  • It should. Its the same application, X, just using (compiled and linked with) a newer version of Silo. However, if release B of Silo involves changes to existing objects, its possible those changes could prevent release A from properly reading them. Whenever possible (and this has been true throughout 99% of Silo’s history), we try to avoid this situation.
• Does X/B <→ U/A work?
  • It should work as well as X/A <→ U/A did.