Investigate Ways to Extend or Replace Serialization Versioning

[ilexp]

Summary

Duality currently allows to handle simple serialization errors like renamed types or members, or type conversions using custom SerializeErrorHandler implementations. While nice, what is lacking is a more fundamental way to version serialized data, which is supported by the serialization system itself.

Analysis

The basic idea of an extended approach to handling serialization between different type versions is to serialize a version number along with each type and, when detecting a mismatch, look for classes implementing a handler interface across all plugins that report to be able to handle that specific type and version change.

Type Identification

• Renaming a type or moving it to a different namespace is a reasonably common operation that should not require a full-fledged handler implementation.
• Instead, adopt a declarative, attribute-based approach similar to FormerlySerializedAs that Unity provides for fields that have been renamed - just for type names instead of fields.

Type Versioning

• Due to the plugin-based nature of Duality, versioning needs to be fine-grained down to a type level. One version number per file is not going to be enough.
• Serializer implementations could create a registry of serialized versions once per continuous serialization operation where each encountered type is mapped to a version.
• Versioning could be based on the types assembly version number, aligning with semantic versioning release models and implying a deliberate change from version A to version B that can be addressed in a handler when needed.
• For developers who operate between versions, it might also be useful to be able to temporarily assign a specific version number to a type (that differs from the assembly version) via attribute.
• Issue: The vast majority of assembly version changes do not entail breaking serialization changes. They should be treated accordingly.
  • Instead of assuming that a version change is breaking by default, assume they are non-breaking by default.
  • A breaking change needs to be stated explicitly to trigger the version mismatch handling - for example by allowing to state a "minimum serialized version" for a type when a breaking change is introduced, serialization-wise.
  • Could be stated via attribute.
• Issue: Cannot handle data from types that no longer exist, i.e. failing to resolve a type is a full data loss of the entire object tree below the unresolved object.
  • Manual workaround: When a type is removed entirely, provide a private / nested data-only replacement that is used as a temporary container.
  • Automatic workaround: ?

Version Mismatch Handling

• A handler interface / type group could use a ISerializeVersionMismatchHandler terminology. Open for suggestions if you can come up with something shorter.
• Exact API will have to be prototyped alongside a concrete example.
• When a serializer encounters an old version of a type and identified a mismatch handler implementation able to handle that, it would read all of the objects fields, store them as a list of key-value-pairs and then invoke the mismatch handler API in a similar way that ISerializeExplicit works: By handing over an IDataReader and letting the handler create and populate an object from that.

Serialization Versioning Plugins

• Ideally, it should be possible (but by no means required!) to group all information related to serialization compatibility (FormerlySerializedAs, MinSerializationVersion attributes, ISerializeVersionMismatchHandler implementations, other stuff?) in a distinct plugin, so Duality and other libraries could ship with a "compatibility" plugin and keeping their core free of legacy clutter.
• Handlers are separate by default, the attributes could come in two versions: One attached to a type directly, another on an assembly-level.
  • This info should be cached in SerializeType for perf reasons.

Backwards Compatibility

• It should be possible to implement this in a 100% backwards compatible way, from a serialized data point of view.
• However, since the old SerializeErrorHandlers will likely no longer exist, this would still be a breaking change for all plugins relying on them.
• It might be possible to implement this in v2.x without removing any of the old stuff and only removing it in v3.0 though. Could be worth the effort.