Improve Zserio Encoding Guide for packed arrays

[mkutscherauer]

Trying to create my own extension for the zserio generator and i'm struggling a bit with packed arrays.

According to the Zserio Language Overview following types can be packed:

• all integer types (uint32, varuint, ...)
• enumeration types
• bitmask types
• compound types which contain integer types, enumeration, bitmask types or inner packable arrays
• all union types (even if do not contain any packable fields because selector is always packable integer type)

The first 3 bulletpoints more or less correspond to integer values.

For compound types i am not quite clear on their working:

• How are conditionals handled ?
• Are inner packable arrays just inserted into the bytestream as is or is additional packing applied, e.g. packing maxBitNumber ?
• How are inner non packed arrays handled, are there differences between fixed/const length and variable length. How are auto arrays handled?
• It is implied that compound types which contain types beside integer types, enumeration, bitmask types or inner packable arrays are not packable, but the encoding guide uses examples with string and non-packable types as being inlined without PackingDescriptor

How are unions handled? They are always considered packable because of their tag, but what does this mean for the actual value, e.g. array[0] contains union with field1 = 1, and array[1] contains union with field2 = 5?

I found this sentence but an example would be really useful (also mentions choices, which are not listed as packable in the language overview)

Packed arrays of choices and unions are encoding in the same way, meaning that all fields corresponded to the same case are considered as an array. If such array is empty, nothing is encoded in the bit stream.

[fklebert]

Trying to create my own extension for the zserio generator

May I ask what language you are targetting? Just out of curiousity 😃

[mkutscherauer]

I'm toying around with Rust

[mkutscherauer]

Further findings (don't wanna open new issues for every new thing i encounter)

• Function documentation could be more explicit, especially considering that they are allowed to use type parameters
• Implicit length arrays in the language overview the code sample implies that they support compound types (Element), but the implementation of the Java runtime explicitly throws when encountering Compound types

[fklebert]

I'm toying around with Rust

Then you might want to check out https://github.com/Danaozhong/rust-zserio
Maybe you guys can join forces 💪

[mikir]

Packing of compound types is unfortunatelly pretty complicated. Just documenting this is a time-consuming task in itself.

How are conditionals handled ?

You can imagine this that some temporary array with all presented values are created, then this array is delta compressed and then the compressed elements are used in the corresponded place of the original array.

Example:

struct Element
{
    optional uint32 value;
};

struct PackedArray
{
    packed Element array[];
};

Let's suppose that the array has three elements: array = { {10}, {/* optional is not there */}, {11} }.
Then, the temporary array which will be packed seems like Temp = { {10}, {11} }.
And the packed array will be packedArray = { {Packing Descriptor, 10}, {/* optional is not there */}, {1 /* delta */} }. For simplicity, the optional bool flag is not shown in this example.

Are inner packable arrays just inserted into the bytestream as is or is additional packing applied, e.g. packing maxBitNumber ?

Inner packable arrays are automatically packed as well.

How are inner non packed arrays handled, are there differences between fixed/const length and variable length. How are auto arrays handled?

Inner packed arrays are handled in the same way as outer arrays. There is no difference between fixed, variable length and auto arrays.

It is implied that compound types which contain types beside integer types, enumeration, bitmask types or inner packable arrays are not packable, but the encoding guide uses examples with string and non-packable types as being inlined without PackingDescriptor

Not sure if I undestand this correctly but if compound type is not packable and such compound type is used as an element in the packed array, then the packing is completely ignored and the array will be encoded as an unpacked array. In other words, the packed keyword is ignored, and a corresponding warning is fired.

How are unions handled?

Unions and choices are handled in the same way than optionals. For choices it means that all fields corresponded to the same case are considered as an temporary arrays. For unions, it means that all fields are considered as an temporary arrays.

Example:

union Element
{
    uint16 value16;
    uint32 value32;
};

struct PackedArray
{
    packed Element array[];
};

Let's suppose that the array has four elements: array = { {value16: 10}, {value32: 100}, {value16:11}, {value32: 111 }.
Then, there will be two temporary arrays which will be packed: TempValue16 = { {10}, {11} } and TempValue32 = { {100}, {111} }.
Then the packed array will be packedArray = { {value16: Packing Descriptor, 10}, {value32: Packing Descriptor, 100}, {value16: 1 /* delta */}, {value32: 1 /* delta */} }.
For simplicity, the packing of the tag is not shown in this example.

I agree that we should improve Zserio Encoding Guide but because of our capacity we are not able to do it quickly. If my brief explanation does not help and if this is a blocker for you, then I would suggest making a quick call.

[mikir]

Further findings (don't wanna open new issues for every new thing i encounter)

• Function documentation could be more explicit, especially considering that they are allowed to use type parameters

• Implicit length arrays in the language overview the code sample implies that they support compound types (Element), but the implementation of the Java runtime explicitly throws when encountering Compound types

Thank you very much for this report. You are right, we will improve the Zserio Language Overview document. New issue #679 has been created because of that.

[mkutscherauer]

I agree that we should improve Zserio Encoding Guide but because of our capacity we are not able to do it quickly. If my brief explanation does not help and if this is a blocker for you, then I would suggest making a quick call.

None of this is a blocker, this is merely a private toy project and if absolutely necessary i can figure this out by consulting the existing Extensions. Nevertheless explicit documentation always is preferable.

Not sure if I undestand this correctly but if compound type is not packable and such compound type is used as an element in the packed array, then the packing is completely ignored and the array will be encoded as an unpacked array. In other words, the packed keyword is ignored, and a corresponding warning is fired.

So the Language Overview states the following:

An array type can be packed indicated by a packed keyword. The only supported packing algorithm is delta compression which is applied for packed arrays of:

• ...
• compound types which contain integer types, enumeration, bitmask types or inner packable arrays
  all union types (even if do not contain any packable fields because selector is always packable integer type)

On the other hand the Encoding Guide states:

...
If the compound field is unpackable, it's value is simply written in the stream.

So to me this reads that both documents disagree on the handling of below code.
In my reading (and maybe this is too strict) the language overview would suggest that PackableStructure is not packable because it contains types which are not packable.
On other hand the Encoding Guide makes it clear that packing for compound types is applied field wise and not for the entire compound type. So in a sense talking about a compound type being packable or not is a misnomer because being packable applies to its parts but not to the whole.

struct PackableStructure
{
    uint32 value; // packable
    string text;  // unpackable
};

struct PackedArray
{
    packed PackableStructure list[5];
};

[mikir]

OK, I see your point. We will extend Language Overview to explicitly state that PackableStructure is packable if at least one its field is packable. Or something similar.