You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
var df = await parquetFileStream.ReadParquetAsDataFrameAsync(cancellationToken: cancellationToken);
var sortedDataFrame = df.OrderBy("ReadingValue");
var newStream = new MemoryStream();
await sortedDataFrame.WriteAsync(newStream, cancellationToken: cancellationToken);
The parquetFileStream variable contains a schema where we have a non-nullable DateTime.
Afterwards the newStream variable contains a schema where we have a nullable DateTime.
And this returns an InvalidDataException when deserializing the newStream variable again with ParquetSerializer.DeserializeAsync
System.IO.InvalidDataException: class definition level (0) does not match file's definition level (1) in field 'ReadingDateTime'. This usually means nullability in class definiton is incompatible.
The text was updated successfully, but these errors were encountered:
Issue description
Is there a reason why it's defaulting to nullable data types when writing DataFrame back to a Stream?
See
parquet-dotnet/src/Parquet.Data.Analysis/Extensions.cs
Line 65 in 1846cc7
The
parquetFileStream
variable contains a schema where we have a non-nullable DateTime.Afterwards the
newStream
variable contains a schema where we have a nullable DateTime.And this returns an
InvalidDataException
when deserializing thenewStream
variable again withParquetSerializer.DeserializeAsync
System.IO.InvalidDataException: class definition level (0) does not match file's definition level (1) in field 'ReadingDateTime'. This usually means nullability in class definiton is incompatible.
The text was updated successfully, but these errors were encountered: