Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DataFrame WriteAsync always creates nullable data types in schema #589

Open
thomasvdb opened this issue Jan 14, 2025 · 1 comment
Open

Comments

@thomasvdb
Copy link

Issue description

Is there a reason why it's defaulting to nullable data types when writing DataFrame back to a Stream?
See

df.Columns.Select(col => new DataField(col.Name, col.DataType.GetNullable())));

var df = await parquetFileStream.ReadParquetAsDataFrameAsync(cancellationToken: cancellationToken);
var sortedDataFrame = df.OrderBy("ReadingValue");

var newStream = new MemoryStream();
await sortedDataFrame.WriteAsync(newStream, cancellationToken: cancellationToken);

The parquetFileStream variable contains a schema where we have a non-nullable DateTime.
Afterwards the newStream variable contains a schema where we have a nullable DateTime.

And this returns an InvalidDataException when deserializing the newStream variable again with ParquetSerializer.DeserializeAsync

System.IO.InvalidDataException: class definition level (0) does not match file's definition level (1) in field 'ReadingDateTime'. This usually means nullability in class definiton is incompatible.

@thomasvdb
Copy link
Author

Looked furter into it and it seems (correct me if I'm wrong) that DataFrame doesn't keep track if the type was nullable when reading the value.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant