Better way of writing data to disk?

[ali-ramadhan]

Right now FieldWriter writes field.data using write(filepath, array) so you lose the size of the array (and the field type) when writing. This is probably inevitable here.

A better solution would be to write output as NetCDF.

[ali-ramadhan]

Decision was made: NetCDF first for now.

[ali-ramadhan]

There are two packages providing high-level interfaces for reading/writing NetCDF files. I went with NetCDF.jl as it seemed older and more mature maybe, but would be good to keep track of NCDatasets.jl as it seems to use data frames instead of just arrays.

[ali-ramadhan]

NetCDF.jl (so does NCDatasets.jl) has some heavy dependencies like Conda, CMake, and HDF5 which must be built. Don't think there's any way around this, NetCDF output is a must.

[ali-ramadhan]

NetCDF.jl seems to be missing some features and isn't really being maintained (See JuliaGeo/NetCDF.jl#62 about saving time values and JuliaGeo/NetCDF.jl#39). Maybe it's worth switching to NCDatasets.jl which takes a more data frames approach to NetCDF and is actively maintained and grew out of bugs that weren't being fixed in NetCDF.jl. Unfortunately we're choosing between two relatively young packages. An alternative would be to use the much more mature netcdf4-python but I'd rather not have to use PyCall...

[glwagner]

We may have to contribute to the development of whatever NetCDF package we choose to use. Let’s pick the project we’d most like to contribute to.

The developer of NCDatasets is an oceanographer. That’s a plus.

[glwagner]

By “dataframes” approach, do you mean the dictionary-like interface?

[ali-ramadhan]

Yeah sorry I thought it used DataFrames.jl or something but yeah it just uses Dicts for the attributes.

Might be good to see what CliMA.jl is thinking of using so we don't end up building towards two different solutions.

[ali-ramadhan]

Output needs to be arbitrary. We may need to perform on-line analysis and output the result (example: turbulent dissipation rate, time-averages, slices of fields, point values, etc).

We should design an additional interface for Fields. The type of the field indicates the coordinates on which the field is defined, so we should design an interface that uses that information.

Originally posted by @glwagner in https://github.com/ali-ramadhan/Oceananigans.jl/pull/93#issuecomment-468290310

Just adding your comment here as I think there are two new questions raised:

1. How to integrate diagnostics with the output writing framework?
2. Right now each NetCDF output file shows a single snapshot. Maybe it makes more sense to keep appending to an existing NetCDF file. This might also make addressing (1) easier especially if the diagnostics have a different output frequency that other fields.

[ali-ramadhan]

Closing this as I feel like we resolved it. The original issue was that we were writing binary output and now we're writing NetCDF output.

I will open a new issue discussing our needs for better NetCDF output.