diff --git a/dev/assets/example_SST_CMEMS.png b/dev/assets/example_SST_CMEMS.png index 3bdc4e6c..8e0b0c49 100644 Binary files a/dev/assets/example_SST_CMEMS.png and b/dev/assets/example_SST_CMEMS.png differ diff --git a/dev/attributes/index.html b/dev/attributes/index.html index 77a5ea25..c57e3816 100644 --- a/dev/attributes/index.html +++ b/dev/attributes/index.html @@ -1,10 +1,9 @@ -Attributes · NCDatasets.jl
Edit on GitHub

Attributes

The NetCDF dataset (as returned by NCDataset or NetCDF groups) and the NetCDF variables (as returned by getindex, variable or defVar) have the field attrib which has the type NCDatasets.Attributes and behaves like a julia dictionary.

Base.getindex — Method
getindex(a::Attributes,name::SymbolOrString)

Return the value of the attribute called name from the attribute list a. Generally the attributes are loaded by indexing, for example:

ds = NCDataset("file.nc")
-title = ds.attrib["title"]
source
Base.setindex! — Method
Base.setindex!(a::Attributes,data,name::SymbolOrString)

Set the attribute called name to the value data in the attribute list a. data can be a vector or a scalar. A scalar is handeld as a vector with one element in the NetCDF data model.

Generally the attributes are defined by indexing, for example:

ds = NCDataset("file.nc","c")
+Attributes · NCDatasets.jl
Edit on GitHub
Attributes
The NetCDF dataset (as returned by NCDataset or NetCDF groups) and the NetCDF variables (as returned by getindex, variable or defVar) have the field attrib which has the type NCDatasets.Attributes and behaves like a julia dictionary.
Base.getindex — Method
getindex(a::Attributes,name::SymbolOrString)
Return the value of the attribute called name from the attribute list a. Generally the attributes are loaded by indexing, for example:
using NCDatasets
+ds = NCDataset("file.nc")
+title = ds.attrib["title"]
Base.setindex! — Method
Base.setindex!(a::Attributes,data,name::SymbolOrString)
Set the attribute called name to the value data in the attribute list a. data can be a vector or a scalar. A scalar is handeld as a vector with one element in the NetCDF data model.
Generally the attributes are defined by indexing, for example:
ds = NCDataset("file.nc","c")
 ds.attrib["title"] = "my title"
-close(ds)
If data is a string, then attribute is saved as a list of NetCDF characters (NC_CHAR) with the appropriate length. To save the attribute as a string (NC_STRING) you can use the following:
ds = NCDataset("file.nc","c")
-ds.attrib["title"] = ["my title"]
-close(ds)
source
Base.keys — Method
Base.keys(a::Attributes)
Return a list of the names of all attributes.
source
Base.delete! — Method
Base.delete!(a::Attributes, name)
Delete the attribute name from the attribute list a.
source
Loading all attributes as a Dict can be achieved by passing ds.attrib (where ds is the NCDataset) as argument to Dict.
using NCDatasets
+close(ds)
Base.keys — Method
Base.keys(a::Attributes)

Return a list of the names of all attributes.

Base.delete! — Method
Base.delete!(a::Attributes, name)

Delete the attribute name from the attribute list a.

Loading all attributes as a Dict can be achieved by passing ds.attrib (where ds is the NCDataset) as argument to Dict.

using NCDatasets
 ncfile = download("https://www.unidata.ucar.edu/software/netcdf/examples/sresa1b_ncar_ccsm3-example.nc");
 ds = NCDataset(ncfile);
 attributes_as_dictionary = Dict(ds.attrib)
@@ -29,4 +28,4 @@
 ))

Note that Dict does not perserve the order of the attributes. Therefore an OrderedDict from the package DataStructures is preferable.

Or one could use simply the fillvalue parameter of defVar.

ncv1 = defVar(ds,"v1", UInt8, ("longitude", "latitude", "time"), fillvalue = UInt8(255), attrib = [
     "add_offset"                => -1.0,
     "scale_factor"              => 5.0,
-])
+]) diff --git a/dev/dataset/index.html b/dev/dataset/index.html index 868f8366..9e7aeb1e 100644 --- a/dev/dataset/index.html +++ b/dev/dataset/index.html @@ -14,7 +14,7 @@ resp = HTTP.get("https://www.unidata.ucar.edu/software/netcdf/examples/ECMWF_ERA-40_subset.nc") ds = NCDataset("some_string","r",memory = resp.body) total_precipitation = ds["tp"][:,:,:] -close(ds)

Dataset is an alias of NCDataset.

source
mfds = NCDataset(fnames, mode = "r"; aggdim = nothing, deferopen = true,
+close(ds)

Dataset is an alias of NCDataset.

source
mfds = NCDataset(fnames, mode = "r"; aggdim = nothing, deferopen = true,
               isnewdim = false,
               constvars = [])

Opens a multi-file dataset in read-only "r" or append mode "a". fnames is a vector of file names.

Variables are aggregated over the first unlimited dimension or over the dimension aggdim if specified. Variables without the dimensions aggdim are not aggregated. All variables containing the dimension aggdim are aggregated. The variable who do not contain the dimension aggdim are assumed constant.

If variables should be aggregated over a new dimension (not present in the NetCDF file), one should set isnewdim to true. All NetCDF files should have the same variables, attributes and groupes. Per default, all variables will have an additional dimension unless they are marked as constant using the constvars parameter.

The append mode is only implemented when deferopen is false. If deferopen is false, all files are opened at the same time. However the operating system might limit the number of open files. In Linux, the limit can be controled with the command ulimit.

All metadata (attributes and dimension length are assumed to be the same for all NetCDF files. Otherwise reading the attribute of a multi-file dataset would be ambiguous. An exception to this rule is the length of the dimension over which the data is aggregated. This aggregation dimension can varify from file to file.

Setting the experimental flag _aggdimconstant to true means that the length of the aggregation dimension is constant. This speeds up the creating of a multi-file dataset as only the metadata of the first file has to be loaded.

Examples:

You can use Glob.jl to make fnames from a file pattern, e.g.

using NCDatasets, Glob
 ds = NCDataset(glob("ERA5_monthly3D_reanalysis_*.nc"))

Aggregation over a new dimension:

using NCDatasets
@@ -27,7 +27,7 @@
 ds = NCDataset(["foo$i.nc" for i = 1:3],aggdim = "sample", isnewdim = true)
 size(ds["data"])
 # output
-# (4, 3)
source

Useful functions that operate on datasets are:

Base.keys — Method
keys(ds::NCDataset)

Return a list of all variables names in NCDataset ds.

source
Base.haskey — Function
haskey(ds::NCDataset,name)
+# (4, 3)
source

Useful functions that operate on datasets are:

Base.keys — Method
keys(ds::NCDataset)

Return a list of all variables names in NCDataset ds.

source
Base.haskey — Function
haskey(ds::NCDataset,name)
 haskey(d::Dimensions,name)
 haskey(ds::Attributes,name)

Return true if the NCDataset ds (or dimension/attribute list) has a variable (dimension/attribute) with the name name. For example:

ds = NCDataset("/tmp/test.nc","r")
 if haskey(ds,"temperature")
@@ -36,7 +36,7 @@
 
 if haskey(ds.dim,"lon")
     println("The file has a dimension 'lon'")
-end

This example checks if the file /tmp/test.nc has a variable with the name temperature and a dimension with the name lon.

source
Base.haskey(a::Attributes,name)

Check if name is an attribute

source
Base.getindex — Method
v = getindex(ds::NCDataset, varname::AbstractString)

Return the NetCDF variable varname in the dataset ds as a NCDataset.CFVariable. The following CF convention are honored when the variable is indexed:

  • _FillValue or missing_value (which can be a list) will be returned as missing. NCDatasets does not use implicitely the default NetCDF fill values when reading data.
  • scale_factor and add_offset are applied (output = scale_factor * data_in_file + add_offset)
  • time variables (recognized by the units attribute and possibly the calendar attribute) are returned usually as DateTime object. Note that CFTime.DateTimeAllLeap, CFTime.DateTimeNoLeap and CF.TimeDateTime360Day cannot be converted to the proleptic gregorian calendar used in julia and are returned as such. (See CFTime.jl for more information about those date types.) If a calendar is defined but not among the ones specified in the CF convention, then the data in the NetCDF file is not converted into a date structure.

A call getindex(ds, varname) is usually written as ds[varname].

If variable represents a cell boundary, the attributes calendar and units of the related NetCDF variables are used, if they are not specified. For example:

dimensions:
+end

This example checks if the file /tmp/test.nc has a variable with the name temperature and a dimension with the name lon.

source
Base.haskey(a::Attributes,name::SymbolOrString)

Check if name is an attribute

source
Base.getindex — Method
v = getindex(ds::NCDataset, varname::AbstractString)

Return the variable varname in the dataset ds as a CFVariable. The following CF convention are honored when the variable is indexed:

  • _FillValue or missing_value (which can be a list) will be returned as missing.
  • scale_factor and add_offset are applied (output = scale_factor * data_in_file + add_offset)
  • time variables (recognized by the units attribute and possibly the calendar attribute) are returned usually as DateTime object. Note that CFTime.DateTimeAllLeap, CFTime.DateTimeNoLeap and CF.TimeDateTime360Day cannot be converted to the proleptic gregorian calendar used in julia and are returned as such. (See CFTime.jl for more information about those date types.) If a calendar is defined but not among the ones specified in the CF convention, then the data in the file is not converted into a date structure.

A call getindex(ds, varname) is usually written as ds[varname].

If variable represents a cell boundary, the attributes calendar and units of the related variables are used, if they are not specified. For example:

dimensions:
   time = UNLIMITED; // (5 currently)
   nv = 2;
 variables:
@@ -44,7 +44,7 @@
     time:long_name = "time";
     time:units = "hours since 1998-04-019 06:00:00";
     time:bounds = "time_bnds";
-  double time_bnds(time,nv);

In this case, the variable time_bnds uses the units and calendar of time because both variables are related thought the bounds attribute following the CF conventions.

See also cfvariable(ds, varname).

source
CommonDataModel.variable — Function
v = variable(ds::NCDataset,varname::String)

Return the NetCDF variable varname in the dataset ds as a NCDataset.Variable. No scaling or other transformations are applied when the variable v is indexed.

source
CommonDataModel.variable(ds::AbstractDataset,variablename::SymbolOrString)

Return the variable with the name variablename from the data set ds.

CommonDataModel.cfvariable — Function
v = cfvariable(ds::NCDataset,varname::SymbolOrString; <attrib> = <value>)

Return the variable varname in the dataset ds as a NCDataset.CFVariable. The keyword argument <attrib> are the attributes (fillvalue, missing_value, scale_factor, add_offset, units and calendar) relevant to the CF conventions. By specifing the value of these attributes, the one can override the value specified in the data set. If the attribute is set to nothing, then the attribute is not loaded and the corresponding transformation is ignored. This function is similar to ds[varname] with the additional flexibility that some variable attributes can be overridden.

Example:

NCDataset("foo.nc","c") do ds
+  double time_bnds(time,nv);

In this case, the variable time_bnds uses the units and calendar of time because both variables are related thought the bounds attribute following the CF conventions.

See also cfvariable(ds, varname).

CommonDataModel.variable — Function
v = variable(ds::NCDataset,varname::String)

Return the NetCDF variable varname in the dataset ds as a NCDataset.Variable. No scaling or other transformations are applied when the variable v is indexed.

source
CommonDataModel.variable(ds::AbstractDataset,variablename::SymbolOrString)

Return the variable with the name variablename from the data set ds.

CommonDataModel.cfvariable — Function
v = cfvariable(ds::NCDataset,varname::SymbolOrString; <attrib> = <value>)

Return the variable varname in the dataset ds as a NCDataset.CFVariable. The keyword argument <attrib> are the attributes (fillvalue, missing_value, scale_factor, add_offset, units and calendar) relevant to the CF conventions. By specifing the value of these attributes, the one can override the value specified in the data set. If the attribute is set to nothing, then the attribute is not loaded and the corresponding transformation is ignored. This function is similar to ds[varname] with the additional flexibility that some variable attributes can be overridden.

Example:

NCDataset("foo.nc","c") do ds
   defVar(ds,"data",[10., 11., 12., 13.], ("time",), attrib = Dict(
       "add_offset" => 10.,
       "scale_factor" => 0.2))
@@ -77,16 +77,11 @@
 @show cfvariable(ds,"data", units = "days since 2000-01-01")[:]
 # returns [DateTime(2000,1,11), DateTime(2000,1,12), DateTime(2000,1,13), DateTime(2000,1,14)]
 
-close(ds)
NCDatasets.sync — Function
sync(ds::NCDataset)

Write all changes in NCDataset ds to the disk.

source
Base.close — Function
close(ds::NCDataset)

Close the NCDataset ds. All pending changes will be written to the disk.

source
CommonDataModel.path — Function
path(ds::NCDataset)

Return the file path (or the opendap URL) of the NCDataset ds

source
CommonDatamodel.path(ds::AbstractDataset)

File path of the data set ds.

NCDatasets.ncgen — Function
ncgen(fname; ...)
-ncgen(fname,jlname; ...)

Generate the Julia code that would produce a NetCDF file with the same metadata as the NetCDF file fname. The code is placed in the file jlname or printed to the standard output. By default the new NetCDF file is called filename.nc. This can be changed with the optional parameter newfname.

source
NCDatasets.varbyattrib — Function
varbyattrib(ds, attname = attval)

Returns a list of variable(s) which has the attribute attname matching the value attval in the dataset ds. The list is empty if the none of the variables has the match. The output is a list of CFVariables.

Examples

Load all the data of the first variable with standard name "longitude" from the NetCDF file results.nc.

julia> ds = NCDataset("results.nc", "r");
-julia> data = varbyattrib(ds, standard_name = "longitude")[1][:]
source
Base.write — Function
write(dest_filename::AbstractString, src::AbstractNCDataset; include = keys(src), exclude = [], idimensions = Dict())
-write(dest::NCDataset, src::AbstractNCDataset; include = keys(src), exclude = [], idimensions = Dict())

Write the variables of src dataset into an empty dest dataset (which must be opened in mode "a" or "c"). The keywords include and exclude configure which variable of src should be included (by default all), or which should be excluded (by default none).

If the first argument is a file name, then the dataset is open in create mode ("c").

This function is useful when you want to save the dataset from a multi-file dataset.

To save a subset, one can use the view function view to virtually slice a dataset:

Example

NCDataset(fname_src) do ds
+close(ds)
CommonDataModel.sync — Function
sync(ds::NCDataset)

Write all changes in NCDataset ds to the disk.

source
Base.close — Function
close(ds::NCDataset)

Close the NCDataset ds. All pending changes will be written to the disk.

source
CommonDataModel.path — Function
path(ds::NCDataset)

Return the file path (or the opendap URL) of the NCDataset ds

source
CommonDatamodel.path(ds::AbstractDataset)

File path of the data set ds.

NCDatasets.ncgen — Function
ncgen(fname; ...)
+ncgen(fname,jlname; ...)

Generate the Julia code that would produce a NetCDF file with the same metadata as the NetCDF file fname. The code is placed in the file jlname or printed to the standard output. By default the new NetCDF file is called filename.nc. This can be changed with the optional parameter newfname.

source
CommonDataModel.varbyattrib — Function
varbyattrib(ds, attname = attval)

Returns a list of variable(s) which has the attribute attname matching the value attval in the dataset ds. The list is empty if the none of the variables has the match. The output is a list of CFVariables.

Examples

Load all the data of the first variable with standard name "longitude" from the NetCDF file results.nc.

julia> ds = NCDataset("results.nc", "r");
+julia> data = varbyattrib(ds, standard_name = "longitude")[1][:]
Base.write — Function
write(dest::AbstractDataset, src::AbstractDataset; include = keys(src), exclude = [])

Write the variables of src dataset into an empty dest dataset (which must be opened in mode "a" or "c"). The keywords include and exclude configure which variable of src should be included (by default all), or which should be excluded (by default none).

If the first argument is a file name, then the dataset is open in create mode ("c").

This function is useful when you want to save the dataset from a multi-file dataset.

To save a subset, one can use the view function view to virtually slice a dataset:

Example

NCDataset(fname_src) do ds
     write(fname_slice,view(ds, lon = 2:3))
-    # deprecated
-    # write(fname_slice,ds,idimensions = Dict("lon" => 2:3))
-end

All variables in the source file fname_src with a dimension lon will be sliced along the indices 2:3 for the lon dimension. All attributes (and variables without a dimension lon) will be copied over unmodified.

source

Notice that DateTime-structures from CFTime are used to represent time for non-standard calendars. Otherwise, we attempt to use standard structures from the Julia standard library Dates.

Groups

CommonDataModel.defGroup — Function
defGroup(ds::NCDataset,groupname; attrib = []))

Create the group with the name groupname in the dataset ds. attrib is a list of attribute name and attribute value pairs (see NCDataset).

source
group = CommonDatamodel.defGroup(ds::AbstractDataset,name::SymbolOrString)

Create an empty sub-group with the name name in the data set ds. The group is a sub-type of AbstractDataset.

Base.getindex — Method
group = getindex(g::NCDatasets.Groups,groupname::AbstractString)

Return the NetCDF group with the name groupname. For example:

julia> ds = NCDataset("results.nc", "r");
-julia> forecast_group = ds.group["forecast"]
-julia> forecast_temp = forecast_group["temperature"]
source
Base.keys — Method
Base.keys(g::NCDatasets.Groups)

Return the names of all subgroubs of the group g.

source

Common methods

One can iterate over a dataset, attribute list, dimensions and NetCDF groups.

for (varname,var) in ds
+end

All variables in the source file fname_src with a dimension lon will be sliced along the indices 2:3 for the lon dimension. All attributes (and variables without a dimension lon) will be copied over unmodified.

Notice that DateTime-structures from CFTime are used to represent time for non-standard calendars. Otherwise, we attempt to use standard structures from the Julia standard library Dates.

Groups

CommonDataModel.defGroup — Function
defGroup(ds::NCDataset,groupname; attrib = []))

Create the group with the name groupname in the dataset ds. attrib is a list of attribute name and attribute value pairs (see NCDataset).

source
group = CommonDatamodel.defGroup(ds::AbstractDataset,name::SymbolOrString)

Create an empty sub-group with the name name in the data set ds. The group is a sub-type of AbstractDataset.

Missing docstring.

Missing docstring for getindex(g::NCDatasets.Groups,groupname::AbstractString). Check Documenter's build log for details.

Missing docstring.

Missing docstring for Base.keys(g::NCDatasets.Groups). Check Documenter's build log for details.

Common methods

One can iterate over a dataset, attribute list, dimensions and NetCDF groups.

for (varname,var) in ds
     # all variables
     @show (varname,size(var))
 end
@@ -99,4 +94,4 @@
 for (groupname,group) in ds.groups
     # all groups
     @show (groupname,group)
-end
+end diff --git a/dev/dimensions/index.html b/dev/dimensions/index.html index ac874621..b6b27c08 100644 --- a/dev/dimensions/index.html +++ b/dev/dimensions/index.html @@ -1,6 +1,5 @@ -Dimensions · NCDatasets.jl
Edit on GitHub

Dimensions

In the NetCDF data model, dimensions have names and a length (but possibly an unlimited length) and are defined for a NetCDF dataset (or group). For a given Variable or CFVariable,the names of the corresponding dimensions are obtained with using dimnames.

Base.keys — Method
keys(d::Dimensions)

Return a list of all dimension names in NCDataset ds.

Examples

julia> ds = NCDataset("results.nc", "r");
-julia> dimnames = keys(ds.dim)
source
Base.haskey — Method
haskey(ds::NCDataset,name)
+Dimensions · NCDatasets.jl
Edit on GitHub
Dimensions
In the NetCDF data model, dimensions have names and a length (but possibly an unlimited length) and are defined for a NetCDF dataset (or group). For a given Variable or CFVariable,the names of the corresponding dimensions are obtained with using dimnames.
Missing docstring.
Missing docstring for keys(d::NCDatasets.Dimensions). Check Documenter's build log for details.
Base.haskey — Method
haskey(ds::NCDataset,name)
 haskey(d::Dimensions,name)
 haskey(ds::Attributes,name)
Return true if the NCDataset ds (or dimension/attribute list) has a variable (dimension/attribute) with the name name. For example:
ds = NCDataset("/tmp/test.nc","r")
 if haskey(ds,"temperature")
@@ -9,7 +8,7 @@
 
 if haskey(ds.dim,"lon")
     println("The file has a dimension 'lon'")
-end
This example checks if the file /tmp/test.nc has a variable with the name temperature and a dimension with the name lon.
source
CommonDataModel.defDim — Function
defDim(ds::NCDataset,name,len)
Define a dimension in the data set ds with the given name and length len. If len is the special value Inf, then the dimension is considered as unlimited, i.e. it will grow as data is added to the NetCDF file.
For example:
using NCDatasets
+end
This example checks if the file /tmp/test.nc has a variable with the name temperature and a dimension with the name lon.
source
CommonDataModel.defDim — Function
defDim(ds::NCDataset,name,len)
Define a dimension in the data set ds with the given name and length len. If len is the special value Inf, then the dimension is considered as unlimited, i.e. it will grow as data is added to the NetCDF file.
For example:
using NCDatasets
 ds = NCDataset("/tmp/test.nc","c")
 defDim(ds,"lon",100)
 # [...]
@@ -24,9 +23,7 @@
 ds["unlimited_variable"][:,:,:] = randn(10,10,4)
 @show ds.dim["time"]
 # returns now 4 as 4 time slice have been added
-close(ds)
source
CommonDatamodel.defDim(ds::AbstractDataset,name::SymbolOrString,len)
Create dimension with the name name in the data set ds with the length len. len can be Inf for unlimited dimensions.
Base.setindex! — Method
Base.setindex!(d::Dimensions,len,name::AbstractString)
Defines the dimension called name to the length len. Generally dimension are defined by indexing, for example:
ds = NCDataset("file.nc","c")
-ds.dim["longitude"] = 100
If len is the special value Inf, then the dimension is considered as unlimited, i.e. it will grow as data is added to the NetCDF file.
source
One can iterate over a list of dimensions as follows:
for (dimname,dim) in ds.dim
+close(ds)
source
CommonDatamodel.defDim(ds::AbstractDataset,name::SymbolOrString,len)

Create dimension with the name name in the data set ds with the length len. len can be Inf for unlimited dimensions.

Missing docstring.

Missing docstring for unlimited(d::NCDatasets.Dimensions). Check Documenter's build log for details.

Missing docstring.

Missing docstring for setindex!(d::NCDatasets.Dimensions,len,name::AbstractString). Check Documenter's build log for details.

One can iterate over a list of dimensions as follows:

for (dimname,dim) in ds.dim
     # all dimensions
     @show (dimname,dim)
-end
+end diff --git a/dev/experimental/index.html b/dev/experimental/index.html index 952f09c6..386ca225 100644 --- a/dev/experimental/index.html +++ b/dev/experimental/index.html @@ -75,8 +75,8 @@ ds_subset.dim["lon"] # output -2

Such sliced datasets can for example be saved into a new NetCDF file using write:

write("slice.nc",ds_subset)

Any dimension not mentioned in the @view call is not sliced. While @view produces a slice based on indices, the NCDatasets.@select macro produces a slice (of an NetCDF variable or dataset) based on the values of other related variables (typically coordinates).

Data selection based on values

NCDatasets.@select — Macro
vsubset = NCDatasets.@select(v,expression)
-dssubset = NCDatasets.@select(ds,expression)

Return a subset of the variable v (or dataset ds) satisfying the condition expression as a view. The condition has the following form:

condition₁ && condition₂ && condition₃ ... conditionₙ

Every condition should involve a single 1D NetCDF variable (typically a coordinate variable, referred as coord below). If v is a variable, the related 1D NetCDF variable should have a shared dimension with the variable v. All local variables need to have a $ prefix (see examples below). This macro is experimental and subjected to change.

Every condition can either perform:

  • a nearest match: coord ≈ target_coord (for ≈ type \approx followed by the TAB-key). Only the data corresponding to the index closest to target_coord is loaded.

  • a nearest match with tolerance: coord ≈ target_coord ± tolerance. As before, but if the difference between the closest value in coord and target_coord is larger (in absolute value) than tolerance, an empty array is returned.

  • a condition operating on scalar values. For example, a condition equal to 10 <= lon <= 20 loads all data with the longitude between 10 and 20 or abs(lat) > 60 loads all variables with a latitude north of 60° N and south of 60° S (assuming that the NetCDF has the 1D variables lon and lat for longitude and latitude).

Only the data which satisfies all conditions is loaded. All conditions must be chained with an && (logical and). They should not contain additional parenthesis or other logical operators such as || (logical or).

To convert the view into a regular array one can use collect, Array or regular indexing. As in julia, views of scalars are wrapped into a zero dimensional arrays which can be dereferenced by using []. Modifying a view will modify the underlying NetCDF file (if the file is opened as writable, otherwise an error is issued).

As for any view, one can use parentindices(vsubset) to get the indices matching a select query.

Examples

Create a sample file with random data:

using NCDatasets, Dates
+2

Such sliced datasets can for example be saved into a new NetCDF file using write:

write("slice.nc",ds_subset)

Any dimension not mentioned in the @view call is not sliced. While @view produces a slice based on indices, the NCDatasets.@select macro produces a slice (of an NetCDF variable or dataset) based on the values of other related variables (typically coordinates).

Data selection based on values

CommonDataModel.@select — Macro
vsubset = CommonDataModel.@select(v,expression)
+dssubset = CommonDataModel.@select(ds,expression)

Return a subset of the variable v (or dataset ds) satisfying the condition expression as a view. The condition has the following form:

condition₁ && condition₂ && condition₃ ... conditionₙ

Every condition should involve a single 1D variable (typically a coordinate variable, referred as coord below). If v is a variable, the related 1D variable should have a shared dimension with the variable v. All local variables need to have a $ prefix (see examples below). This macro is experimental and subjected to change.

Every condition can either perform:

  • a nearest match: coord ≈ target_coord (for ≈ type \approx followed by the TAB-key). Only the data corresponding to the index closest to target_coord is loaded.

  • a nearest match with tolerance: coord ≈ target_coord ± tolerance. As before, but if the difference between the closest value in coord and target_coord is larger (in absolute value) than tolerance, an empty array is returned.

  • a condition operating on scalar values. For example, a condition equal to 10 <= lon <= 20 loads all data with the longitude between 10 and 20 or abs(lat) > 60 loads all variables with a latitude north of 60° N and south of 60° S (assuming that the has the 1D variables lon and lat for longitude and latitude).

Only the data which satisfies all conditions is loaded. All conditions must be chained with an && (logical and). They should not contain additional parenthesis or other logical operators such as || (logical or).

To convert the view into a regular array one can use collect, Array or regular indexing. As in julia, views of scalars are wrapped into a zero dimensional arrays which can be dereferenced by using []. Modifying a view will modify the underlying file (if the file is opened as writable, otherwise an error is issued).

As for any view, one can use parentindices(vsubset) to get the indices matching a select query.

Examples

Create a sample file with random data:

using NCDatasets, Dates
 fname = "sample_file.nc"
 lon = -180:180
 lat = -90:90
@@ -142,5 +142,5 @@
 v2 = ds["temperature"][findall(Dates.month.(time) .== 1 .&& salinity .>= 35)]
 
 @test v == v2
-close(ds)
Note

For optimal performance, one should try to load contiguous data ranges, in particular when the data is loaded over HTTP/OPeNDAP.

source

Experimental functions

NCDatasets.ancillaryvariables — Function
ncvar = NCDatasets.ancillaryvariables(ncv::NCDatasets.CFVariable,modifier)

Return the first ancillary variables from the NetCDF variable ncv with the standard name modifier modifier. It can be used for example to access related variable like status flags.

source
Base.filter — Function
data = NCDatasets.filter(ncv, indices...; accepted_status_flags = nothing)

Load and filter observations by replacing all variables without an acepted status flag to missing. It is used the attribute ancillary_variables to identify the status flag.

# da["data"] is 2D matrix
-good_data = NCDatasets.filter(ds["data"],:,:, accepted_status_flags = ["good_data","probably_good_data"])
source
+close(ds)
Note

For optimal performance, one should try to load contiguous data ranges, in particular when the data is loaded over HTTP/OPeNDAP.

Experimental functions

CommonDataModel.ancillaryvariables — Function
ncvar = CommonDataModel.ancillaryvariables(ncv::CFVariable,modifier)

Return the first ancillary variables from the NetCDF (or other format) variable ncv with the standard name modifier modifier. It can be used for example to access related variable like status flags.

Base.filter — Function
data = CommonDataModel.filter(ncv, indices...; accepted_status_flags = nothing)

Load and filter observations by replacing all variables without an acepted status flag to missing. It is used the attribute ancillary_variables to identify the status flag.

# da["data"] is 2D matrix
+good_data = NCDatasets.filter(ds["data"],:,:, accepted_status_flags = ["good_data","probably_good_data"])
diff --git a/dev/index.html b/dev/index.html index c77c9a70..6679cf6f 100644 --- a/dev/index.html +++ b/dev/index.html @@ -131,4 +131,4 @@ # if the attribute does not exists units = get(v,"units","adimensional") -close(ds) +close(ds) diff --git a/dev/issues/index.html b/dev/issues/index.html index 15e68b83..3802662a 100644 --- a/dev/issues/index.html +++ b/dev/issues/index.html @@ -36,4 +36,4 @@ @ stdin:1 during initialization of module NetCDF_jll

You will likely have similar issues with julia installed from other package managers (like Debian/Ubuntu apt, Homebrew...). The only supported solution is to install the offical julia builds.

version `CURL_4' not found

If you see the following error:

ERROR: LoadError: InitError: could not load library "/home/user/.julia/artifacts/461703969206dd426cc6b4d99f69f6ffab2a9779/lib/libnetcdf.so"
 /usr/lib/x86_64-linux-gnu/libcurl.so: version `CURL_4' not found (required by /home/user/.julia/artifacts/461703969206dd426cc6b4d99f69f6ffab2a9779/lib/libnetcdf.so)

make sure that you are using the offical julia builds (see above) and that your LD_LIBRARY_PATH and LD_PRELOAD are empty. You can verify this by running the following commands in a terminal:

echo $LD_PRELOAD
-echo $LD_LIBRARY_PATH

Corner cases

+echo $LD_LIBRARY_PATH

Corner cases

diff --git a/dev/performance/index.html b/dev/performance/index.html index dca191ca..d7316f89 100644 --- a/dev/performance/index.html +++ b/dev/performance/index.html @@ -28,4 +28,4 @@ v = ds["v1"][:,1:3,:]; # fast v = ds["v1"][:,:,CartesianIndex(1)] # slow v = ds["v1"][:,:,1] # fast -close(ds) +close(ds) diff --git a/dev/search/index.html b/dev/search/index.html index e54b63f2..65f57857 100644 --- a/dev/search/index.html +++ b/dev/search/index.html @@ -1,2 +1,2 @@ -Search · NCDatasets.jl

Loading search...

    +Search · NCDatasets.jl

    Loading search...

      diff --git a/dev/search_index.js b/dev/search_index.js index 43d57958..9bf9ea44 100644 --- a/dev/search_index.js +++ b/dev/search_index.js @@ -1,3 +1,3 @@ var documenterSearchIndex = {"docs": -[{"location":"dataset/#Datasets","page":"Datasets","title":"Datasets","text":"","category":"section"},{"location":"dataset/","page":"Datasets","title":"Datasets","text":"This page is about loading/writing, examining and operating directly on entire NetCDF datasets. For functions regarding the variables stored in them, see the Variables page.","category":"page"},{"location":"dataset/","page":"Datasets","title":"Datasets","text":"Both variables and datasets share the functionality of the Attributes section.","category":"page"},{"location":"dataset/","page":"Datasets","title":"Datasets","text":"NCDataset","category":"page"},{"location":"dataset/#NCDatasets.NCDataset","page":"Datasets","title":"NCDatasets.NCDataset","text":"NCDataset(filename::AbstractString, mode = \"r\";\n format::Symbol = :netcdf4,\n share::Bool = false,\n diskless::Bool = false,\n persist::Bool = false,\n memory::Union{Vector{UInt8},Nothing} = nothing,\n attrib = [])\n\nLoad, create, or even overwrite a NetCDF file at filename, depending on mode\n\n\"r\" (default) : open an existing netCDF file or OPeNDAP URL in read-only mode.\n\"c\" : create a new NetCDF file at filename (an existing file with the same name will be overwritten).\n\"a\" : open filename into append mode (i.e. existing data in the netCDF file is not overwritten and a variable can be added).\n\nIf share is true, the NC_SHARE flag is set allowing to have multiple processes to read the file and one writer process. Likewise setting diskless or persist to true will enable the flags NC_DISKLESS or NC_PERSIST flag. More information is available in the NetCDF C-API.\n\nNotice that this does not close the dataset, use close on the result (or see below the do-block).\n\nThe optional parameter attrib is an iterable of attribute name and attribute value pairs, for example a Dict, DataStructures.OrderedDict or simply a vector of pairs (see example below).\n\nSupported format values:\n\n:netcdf4 (default): HDF5-based NetCDF format.\n:netcdf4_classic: Only netCDF 3 compatible API features will be used.\n:netcdf3_classic: classic netCDF format supporting only files smaller than 2GB.\n:netcdf3_64bit_offset: improved netCDF format supporting files larger than 2GB.\n:netcdf5_64bit_data: improved netCDF format supporting 64-bit integer data types.\n\nFiles can also be open and automatically closed with a do block.\n\nNCDataset(\"file.nc\") do ds\n data = ds[\"temperature\"][:,:]\nend\n\nHere is an attribute example:\n\nusing DataStructures\nNCDataset(\"file.nc\", \"c\", attrib = OrderedDict(\"title\" => \"my first netCDF file\")) do ds\n defVar(ds,\"temp\",[10.,20.,30.],(\"time\",))\nend;\n\nThe NetCDF dataset can also be a memory as a vector of bytes. A non-empty string a filename is still required, for example:\n\nusing NCDataset, HTTP\nresp = HTTP.get(\"https://www.unidata.ucar.edu/software/netcdf/examples/ECMWF_ERA-40_subset.nc\")\nds = NCDataset(\"some_string\",\"r\",memory = resp.body)\ntotal_precipitation = ds[\"tp\"][:,:,:]\nclose(ds)\n\nDataset is an alias of NCDataset.\n\n\n\n\n\nmfds = NCDataset(fnames, mode = \"r\"; aggdim = nothing, deferopen = true,\n isnewdim = false,\n constvars = [])\n\nOpens a multi-file dataset in read-only \"r\" or append mode \"a\". fnames is a vector of file names.\n\nVariables are aggregated over the first unlimited dimension or over the dimension aggdim if specified. Variables without the dimensions aggdim are not aggregated. All variables containing the dimension aggdim are aggregated. The variable who do not contain the dimension aggdim are assumed constant.\n\nIf variables should be aggregated over a new dimension (not present in the NetCDF file), one should set isnewdim to true. All NetCDF files should have the same variables, attributes and groupes. Per default, all variables will have an additional dimension unless they are marked as constant using the constvars parameter.\n\nThe append mode is only implemented when deferopen is false. If deferopen is false, all files are opened at the same time. However the operating system might limit the number of open files. In Linux, the limit can be controled with the command ulimit.\n\nAll metadata (attributes and dimension length are assumed to be the same for all NetCDF files. Otherwise reading the attribute of a multi-file dataset would be ambiguous. An exception to this rule is the length of the dimension over which the data is aggregated. This aggregation dimension can varify from file to file.\n\nSetting the experimental flag _aggdimconstant to true means that the length of the aggregation dimension is constant. This speeds up the creating of a multi-file dataset as only the metadata of the first file has to be loaded.\n\nExamples:\n\nYou can use Glob.jl to make fnames from a file pattern, e.g.\n\nusing NCDatasets, Glob\nds = NCDataset(glob(\"ERA5_monthly3D_reanalysis_*.nc\"))\n\nAggregation over a new dimension:\n\nusing NCDatasets\nfor i = 1:3\n NCDataset(\"foo$i.nc\",\"c\") do ds\n defVar(ds,\"data\",[10., 11., 12., 13.], (\"lon\",))\n end\nend\n\nds = NCDataset([\"foo$i.nc\" for i = 1:3],aggdim = \"sample\", isnewdim = true)\nsize(ds[\"data\"])\n# output\n# (4, 3)\n\n\n\n\n\n","category":"type"},{"location":"dataset/","page":"Datasets","title":"Datasets","text":"Useful functions that operate on datasets are:","category":"page"},{"location":"dataset/","page":"Datasets","title":"Datasets","text":"keys(ds::NCDataset)\nhaskey\ngetindex(ds::NCDataset,varname::AbstractString)\nvariable\ncfvariable\nsync\nclose\nNCDatasets.path\nncgen\nvarbyattrib\nwrite","category":"page"},{"location":"dataset/#Base.keys-Tuple{NCDataset}","page":"Datasets","title":"Base.keys","text":"keys(ds::NCDataset)\n\nReturn a list of all variables names in NCDataset ds.\n\n\n\n\n\n","category":"method"},{"location":"dataset/#Base.haskey","page":"Datasets","title":"Base.haskey","text":"haskey(ds::NCDataset,name)\nhaskey(d::Dimensions,name)\nhaskey(ds::Attributes,name)\n\nReturn true if the NCDataset ds (or dimension/attribute list) has a variable (dimension/attribute) with the name name. For example:\n\nds = NCDataset(\"/tmp/test.nc\",\"r\")\nif haskey(ds,\"temperature\")\n println(\"The file has a variable 'temperature'\")\nend\n\nif haskey(ds.dim,\"lon\")\n println(\"The file has a dimension 'lon'\")\nend\n\nThis example checks if the file /tmp/test.nc has a variable with the name temperature and a dimension with the name lon.\n\n\n\n\n\nBase.haskey(a::Attributes,name)\n\nCheck if name is an attribute\n\n\n\n\n\n","category":"function"},{"location":"dataset/#Base.getindex-Tuple{NCDataset, AbstractString}","page":"Datasets","title":"Base.getindex","text":"v = getindex(ds::NCDataset, varname::AbstractString)\n\nReturn the NetCDF variable varname in the dataset ds as a NCDataset.CFVariable. The following CF convention are honored when the variable is indexed:\n\n_FillValue or missing_value (which can be a list) will be returned as missing. NCDatasets does not use implicitely the default NetCDF fill values when reading data.\nscale_factor and add_offset are applied (output = scale_factor * data_in_file + add_offset)\ntime variables (recognized by the units attribute and possibly the calendar attribute) are returned usually as DateTime object. Note that CFTime.DateTimeAllLeap, CFTime.DateTimeNoLeap and CF.TimeDateTime360Day cannot be converted to the proleptic gregorian calendar used in julia and are returned as such. (See CFTime.jl for more information about those date types.) If a calendar is defined but not among the ones specified in the CF convention, then the data in the NetCDF file is not converted into a date structure.\n\nA call getindex(ds, varname) is usually written as ds[varname].\n\nIf variable represents a cell boundary, the attributes calendar and units of the related NetCDF variables are used, if they are not specified. For example:\n\ndimensions:\n time = UNLIMITED; // (5 currently)\n nv = 2;\nvariables:\n double time(time);\n time:long_name = \"time\";\n time:units = \"hours since 1998-04-019 06:00:00\";\n time:bounds = \"time_bnds\";\n double time_bnds(time,nv);\n\nIn this case, the variable time_bnds uses the units and calendar of time because both variables are related thought the bounds attribute following the CF conventions.\n\nSee also cfvariable(ds, varname).\n\n\n\n\n\n","category":"method"},{"location":"dataset/#CommonDataModel.variable","page":"Datasets","title":"CommonDataModel.variable","text":"v = variable(ds::NCDataset,varname::String)\n\nReturn the NetCDF variable varname in the dataset ds as a NCDataset.Variable. No scaling or other transformations are applied when the variable v is indexed.\n\n\n\n\n\nCommonDataModel.variable(ds::AbstractDataset,variablename::SymbolOrString)\n\nReturn the variable with the name variablename from the data set ds.\n\n\n\n\n\n","category":"function"},{"location":"dataset/#CommonDataModel.cfvariable","page":"Datasets","title":"CommonDataModel.cfvariable","text":"v = cfvariable(ds::NCDataset,varname::SymbolOrString; = )\n\nReturn the variable varname in the dataset ds as a NCDataset.CFVariable. The keyword argument are the attributes (fillvalue, missing_value, scale_factor, add_offset, units and calendar) relevant to the CF conventions. By specifing the value of these attributes, the one can override the value specified in the data set. If the attribute is set to nothing, then the attribute is not loaded and the corresponding transformation is ignored. This function is similar to ds[varname] with the additional flexibility that some variable attributes can be overridden.\n\nExample:\n\nNCDataset(\"foo.nc\",\"c\") do ds\n defVar(ds,\"data\",[10., 11., 12., 13.], (\"time\",), attrib = Dict(\n \"add_offset\" => 10.,\n \"scale_factor\" => 0.2))\nend\n\n# The stored (packed) valued are [0., 5., 10., 15.]\n# since 0.2 .* [0., 5., 10., 15.] .+ 10 is [10., 11., 12., 13.]\n\nds = NCDataset(\"foo.nc\");\n\n@show ds[\"data\"].var[:]\n# returns [0., 5., 10., 15.]\n\n@show cfvariable(ds,\"data\")[:]\n# returns [10., 11., 12., 13.]\n\n# neither add_offset nor scale_factor are applied\n@show cfvariable(ds,\"data\", add_offset = nothing, scale_factor = nothing)[:]\n# returns [0, 5, 10, 15]\n\n# add_offset is applied but not scale_factor\n@show cfvariable(ds,\"data\", scale_factor = nothing)[:]\n# returns [10, 15, 20, 25]\n\n# 0 is declared as the fill value (add_offset and scale_factor are applied as usual)\n@show cfvariable(ds,\"data\", fillvalue = 0)[:]\n# return [missing, 11., 12., 13.]\n\n# Use the time units: days since 2000-01-01\n@show cfvariable(ds,\"data\", units = \"days since 2000-01-01\")[:]\n# returns [DateTime(2000,1,11), DateTime(2000,1,12), DateTime(2000,1,13), DateTime(2000,1,14)]\n\nclose(ds)\n\n\n\n\n\n","category":"function"},{"location":"dataset/#NCDatasets.sync","page":"Datasets","title":"NCDatasets.sync","text":"sync(ds::NCDataset)\n\nWrite all changes in NCDataset ds to the disk.\n\n\n\n\n\n","category":"function"},{"location":"dataset/#Base.close","page":"Datasets","title":"Base.close","text":"close(ds::NCDataset)\n\nClose the NCDataset ds. All pending changes will be written to the disk.\n\n\n\n\n\n","category":"function"},{"location":"dataset/#CommonDataModel.path","page":"Datasets","title":"CommonDataModel.path","text":"path(ds::NCDataset)\n\nReturn the file path (or the opendap URL) of the NCDataset ds\n\n\n\n\n\nCommonDatamodel.path(ds::AbstractDataset)\n\nFile path of the data set ds.\n\n\n\n\n\n","category":"function"},{"location":"dataset/#NCDatasets.ncgen","page":"Datasets","title":"NCDatasets.ncgen","text":"ncgen(fname; ...)\nncgen(fname,jlname; ...)\n\nGenerate the Julia code that would produce a NetCDF file with the same metadata as the NetCDF file fname. The code is placed in the file jlname or printed to the standard output. By default the new NetCDF file is called filename.nc. This can be changed with the optional parameter newfname.\n\n\n\n\n\n","category":"function"},{"location":"dataset/#NCDatasets.varbyattrib","page":"Datasets","title":"NCDatasets.varbyattrib","text":"varbyattrib(ds, attname = attval)\n\nReturns a list of variable(s) which has the attribute attname matching the value attval in the dataset ds. The list is empty if the none of the variables has the match. The output is a list of CFVariables.\n\nExamples\n\nLoad all the data of the first variable with standard name \"longitude\" from the NetCDF file results.nc.\n\njulia> ds = NCDataset(\"results.nc\", \"r\");\njulia> data = varbyattrib(ds, standard_name = \"longitude\")[1][:]\n\n\n\n\n\n","category":"function"},{"location":"dataset/#Base.write","page":"Datasets","title":"Base.write","text":"write(dest_filename::AbstractString, src::AbstractNCDataset; include = keys(src), exclude = [], idimensions = Dict())\nwrite(dest::NCDataset, src::AbstractNCDataset; include = keys(src), exclude = [], idimensions = Dict())\n\nWrite the variables of src dataset into an empty dest dataset (which must be opened in mode \"a\" or \"c\"). The keywords include and exclude configure which variable of src should be included (by default all), or which should be excluded (by default none).\n\nIf the first argument is a file name, then the dataset is open in create mode (\"c\").\n\nThis function is useful when you want to save the dataset from a multi-file dataset.\n\nTo save a subset, one can use the view function view to virtually slice a dataset:\n\nExample\n\nNCDataset(fname_src) do ds\n write(fname_slice,view(ds, lon = 2:3))\n # deprecated\n # write(fname_slice,ds,idimensions = Dict(\"lon\" => 2:3))\nend\n\nAll variables in the source file fname_src with a dimension lon will be sliced along the indices 2:3 for the lon dimension. All attributes (and variables without a dimension lon) will be copied over unmodified.\n\n\n\n\n\n","category":"function"},{"location":"dataset/","page":"Datasets","title":"Datasets","text":"Notice that DateTime-structures from CFTime are used to represent time for non-standard calendars. Otherwise, we attempt to use standard structures from the Julia standard library Dates.","category":"page"},{"location":"dataset/#Groups","page":"Datasets","title":"Groups","text":"","category":"section"},{"location":"dataset/","page":"Datasets","title":"Datasets","text":"defGroup\ngetindex(g::NCDatasets.Groups,groupname::AbstractString)\nBase.keys(g::NCDatasets.Groups)","category":"page"},{"location":"dataset/#CommonDataModel.defGroup","page":"Datasets","title":"CommonDataModel.defGroup","text":"defGroup(ds::NCDataset,groupname; attrib = []))\n\nCreate the group with the name groupname in the dataset ds. attrib is a list of attribute name and attribute value pairs (see NCDataset).\n\n\n\n\n\ngroup = CommonDatamodel.defGroup(ds::AbstractDataset,name::SymbolOrString)\n\nCreate an empty sub-group with the name name in the data set ds. The group is a sub-type of AbstractDataset.\n\n\n\n\n\n","category":"function"},{"location":"dataset/#Base.getindex-Tuple{NCDatasets.Groups, AbstractString}","page":"Datasets","title":"Base.getindex","text":"group = getindex(g::NCDatasets.Groups,groupname::AbstractString)\n\nReturn the NetCDF group with the name groupname. For example:\n\njulia> ds = NCDataset(\"results.nc\", \"r\");\njulia> forecast_group = ds.group[\"forecast\"]\njulia> forecast_temp = forecast_group[\"temperature\"]\n\n\n\n\n\n","category":"method"},{"location":"dataset/#Base.keys-Tuple{NCDatasets.Groups}","page":"Datasets","title":"Base.keys","text":"Base.keys(g::NCDatasets.Groups)\n\nReturn the names of all subgroubs of the group g.\n\n\n\n\n\n","category":"method"},{"location":"dataset/#Common-methods","page":"Datasets","title":"Common methods","text":"","category":"section"},{"location":"dataset/","page":"Datasets","title":"Datasets","text":"One can iterate over a dataset, attribute list, dimensions and NetCDF groups.","category":"page"},{"location":"dataset/","page":"Datasets","title":"Datasets","text":"for (varname,var) in ds\n # all variables\n @show (varname,size(var))\nend\n\nfor (attribname,attrib) in ds.attrib\n # all attributes\n @show (attribname,attrib)\nend\n\nfor (groupname,group) in ds.groups\n # all groups\n @show (groupname,group)\nend","category":"page"},{"location":"performance/#performance_tips","page":"Performance tips","title":"Performance tips","text":"","category":"section"},{"location":"performance/","page":"Performance tips","title":"Performance tips","text":"Reading data from a file is not type-stable, because the type of the output of the read operation is dependent on the type defined in the NetCDF files and the value of various attribute (like scale_factor, add_offset and units for time conversion). All this information cannot be inferred from a static analysis of the source code. It is therefore recommended to use type annotation if the resulting type of a read operation in known:","category":"page"},{"location":"performance/","page":"Performance tips","title":"Performance tips","text":"ds = NCDataset(\"file.nc\")\nnctemp = ds[\"temp\"]\ntemp = nctemp[:,:] :: Array{Float32,2}\n\n# heavy computation using temp\n# ...","category":"page"},{"location":"performance/","page":"Performance tips","title":"Performance tips","text":"Alternatively, one can also use so-called function barriers since the function heavy_computation will be specialized based on the type its input parameters.","category":"page"},{"location":"performance/","page":"Performance tips","title":"Performance tips","text":"function heavy_computation(temp)\n# heavy computation using temp\n# ...\nend\n\nds = NCDataset(\"file.nc\")\nnctemp = ds[\"temp\"]\ntemp = nctemp[:,:]\noutput = heavy_computation(temp)","category":"page"},{"location":"performance/","page":"Performance tips","title":"Performance tips","text":"Calling the barrier function with nctemp would also be type-stable. Using the in-place NCDatasets.load! function (which is unexported, so it has to be prefixed with the module name) does also lead to type-stable code and allows to reuse a memory buffer:","category":"page"},{"location":"performance/","page":"Performance tips","title":"Performance tips","text":"ds = NCDataset(\"file.nc\")\n\ntemp = zeros(Float32,10,20)\nNCDatasets.load!(variable(ds,\"temp\"),temp,:,:)","category":"page"},{"location":"performance/","page":"Performance tips","title":"Performance tips","text":"Most julia functions (like mean, sum,... from the module Statistics) access an array element-wise. It is generally much faster to load the data in memory (if possible) to make the computation.","category":"page"},{"location":"performance/","page":"Performance tips","title":"Performance tips","text":"using NCDatasets, BenchmarkTools, Statistics\nds = NCDataset(\"file.nc\",\"c\")\ndata = randn(100,100);\ndefVar(ds,\"myvar\",data,(\"lon\",\"lat\"))\nclose(ds)\n\nds = NCDataset(\"file.nc\")\n@btime mean(ds[\"myvar\"]) # takes 107.357 ms\n@btime mean(ds[\"myvar\"][:,:]) # takes 106.873 μs, 1000 times faster\nclose(ds)","category":"page"},{"location":"performance/","page":"Performance tips","title":"Performance tips","text":"Avoid, when possible, indexing with arrays and CartesianIndex as they also result in loading the data element-wise.","category":"page"},{"location":"performance/","page":"Performance tips","title":"Performance tips","text":"ds = NCDataset(\"dataset.nc\");\nv = ds[\"v1\"][:,1:3,:]; # fast\nv = ds[\"v1\"][:,:,CartesianIndex(1)] # slow\nv = ds[\"v1\"][:,:,1] # fast\nclose(ds)","category":"page"},{"location":"experimental/#Experimental-features","page":"Experimental features","title":"Experimental features","text":"","category":"section"},{"location":"experimental/#Multi-file-support","page":"Experimental features","title":"Multi-file support","text":"","category":"section"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"Multiple files can also be aggregated over a given dimension (or the record dimension). In this example, 3 sea surface temperature fields from the 1992-01-01 to 1992-01-03 are aggregated using the OPeNDAP service from PODAAC.","category":"page"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"using NCDatasets, Printf, Dates\n\nfunction url(dt)\n doy = @sprintf(\"%03d\",Dates.dayofyear(dt))\n y = @sprintf(\"%04d\",Dates.year(dt))\n yyyymmdd = Dates.format(dt,\"yyyymmdd\")\n return \"https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/CMC/CMC0.2deg/v2/$y/$doy/$(yyyymmdd)120000-CMC-L4_GHRSST-SSTfnd-CMC0.2deg-GLOB-v02.0-fv02.0.nc\"\nend\n\nds = NCDataset(url.(DateTime(1992,1,1):Dates.Day(1):DateTime(1992,1,3)),aggdim = \"time\");\nSST2 = ds[\"analysed_sst\"][:,:,:];\nclose(ds)","category":"page"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"If there is a network or server issue, you will see an error message like NetCDF: I/O failure.","category":"page"},{"location":"experimental/#CF-Standard-Names","page":"Experimental features","title":"CF Standard Names","text":"","category":"section"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"The CF Conventions do not define how the different NetCDF variables are named, but the meaning of a variable is defined by the standard_name attribute.","category":"page"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"using NCDatasets, DataStructures\nds = NCDataset(tempname(),\"c\")\n\nnclon = defVar(ds,\"lon\", 1:10, (\"lon\",),attrib = OrderedDict(\n \"standard_name\" => \"longitude\",\n))\nnclat = defVar(ds,\"lat\", 1:11, (\"lat\",),attrib = OrderedDict(\n \"standard_name\" => \"latitude\",\n))\nncvar = defVar(ds,\"bat\", zeros(10,11), (\"lon\", \"lat\"), attrib = OrderedDict(\n \"standard_name\" => \"height\",\n))\n\nncbat = ds[CF\"height\"]\n# the same as\n# ncbat = varbyattrib(ds,standard_name = \"height\")[1]\n\nname(ncbat)\n# output\n\"bat\"","category":"page"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"If there are multiple variables with the standard_name equal to height, an error is returned because it is ambiguous which variable should be accessed.","category":"page"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"All variables whose dimensions are also dimensions of ncbat are considered as related and can also be accessed by sub-setting ncbat with their variable names of CF Standard name:","category":"page"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"nclon_of_bat = ncbat[CF\"longitude\"]\n# same as\n# nclon_of_bat = ncbat[\"lon\"]\nname(nclon_of_bat)\n# output\n\"lon\"","category":"page"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"The previous call to ncbat[CF\"longitude\"] would also worked if there are multiple variables with a standard name longitude defined in a dataset as long as they have different dimension names (which is commonly the case for model output on staggered grid such as Regional Ocean Modeling System).","category":"page"},{"location":"experimental/#Views","page":"Experimental features","title":"Views","text":"","category":"section"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"In Julia, a view of an array is a subset of an array but whose elements still point to the original parent array. If one modifies an element of a view, the corresponding element in the parent array is modified too:","category":"page"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"A = zeros(4,4)\nsubset = @view A[2:3,2:4]\n# or\n# subset = view(A,2:3,2:4)\n\nsubset[1,1] = 2\nA[2,2]\n# output\n2.0","category":"page"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"Views do not use copy of the array. The parent array and the indices of the view are obtained via the function parent and parentindices.","category":"page"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"parent(subset) == A\n# true, as both arrays are the same\n\nparentindices(subset)\n# output\n(2:3, 2:4)","category":"page"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"In NCDatasets, variables can also be sliced as a view:","category":"page"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"using NCDatasets, DataStructures\nds = NCDataset(tempname(),\"c\")\n\nnclon = defVar(ds,\"lon\", 1:10, (\"lon\",))\nnclat = defVar(ds,\"lat\", 1:11, (\"lat\",))\nncvar = defVar(ds,\"bat\", zeros(10,11), (\"lon\", \"lat\"), attrib = OrderedDict(\n \"standard_name\" => \"height\",\n))\n\nncvar_subset = @view ncvar[2:4,2:3]\n# or\n# ncvar_subset = view(ncvar,2:4,2:3)\n\nncvar_subset[1,1] = 2\n# ncvar[2,2] is now 2\n\nncvar_subset.attrib[\"standard_name\"]\n\n# output\n\"height\"","category":"page"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"This is useful for example when even the sliced array is too large to be loaded in RAM or when all attributes need to be preserved for the sliced array.","category":"page"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"The variables lon and lat are related to bat because all dimensions of the variables lon and lat are also dimensions of bat (which is commonly the case for coordinate variables). Such related variables can be retrieved by indexing the NetCDF variables with the name of the corresponding variable:","category":"page"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"lon_subset = ncvar_subset[\"lon\"]\nlon_subset[:] == [2, 3, 4]\n# output\ntrue","category":"page"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"A view of a NetCDF variable also implements the function parent and parentindices with the same meaning as for julia Arrays.","category":"page"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"A whole dataset can also be sliced using a view(ds, dim1=range1, dim2=range2...). For example:","category":"page"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"ds_subset = view(ds, lon = 2:3, lat = 2:4)\n# or\n# ds_subset = @view ds[lon = 2:3, lat = 2:4]\nds_subset.dim[\"lon\"]\n\n# output\n2","category":"page"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"Such sliced datasets can for example be saved into a new NetCDF file using write:","category":"page"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"write(\"slice.nc\",ds_subset)","category":"page"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"Any dimension not mentioned in the @view call is not sliced. While @view produces a slice based on indices, the NCDatasets.@select macro produces a slice (of an NetCDF variable or dataset) based on the values of other related variables (typically coordinates).","category":"page"},{"location":"experimental/#Data-selection-based-on-values","page":"Experimental features","title":"Data selection based on values","text":"","category":"section"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"NCDatasets.@select","category":"page"},{"location":"experimental/#NCDatasets.@select","page":"Experimental features","title":"NCDatasets.@select","text":"vsubset = NCDatasets.@select(v,expression)\ndssubset = NCDatasets.@select(ds,expression)\n\nReturn a subset of the variable v (or dataset ds) satisfying the condition expression as a view. The condition has the following form:\n\ncondition₁ && condition₂ && condition₃ ... conditionₙ\n\nEvery condition should involve a single 1D NetCDF variable (typically a coordinate variable, referred as coord below). If v is a variable, the related 1D NetCDF variable should have a shared dimension with the variable v. All local variables need to have a $ prefix (see examples below). This macro is experimental and subjected to change.\n\nEvery condition can either perform:\n\na nearest match: coord ≈ target_coord (for ≈ type \\approx followed by the TAB-key). Only the data corresponding to the index closest to target_coord is loaded.\na nearest match with tolerance: coord ≈ target_coord ± tolerance. As before, but if the difference between the closest value in coord and target_coord is larger (in absolute value) than tolerance, an empty array is returned.\na condition operating on scalar values. For example, a condition equal to 10 <= lon <= 20 loads all data with the longitude between 10 and 20 or abs(lat) > 60 loads all variables with a latitude north of 60° N and south of 60° S (assuming that the NetCDF has the 1D variables lon and lat for longitude and latitude).\n\nOnly the data which satisfies all conditions is loaded. All conditions must be chained with an && (logical and). They should not contain additional parenthesis or other logical operators such as || (logical or).\n\nTo convert the view into a regular array one can use collect, Array or regular indexing. As in julia, views of scalars are wrapped into a zero dimensional arrays which can be dereferenced by using []. Modifying a view will modify the underlying NetCDF file (if the file is opened as writable, otherwise an error is issued).\n\nAs for any view, one can use parentindices(vsubset) to get the indices matching a select query.\n\nExamples\n\nCreate a sample file with random data:\n\nusing NCDatasets, Dates\nfname = \"sample_file.nc\"\nlon = -180:180\nlat = -90:90\ntime = DateTime(2000,1,1):Day(1):DateTime(2000,1,3)\nSST = randn(length(lon),length(lat),length(time))\n\nds = NCDataset(fname,\"c\")\ndefVar(ds,\"lon\",lon,(\"lon\",));\ndefVar(ds,\"lat\",lat,(\"lat\",));\ndefVar(ds,\"time\",time,(\"time\",));\ndefVar(ds,\"SST\",SST,(\"lon\",\"lat\",\"time\"));\n\n\n# load by bounding box\nv = NCDatasets.@select(ds[\"SST\"],30 <= lon <= 60 && 40 <= lat <= 90)\n\n# substitute a local variable in condition using $\nlonr = (30,60) # longitude range\nlatr = (40,90) # latitude range\n\nv = NCDatasets.@select(ds[\"SST\"],$lonr[1] <= lon <= $lonr[2] && $latr[1] <= lat <= $latr[2])\n\n# You can also select based on `ClosedInterval`s from `IntervalSets.jl`.\n# Both 30..60 and 65 ± 25 construct `ClosedInterval`s, see their documentation for details.\n\nlon_interval = 30..60\nlat_interval = 65 ± 25\nv = NCDatasets.@select(ds[\"SST\"], lon ∈ $lon_interval && lat ∈ $lat_interval)\n\n# get the indices matching the select query\n(lon_indices,lat_indices,time_indices) = parentindices(v)\n\n# get longitude matchting the select query\nv_lon = v[\"lon\"]\n\n# find the nearest time instance\nv = NCDatasets.@select(ds[\"SST\"],time ≈ DateTime(2000,1,4))\n\n# find the nearest time instance but not earlier or later than 2 hours\n# an empty array is returned if no time instance is present\n\nv = NCDatasets.@select(ds[\"SST\"],time ≈ DateTime(2000,1,3,1) ± Hour(2))\n\nclose(ds)\n\nAny 1D variable with the same dimension name can be used in @select. For example, if we have a time series of temperature and salinity, the temperature values can also be selected based on salinity:\n\n# create a sample time series\nusing NCDatasets, Dates\nfname = \"sample_series.nc\"\ntime = DateTime(2000,1,1):Day(1):DateTime(2009,12,31)\nsalinity = randn(length(time)) .+ 35\ntemperature = randn(length(time))\n\nNCDataset(fname,\"c\") do ds\n defVar(ds,\"time\",time,(\"time\",));\n defVar(ds,\"salinity\",salinity,(\"time\",));\n defVar(ds,\"temperature\",temperature,(\"time\",));\nend\n\nds = NCDataset(fname)\n\n# load all temperature data from January where the salinity is larger than 35.\nv = NCDatasets.@select(ds[\"temperature\"],Dates.month(time) == 1 && salinity >= 35)\n\n# this is equivalent to\nv2 = ds[\"temperature\"][findall(Dates.month.(time) .== 1 .&& salinity .>= 35)]\n\n@test v == v2\nclose(ds)\n\nnote: Note\nFor optimal performance, one should try to load contiguous data ranges, in particular when the data is loaded over HTTP/OPeNDAP.\n\n\n\n\n\n","category":"macro"},{"location":"experimental/#Experimental-functions","page":"Experimental features","title":"Experimental functions","text":"","category":"section"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"NCDatasets.ancillaryvariables\nNCDatasets.filter","category":"page"},{"location":"experimental/#NCDatasets.ancillaryvariables","page":"Experimental features","title":"NCDatasets.ancillaryvariables","text":"ncvar = NCDatasets.ancillaryvariables(ncv::NCDatasets.CFVariable,modifier)\n\nReturn the first ancillary variables from the NetCDF variable ncv with the standard name modifier modifier. It can be used for example to access related variable like status flags.\n\n\n\n\n\n","category":"function"},{"location":"experimental/#Base.filter","page":"Experimental features","title":"Base.filter","text":"data = NCDatasets.filter(ncv, indices...; accepted_status_flags = nothing)\n\nLoad and filter observations by replacing all variables without an acepted status flag to missing. It is used the attribute ancillary_variables to identify the status flag.\n\n# da[\"data\"] is 2D matrix\ngood_data = NCDatasets.filter(ds[\"data\"],:,:, accepted_status_flags = [\"good_data\",\"probably_good_data\"])\n\n\n\n\n\n","category":"function"},{"location":"tutorials/#Tutorials","page":"Tutorials","title":"Tutorials","text":"","category":"section"},{"location":"tutorials/#Data-from-Copernicus-Marine","page":"Tutorials","title":"Data from Copernicus Marine","text":"","category":"section"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"This examples shows how to download sea surface temperature from the Mediterranean Sea High Resolution and Ultra High Resolution Sea Surface Temperature Analysis provided by Copernicus Marine Service. Username and password are obtained after registration.","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"The username and password can be added to the URL. For example https://example.org/path should become https://username:password@example.org/path:","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"using NCDatasets, PyPlot, Statistics, URIs\n\nusername = \"your_username\"\npassword = \"your_password\"\n\nurl = \"https://nrt.cmems-du.eu/thredds/dodsC/SST_MED_SST_L4_NRT_OBSERVATIONS_010_004_a_V2\"\n\n# add username and password to url\n# username or password can contain special characters\nusername_escaped = URIs.escapeuri(username)\npassword_escaped = URIs.escapeuri(password)\nurl2 = string(URI(URI(url),userinfo = string(username_escaped,\":\",password_escaped)))\n\nds = NCDataset(url2)\n\nncvar = ds[\"analysed_sst\"];\nSST = ncvar[:,:,1]\nlon = ds[\"lon\"][:]\nlat = ds[\"lat\"][:]\ntime = ds[\"time\"][1]\n\nclf()\npcolormesh(lon,lat,nomissing(SST,NaN)')\ncbar = colorbar(orientation=\"horizontal\")\ncbar.set_label(ncvar.attrib[\"units\"])\ngca().set_aspect(1/cosd(mean(lat)))\n\ntitle(\"$(ncvar.attrib[\"long_name\"]) $time\")","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"Alternatively, one can also create the files .netrc and .ncrc in the your home directory with your credentials as explained for the NASA EarthData example which allows you to manage your passwords in a central configuration file and reduce the risk to accidentally share your credentials.","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"(Image: example_SST_CMEMS.png)","category":"page"},{"location":"tutorials/#Data-from-NASA-EarthData","page":"Tutorials","title":"Data from NASA EarthData","text":"","category":"section"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"This example shows show to download data via OPeNDAP from the NASA EarthData which requires a username and password.","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"You need to be registered at https://urs.earthdata.nasa.gov/users/new to get your credentials.","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"The example requires NCDatasets 0.12.5.","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"Simply adding username and password to the URL for NASA Earth Data OPeNDPAP services is unfortunately not sufficient as it uses a different authentication mechanism.","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"Create a .netrc file with the following content in your home directory:","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"machine urs.earthdata.nasa.gov\n login YOUR_USERNAME\n password YOUR_PASSWORD","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"where YOUR_USERNAME and YOUR_PASSWORD is your Earth Data username and password.","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"Create a .ncrc file with the following content in your home directory[1]:","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"HTTP.NETRC=/home/abarth/.netrc","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"where HTTP.NETRC is the full path to your new .netrc file[2]. You can test whether your configuration files are correct independently of NCDatasets by using the tool ncdump:","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"ncdump -h \"https://opendap.earthdata.nasa.gov/providers/POCLOUD/collections/GHRSST%20Level%204%20MUR%20Global%20Foundation%20Sea%20Surface%20Temperature%20Analysis%20(v4.1)/granules/20190101090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1\"","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"This should return the metadata of the OPeNDAP resource:","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"netcdf \\20190101090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04 {\ndimensions:\n\tlat = 17999 ;\n\tlon = 36000 ;\n\ttime = 1 ;\nvariables:\n\tshort analysed_sst(time, lat, lon) ;\n[...]","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"This is the typical error message which is returned when the credentials are not configured properly:","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"syntax error, unexpected WORD_WORD, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR\ncontext: HTTP^ Basic: Access denied.","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"When there is an error on the server side (500 internal server error), you might get the following error:","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"syntax error, unexpected $end, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR\ncontext: ^\nncdump: https://opendap.earthdata.nasa.gov/providers/POCLOUD/collections/GHRSST%20Level%204%20MUR%20Global%20Foundation%20Sea%20Surface%20Temperature%20Analysis%20(v4.1)/granules/20190101090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1: NetCDF: Malformed or inaccessible DAP2 DDS or DAP4 DMR response","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"Here we use the GHRSST Level 4 MUR Global Foundation Sea Surface Temperature Analysis (v4.1) dataset. In the following example, we download the data via OPeNDAP for a chosen bounding box and given time instance.","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"using NCDatasets, PyPlot, Dates, Statistics\n\nurl = \"https://opendap.earthdata.nasa.gov/providers/POCLOUD/collections/GHRSST%20Level%204%20MUR%20Global%20Foundation%20Sea%20Surface%20Temperature%20Analysis%20(v4.1)/granules/20190101090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1\"\n\nds = NCDataset(url)\n\n# range of longitude\nlonr = (-6, 37.0)\n\n# range of latitude\nlatr = (29, 45.875)\n\nds_subset = NCDatasets.@select(\n ds[\"analysed_sst\"],\n $lonr[1] <= lon <= $lonr[2] && $latr[1] <= lat <= $latr[2])\n\nncvar = ds_subset[\"analysed_sst\"]\nSST = ncvar[:,:,1]\nlon = ds_subset[\"lon\"][:]\nlat = ds_subset[\"lat\"][:]\ntime = ds_subset[\"time\"][1]\n\n\nclf()\npcolormesh(lon,lat,nomissing(SST,NaN)');\ngca().set_aspect(1/cosd(mean(lat)))\n\ncbar = colorbar(orientation=\"horizontal\")\ncbar.set_label(ncvar.attrib[\"units\"])\n\nplt.title(\"$(ncvar.attrib[\"long_name\"]) $time\")","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"This script produces the following plot:","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"(Image: example_SST.png)","category":"page"},{"location":"tutorials/#Access-data-from-S3-object-storage","page":"Tutorials","title":"Access data from S3 object storage","text":"","category":"section"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"NASA EarthData is also available from AWS S3 object storage at the AWS region us-west-2. This example assumes that you have access to an AWS instance in this region (available through e.g. Pangeo Cloud after registration thanks to funding from the NASA's ACCESS/AWS Cloud Credits for Research program and the fine folks at Pangeo).","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"The S3 access tokens available from https://archive.podaac.earthdata.nasa.gov/s3credentials are only valid for 1 hour. It is therefore preferable to get this token programmatically using your EarthData username and password. The function earthdata_s3credentials is a translation of \"Sample script to receive credentials\" to Julia:","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"using AWS, Base64, JSON3, HTTP, NCDatasets, URIs\nusing AWS: @service\n@service S3\n\nfunction earthdata_s3credentials(\n username, password;\n credentials_url = \"https://archive.podaac.earthdata.nasa.gov/s3credentials\")\n\n origin = URI(credentials_url).scheme\n\n resp = HTTP.get(credentials_url,redirect=false)\n authorize_url = Dict(resp.headers)[\"Location\"]\n\n auth = base64encode(string(username,\":\",password))\n\n resp2 = HTTP.post(authorize_url,[\"Origin\" => origin],\n \"credentials=$auth\",redirect=false)\n redirect_url=Dict(resp2.headers)[\"Location\"]\n\n # cookie jar\n jar = Dict{String, Set{HTTP.Cookie}}()\n\n # raises a 500 error, as in the shell script, but we get a cookie!\n HTTP.get(redirect_url; cookies=true,\n cookiejar = jar, redirect = false,\n status_exception = false)\n\n resp4 = HTTP.get(credentials_url; cookies=true, cookiejar = jar, redirect = false)\n\n cred = JSON3.read(resp4.body)\n return cred\nend\n\n# add your credentials here (or get it programmatically from environment variables or a file)\nusername = \"...\"\npassword = \"...\"\n\ncred = earthdata_s3credentials(username,password)\n@info \"Token expires: $(cred.expiration)\"\n\n\nENV[\"AWS_ACCESS_KEY_ID\"] = cred.accessKeyId;\nENV[\"AWS_SECRET_ACCESS_KEY\"] = cred.secretAccessKey;\nENV[\"AWS_SESSION_TOKEN\"] = cred.sessionToken;\n\nc = AWS.global_aws_config();\n\nresp = S3.list_objects(\"podaac-ops-cumulus-protected\",\n Dict(\"prefix\" => \"MODIS_TERRA_L3_SST_MID-IR_DAILY_4KM_NIGHTTIME_V2019.0/\",\n \"delimiter\" => '/'))\n\n# download the first object\ndata = S3.get_object(\"podaac-ops-cumulus-protected\",resp[\"Contents\"][1][\"Key\"]);\n\n# load the NetCDF dataset\nds = NCDataset(\"temp-memory\",\"r\",memory = data)","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"Output:","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"CDataset: temp-memory\nGroup: /\n\nDimensions\n lat = 4320\n lon = 8640\n rgb = 3\n eightbitcolor = 256\n\nVariables\n sst4 (8640 × 4320)\n Datatype: Int16\n Dimensions: lon × lat\n Attributes:\n long_name = 4um Sea Surface Temperature\n[...]","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"The example requires NCDatasets 0.12.5 which allows one to read a NetCDF dataset directly from a vector of bytes in memory.","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"To debug, it is useful to run the aws shell command to list all keys in the buckets (it requires the AWS_* environment variables to be set):","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"aws s3 ls s3://podaac-ops-cumulus-protected/MODIS_TERRA_L3_SST_THERMAL_DAILY_4KM_NIGHTTIME_V2019.0/","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"[1]: Windows users need to create a .dodsrc configuration file (instead of the .ncrc file) and place it in the current working directory or set the HOME environment variable (see https://github.com/Unidata/netcdf-c/issues/2380). This NetCDF bug is likely to be fixed in NetCDF version 4.9.1.","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"[2]: More information is available at https://docs.unidata.ucar.edu/netcdf-c/4.8.1/md_auth.html.","category":"page"},{"location":"variables/#Variables","page":"Variables","title":"Variables","text":"","category":"section"},{"location":"variables/","page":"Variables","title":"Variables","text":"Variables (like e.g. CFVariable) are the quantities contained within a NetCDF dataset. See the Datasets page on how to obtain them from a dataset.","category":"page"},{"location":"variables/","page":"Variables","title":"Variables","text":"Different type of arrays are involved when working with NCDatasets. For instance assume that test.nc is a file with a Float32 variable called variable.","category":"page"},{"location":"variables/","page":"Variables","title":"Variables","text":"using NCDatasets\nds = NCDataset(\"test.nc\")\nncvar_cf = ds[\"variable\"]","category":"page"},{"location":"variables/","page":"Variables","title":"Variables","text":"The variable ncvar_cf has the type CFVariable. No data is actually loaded from disk, but you can query its size, number of dimensions, number elements, etc., using the functions size, ndims, length as if ncvar_cf was an ordinary Julia array.","category":"page"},{"location":"variables/","page":"Variables","title":"Variables","text":"To load the variable ncvar_cf in memory you can convert it into an array with:","category":"page"},{"location":"variables/","page":"Variables","title":"Variables","text":"data = Array(ncvar_cf)\n# or\ndata = ncvar_cf |> Array\n# or if ndims(ncvar_cf) == 2\ndata = ncvar_cf[:,:]","category":"page"},{"location":"variables/","page":"Variables","title":"Variables","text":"Since NCDatasets 0.13, the syntax ncvar_cf[:] flattens the array, and is not equivalent with the above (unless ncvar_cf is a vector).","category":"page"},{"location":"variables/","page":"Variables","title":"Variables","text":"You can only load sub-parts of it in memory via indexing each dimension:","category":"page"},{"location":"variables/","page":"Variables","title":"Variables","text":"ncvar_cf[1:5, 10:20]","category":"page"},{"location":"variables/","page":"Variables","title":"Variables","text":"A scalar variable can be loaded using [], for example:","category":"page"},{"location":"variables/","page":"Variables","title":"Variables","text":"using NCDatasets\nNCDataset(\"test_scalar.nc\",\"c\") do ds\n defVar(ds,\"scalar\",42,())\nend\n\nds = NCDataset(\"test_scalar.nc\")\nvalue = ds[\"scalar\"][] # 42","category":"page"},{"location":"variables/","page":"Variables","title":"Variables","text":"To load all a variable in a NetCDF file ignoring attributes like scale_factor, add_offset, _FillValue and time units one can use the property var or the function variable for example:","category":"page"},{"location":"variables/","page":"Variables","title":"Variables","text":"using NCDatasets\nusing Dates\ndata = [DateTime(2000,1,1), DateTime(2000,1,2)]\nNCDataset(\"test_file.nc\",\"c\") do ds\n defVar(ds,\"time\",data,(\"time\",), attrib = Dict(\n \"units\" => \"days since 2000-01-01\"))\nend;\n\nds = NCDataset(\"test_file.nc\")\nncvar = ds[\"time\"].var\n# or\nncvar = variable(ds,\"time\")\ndata = ncvar[:] # here [0., 1.]","category":"page"},{"location":"variables/","page":"Variables","title":"Variables","text":"The variable ncvar can be indexed in the same way as ncvar_cf explained above.","category":"page"},{"location":"variables/","page":"Variables","title":"Variables","text":"note: Note\nNCDatasets.Variable and NCDatasets.CFVariable implement the interface of AbstractArray. It is thus possible to call any function that accepts an AbstractArray. But functions like mean, sum (and many more) would load every element individually which is very inefficient for large fields read from disk. You should instead convert such a variable to a standard Julia Array and then do computations with it. See also the performance tips for more information.","category":"page"},{"location":"variables/","page":"Variables","title":"Variables","text":"The following functions are convenient for working with variables:","category":"page"},{"location":"variables/","page":"Variables","title":"Variables","text":"Base.size(v::NCDatasets.CFVariable)\ndimnames\ndimsize\nname\nrenameVar\nNCDataset(var::NCDatasets.CFVariable)\nnomissing\nfillvalue","category":"page"},{"location":"variables/#Base.size-Tuple{CommonDataModel.CFVariable}","page":"Variables","title":"Base.size","text":"sz = size(var::CFVariable)\n\nReturn a tuple of integers with the size of the variable var.\n\nnote: Note\nNote that the size of a variable can change, i.e. for a variable with an unlimited dimension.\n\n\n\n\n\n","category":"method"},{"location":"variables/#CommonDataModel.dimnames","page":"Variables","title":"CommonDataModel.dimnames","text":"dimnames(v::Variable)\n\nReturn a tuple of strings with the dimension names of the variable v.\n\n\n\n\n\ndimnames(v::CFVariable)\n\nReturn a tuple of strings with the dimension names of the variable v.\n\n\n\n\n\nCommonDataModel.dimnames(v::AbstractVariable)\n\nReturn an iterable of the dimension names of the variable v.\n\n\n\n\n\nCommonDatamodel.dimnames(ds::AbstractDataset)\n\nReturn an iterable of all dimension names in ds.\n\n\n\n\n\n","category":"function"},{"location":"variables/#NCDatasets.dimsize","page":"Variables","title":"NCDatasets.dimsize","text":"dimsize(v::CFVariable)\n\nGet the size of a CFVariable as a named tuple of dimension → length.\n\n\n\n\n\n","category":"function"},{"location":"variables/#CommonDataModel.name","page":"Variables","title":"CommonDataModel.name","text":"name(ds::NCDataset)\n\nReturn the group name of the NCDataset ds\n\n\n\n\n\nname(v::Variable)\n\nReturn the name of the NetCDF variable v.\n\n\n\n\n\nCommonDatamodel.name(ds::AbstractDataset)\n\nName of the group of the data set ds. For a data set containing only a single group, this will be always the root group \"/\".\n\n\n\n\n\nCommonDataModel.name(v::AbstractVariable)\n\nReturn the name of the variable v as a string.\n\n\n\n\n\n","category":"function"},{"location":"variables/#NCDatasets.renameVar","page":"Variables","title":"NCDatasets.renameVar","text":"renameVar(ds::NCDataset,oldname,newname)\n\nRename the variable called oldname to newname.\n\n\n\n\n\n","category":"function"},{"location":"variables/#NCDatasets.NCDataset-Tuple{CommonDataModel.CFVariable}","page":"Variables","title":"NCDatasets.NCDataset","text":"mfds = NCDataset(fnames, mode = \"r\"; aggdim = nothing, deferopen = true,\n isnewdim = false,\n constvars = [])\n\nOpens a multi-file dataset in read-only \"r\" or append mode \"a\". fnames is a vector of file names.\n\nVariables are aggregated over the first unlimited dimension or over the dimension aggdim if specified. Variables without the dimensions aggdim are not aggregated. All variables containing the dimension aggdim are aggregated. The variable who do not contain the dimension aggdim are assumed constant.\n\nIf variables should be aggregated over a new dimension (not present in the NetCDF file), one should set isnewdim to true. All NetCDF files should have the same variables, attributes and groupes. Per default, all variables will have an additional dimension unless they are marked as constant using the constvars parameter.\n\nThe append mode is only implemented when deferopen is false. If deferopen is false, all files are opened at the same time. However the operating system might limit the number of open files. In Linux, the limit can be controled with the command ulimit.\n\nAll metadata (attributes and dimension length are assumed to be the same for all NetCDF files. Otherwise reading the attribute of a multi-file dataset would be ambiguous. An exception to this rule is the length of the dimension over which the data is aggregated. This aggregation dimension can varify from file to file.\n\nSetting the experimental flag _aggdimconstant to true means that the length of the aggregation dimension is constant. This speeds up the creating of a multi-file dataset as only the metadata of the first file has to be loaded.\n\nExamples:\n\nYou can use Glob.jl to make fnames from a file pattern, e.g.\n\nusing NCDatasets, Glob\nds = NCDataset(glob(\"ERA5_monthly3D_reanalysis_*.nc\"))\n\nAggregation over a new dimension:\n\nusing NCDatasets\nfor i = 1:3\n NCDataset(\"foo$i.nc\",\"c\") do ds\n defVar(ds,\"data\",[10., 11., 12., 13.], (\"lon\",))\n end\nend\n\nds = NCDataset([\"foo$i.nc\" for i = 1:3],aggdim = \"sample\", isnewdim = true)\nsize(ds[\"data\"])\n# output\n# (4, 3)\n\n\n\n\n\n","category":"method"},{"location":"variables/#NCDatasets.nomissing","page":"Variables","title":"NCDatasets.nomissing","text":"a = nomissing(da)\n\nReturn the values of the array da of type Array{Union{T,Missing},N} (potentially containing missing values) as a regular Julia array a of the same element type. It raises an error if the array contains at least one missing value.\n\n\n\n\n\na = nomissing(da,value)\n\nRetun the values of the array da of type AbstractArray{Union{T,Missing},N} as a regular Julia array a by replacing all missing value by value (converted to type T). This function is identical to coalesce.(da,T(value)) where T is the element type of da.\n\nExample:\n\njulia> nomissing([missing,1.,2.],NaN)\n# returns [NaN, 1.0, 2.0]\n\n\n\n\n\n","category":"function"},{"location":"variables/#CommonDataModel.fillvalue","page":"Variables","title":"CommonDataModel.fillvalue","text":"fillvalue(::Type{Int8})\nfillvalue(::Type{UInt8})\nfillvalue(::Type{Int16})\nfillvalue(::Type{UInt16})\nfillvalue(::Type{Int32})\nfillvalue(::Type{UInt32})\nfillvalue(::Type{Int64})\nfillvalue(::Type{UInt64})\nfillvalue(::Type{Float32})\nfillvalue(::Type{Float64})\nfillvalue(::Type{Char})\nfillvalue(::Type{String})\n\nDefault fill-value for the given type.\n\n\n\n\n\nfv = fillvalue(v::Variable)\nfv = fillvalue(v::CFVariable)\n\nReturn the fill-value of the variable v.\n\n\n\n\n\n","category":"function"},{"location":"variables/","page":"Variables","title":"Variables","text":"loadragged\nNCDatasets.load!","category":"page"},{"location":"variables/#NCDatasets.loadragged","page":"Variables","title":"NCDatasets.loadragged","text":" data = loadragged(ncvar,index::Union{Colon,UnitRange,Integer})\n\nLoad data from ncvar in the contiguous ragged array representation as a vector of vectors. It is typically used to load a list of profiles or time series of different length each.\n\nThe indexed ragged array representation is currently not supported.\n\n\n\n\n\n","category":"function"},{"location":"variables/#NCDatasets.load!","page":"Variables","title":"NCDatasets.load!","text":"NCDatasets.load!(ncvar::Variable, data, indices)\n\nLoads a NetCDF variables ncvar in-place and puts the result in data along the specified indices. One can use @inbounds annotate code where bounds checking can be elided by the compiler (which typically require type-stable code).\n\nusing NCDatasets\nds = NCDataset(\"file.nc\")\nncv = ds[\"vgos\"].var;\n# data must have the right shape and type\ndata = zeros(eltype(ncv),size(ncv));\nNCDatasets.load!(ncv,data,:,:,:)\n# or\n# @inbounds NCDatasets.load!(ncv,data,:,:,:)\nclose(ds)\n\n# loading a subset\ndata = zeros(5); # must have the right shape and type\nload!(ds[\"temp\"].var,data,:,1) # loads the 1st column\n\nnote: Note\nFor a netCDF variable of type NC_CHAR, the element type of the data array must be UInt8 and cannot be the julia Char type, because the julia Char type uses 4 bytes and the NetCDF NC_CHAR only 1 byte.\n\n\n\n\n\nNCDatasets.load!(ncvar::CFVariable, data, buffer, indices)\n\nLoads a NetCDF variables ncvar in-place and puts the result in data (an array of eltype(ncvar)) along the specified indices. buffer is a temporary array of the same size as data but the type should be eltype(ncv.var), i.e. the corresponding type in the NetCDF files (before applying scale_factor, add_offset and masking fill values). Scaling and masking will be applied to the array data.\n\ndata and buffer can be the same array if eltype(ncvar) == eltype(ncvar.var).\n\nExample:\n\n# create some test array\nDataset(\"file.nc\",\"c\") do ds\n defDim(ds,\"time\",3)\n ncvar = defVar(ds,\"vgos\",Int16,(\"time\",),attrib = [\"scale_factor\" => 0.1])\n ncvar[:] = [1.1, 1.2, 1.3]\n # store 11, 12 and 13 as scale_factor is 0.1\nend\n\n\nds = Dataset(\"file.nc\")\nncv = ds[\"vgos\"];\n# data and buffer must have the right shape and type\ndata = zeros(eltype(ncv),size(ncv)); # here Vector{Float64}\nbuffer = zeros(eltype(ncv.var),size(ncv)); # here Vector{Int16}\nNCDatasets.load!(ncv,data,buffer,:,:,:)\nclose(ds)\n\n\n\n\n\n","category":"function"},{"location":"variables/#Creating-a-variable","page":"Variables","title":"Creating a variable","text":"","category":"section"},{"location":"variables/","page":"Variables","title":"Variables","text":"defVar","category":"page"},{"location":"variables/#CommonDataModel.defVar","page":"Variables","title":"CommonDataModel.defVar","text":"defVar(ds::NCDataset,name,vtype,dimnames; kwargs...)\ndefVar(ds::NCDataset,name,data,dimnames; kwargs...)\n\nDefine a variable with the name name in the dataset ds. vtype can be Julia types in the table below (with the corresponding NetCDF type). The parameter dimnames is a tuple with the names of the dimension. For scalar this parameter is the empty tuple (). The variable is returned (of the type CFVariable).\n\nInstead of providing the variable type one can directly give also the data data which will be used to fill the NetCDF variable. In this case, the dimensions with the appropriate size will be created as required using the names in dimnames.\n\nIf data is a vector or array of DateTime objects, then the dates are saved as double-precision floats and units \"days since 1900-01-01 00:00:00\" (unless a time unit is specifed with the attrib keyword as described below). Dates are converted to the default calendar in the CF conversion which is the mixed Julian/Gregorian calendar.\n\nKeyword arguments\n\nfillvalue: A value filled in the NetCDF file to indicate missing data. It will be stored in the _FillValue attribute.\nchunksizes: Vector integers setting the chunk size. The total size of a chunk must be less than 4 GiB.\ndeflatelevel: Compression level: 0 (default) means no compression and 9 means maximum compression. Each chunk will be compressed individually.\nshuffle: If true, the shuffle filter is activated which can improve the compression ratio.\nchecksum: The checksum method can be :fletcher32 or :nochecksum (checksumming is disabled, which is the default)\nattrib: An iterable of attribute name and attribute value pairs, for example a Dict, DataStructures.OrderedDict or simply a vector of pairs (see example below)\ntypename (string): The name of the NetCDF type required for vlen arrays\n\nchunksizes, deflatelevel, shuffle and checksum can only be set on NetCDF 4 files. Compression of strings and variable-length arrays is not supported by the underlying NetCDF library.\n\nNetCDF data types\n\nNetCDF Type Julia Type\nNC_BYTE Int8\nNC_UBYTE UInt8\nNC_SHORT Int16\nNC_INT Int32\nNC_INT64 Int64\nNC_FLOAT Float32\nNC_DOUBLE Float64\nNC_CHAR Char\nNC_STRING String\n\nDimension ordering\n\nThe data is stored in the NetCDF file in the same order as they are stored in memory. As julia uses the Column-major ordering for arrays, the order of dimensions will appear reversed when the data is loaded in languages or programs using Row-major ordering such as C/C++, Python/NumPy or the tools ncdump/ncgen (NetCDF CDL). NumPy can also use Column-major ordering but Row-major order is the default. For the column-major interpretation of the dimensions (as in Julia), the CF Convention recommends the order \"longitude\" (X), \"latitude\" (Y), \"height or depth\" (Z) and \"date or time\" (T) (if applicable). All other dimensions should, whenever possible, be placed to the right of the spatiotemporal dimensions.\n\nExample:\n\nIn this example, scale_factor and add_offset are applied when the data is saved.\n\njulia> using DataStructures\njulia> data = randn(3,5)\njulia> NCDataset(\"test_file.nc\",\"c\") do ds\n defVar(ds,\"temp\",data,(\"lon\",\"lat\"), attrib = OrderedDict(\n \"units\" => \"degree_Celsius\",\n \"add_offset\" => -273.15,\n \"scale_factor\" => 0.1,\n \"long_name\" => \"Temperature\"\n ))\n end;\n\nnote: Note\nIf the attributes _FillValue, missing_value, add_offset, scale_factor, units and calendar are used, they should be defined when calling defVar by using the parameter attrib as shown in the example above.\n\n\n\n\n\nv = CommonDataModel.defVar(ds::AbstractDataset,src::AbstractVariable)\n\nDefines and return the variable in the data set ds copied from the variable src. The variable name, dimension name, attributes and data are copied from src.\n\n\n\n\n\n","category":"function"},{"location":"variables/#Storage-parameter-of-a-variable","page":"Variables","title":"Storage parameter of a variable","text":"","category":"section"},{"location":"variables/","page":"Variables","title":"Variables","text":"chunking\ndeflate\nchecksum","category":"page"},{"location":"variables/#NCDatasets.chunking","page":"Variables","title":"NCDatasets.chunking","text":"storage,chunksizes = chunking(v::Variable)\n\nReturn the storage type (:contiguous or :chunked) and the chunk sizes of the varable v. Note that chunking reports the same information as nc_inq_var_chunking and therefore considers variables with unlimited dimension as :contiguous.\n\n\n\n\n\nstorage,chunksizes = chunking(v::MFVariable)\n\nReturn the storage type (:contiguous or :chunked) and the chunk sizes of the varable v corresponding to the first NetCDF file. If the first NetCDF file in the collection is chunked then this storage attributes are returns. If not the first NetCDF file is not contiguous, then multi-file variable is still reported as chunked with chunk size equal to the variable size.\n\n\n\n\n\n","category":"function"},{"location":"variables/#NCDatasets.deflate","page":"Variables","title":"NCDatasets.deflate","text":"isshuffled,isdeflated,deflate_level = deflate(v::Variable)\n\nReturn compression information of the variable v. If shuffle is true, then shuffling (byte interlacing) is activated. If deflate is true, then the data chunks (see chunking) are compressed using the compression level deflate_level (0 means no compression and 9 means maximum compression).\n\n\n\n\n\n","category":"function"},{"location":"variables/#NCDatasets.checksum","page":"Variables","title":"NCDatasets.checksum","text":"checksummethod = checksum(v::Variable)\n\nReturn the checksum method of the variable v which can be either be :fletcher32 or :nochecksum.\n\n\n\n\n\n","category":"function"},{"location":"variables/#Coordinate-variables-and-cell-boundaries","page":"Variables","title":"Coordinate variables and cell boundaries","text":"","category":"section"},{"location":"variables/","page":"Variables","title":"Variables","text":"coord\nbounds","category":"page"},{"location":"variables/#NCDatasets.coord","page":"Variables","title":"NCDatasets.coord","text":"cv = coord(v::Union{CFVariable,Variable},standard_name)\n\nFind the coordinate of the variable v by the standard name standard_name or some standardized heuristics based on units. If the heuristics fail to detect the coordinate, consider to modify the netCDF file to add the standard_name attribute. All dimensions of the coordinate must also be dimensions of the variable v.\n\nExample\n\nusing NCDatasets\nds = NCDataset(\"file.nc\")\nncv = ds[\"SST\"]\nlon = coord(ncv,\"longitude\")[:]\nlat = coord(ncv,\"latitude\")[:]\nv = ncv[:]\nclose(ds)\n\n\n\n\n\n","category":"function"},{"location":"variables/#NCDatasets.bounds","page":"Variables","title":"NCDatasets.bounds","text":"b = bounds(ncvar::NCDatasets.CFVariable)\n\nReturn the CFVariable corresponding to the bounds attribute of the variable ncvar. The time units and calendar from the ncvar are used but not the attributes controling the packing of data scale_factor, add_offset and _FillValue.\n\n\n\n\n\n","category":"function"},{"location":"issues/#Known-issues","page":"Known issues","title":"Known issues","text":"","category":"section"},{"location":"issues/#NetCDF:-Not-a-valid-data-type-or-_FillValue-type-mismatch","page":"Known issues","title":"NetCDF: Not a valid data type or _FillValue type mismatch","text":"","category":"section"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"Trying to define the _FillValue, produces the following error:","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"ERROR: LoadError: NCDatasets.NetCDFError(-45, \"NetCDF: Not a valid data type or _FillValue type mismatch\")","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"The error could be generated by a code like this:","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"using NCDatasets, DataStructures\n# ...\ntempvar = defVar(ds,\"temp\",Float32,(\"lonc\",\"latc\",\"time\"), attrib = OrderedDict(\n \"_FillValue\" => -9999.))","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"or","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"using NCDatasets\n# ...\ntempvar = defVar(ds,\"temp\",Float32,(\"lonc\",\"latc\",\"time\"), fillvalue = -9999.)","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"In fact, _FillValue must have the same data type as the corresponding variable. In the case above, tempvar is a 32-bit float and the number -9999. is a 64-bit float (aka double, which is the default floating point type in Julia). It is sufficient to convert the value -9999. to a 32-bit float -9999.f0 (or Float32(-9999.)).","category":"page"},{"location":"issues/#Defining-the-attributes-_FillValue,-add_offset,-scale_factor,-units-and-calendar","page":"Known issues","title":"Defining the attributes _FillValue, add_offset, scale_factor, units and calendar","text":"","category":"section"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"An error like Cannotconvertan object of type Missing (or similar) is generated by code like this:","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"v = defVar(ds,\"var_with_all_missing_data\",Float32,(\"lon\",))\nv.attrib[\"_FillValue\"] = fv\nv[1] = missing","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"This produces the following error:","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"ERROR: LoadError: MethodError: Cannot `convert` an object of type Missing to an object of type Float32\nClosest candidates are:\n convert(::Type{T}, ::T) where T<:Number at number.jl:6\n convert(::Type{T}, ::Number) where T<:Number at number.jl:7\n convert(::Type{T}, ::Base.TwicePrecision) where T<:Number at twiceprecision.jl:250\n ...\nStacktrace:\n [1] fill!(::SubArray{Float32,1,NCDatasets.CFVariable{Float32,1,NCDatasets.Variable{Float32,1},NCDatasets.Attributes},Tuple{UnitRange{Int64}},false}, ::Missing) at ./multidimensional.jl:865\n [2] copyto! at ./broadcast.jl:871 [inlined]\n [3] materialize!(::SubArray{Float32,1,NCDatasets.CFVariable{Float32,1,NCDatasets.Variable{Float32,1},NCDatasets.Attributes},Tuple{UnitRange{Int64}},false}, ::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{0},Nothing,typeof(identity),Tuple{Base.RefValue{Missing}}}) at ./broadcast.jl:822","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"One should use define the _FillValue (and similar attributes like add_offset, scale_factor, units and calendar affecting the type of the data) in the call of defVar:","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"using DataStructures\nv = defVar(ds,\"var_with_all_missing_data\",Int32,(\"lon\",), fillvalue = fv, attrib = OrderedDict(\n \"scale_factor\" => 0.1,\n \"add_offset\" => 0.1\n ))","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"This change was introduced in NCDatasets version 0.10","category":"page"},{"location":"issues/#Multiple-versions-of-HDF5-or-NetCDF-libraries","page":"Known issues","title":"Multiple versions of HDF5 or NetCDF libraries","text":"","category":"section"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"Having outdated versions of HDF5 or NetCDF libraries installed can be an issue on Windows if they are included in the system PATH environment variable. It is advised to adapt the system PATH to remove the locations containing these libraries.","category":"page"},{"location":"issues/#Using-a-custom-NetCDF-library","page":"Known issues","title":"Using a custom NetCDF library","text":"","category":"section"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"The NetCDF library libnetcdf.so is installed as an artifact via the package NetCDF_jll. You can override which libnetcdf.so gets loaded through the Preferences package, as follows:","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"# install these packages if necessary\nusing Preferences, NetCDF_jll\n\nset_preferences!(NetCDF_jll, \"libnetcdf_path\" => \"/path/to/libnetcdf.so.xyz\")","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"where /path/to/libnetcdf.so.xyz is the full path to the NetCDF library. This will create a LocalPreferences.toml file in your top-level project with the following content:","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"[NetCDF_jll]\nlibnetcdf_path = \"/path/to/libnetcdf.so.xyz\"","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"However, the dependencies of the library version libnetcdf.so.xyz (in particular libcurl.so and libmbedtls.so) should be compatible with the dependencies of julia (in the folder .../julia-x.y.z/lib/julia). On Linux, you can list the library dependencies with the shell command ldd, for example:","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"ldd /path/to/libnetcdf.so.xyz","category":"page"},{"location":"issues/#OPeNDAP-on-Windows-fails-with-Assertion-failed:-ocpanic","page":"Known issues","title":"OPeNDAP on Windows fails with Assertion failed: ocpanic","text":"","category":"section"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"On windows, NetCDF 4.7.4 can fail with this error:","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"Assertion failed: ocpanic((\"state->auth.curlflags.cookiejar != NULL\")), file ocinternal.c, line 566","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"when accessing OPeNDAP URLs, like these:","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"using NCDatasets\nds = NCDataset(\"https://thredds.jpl.nasa.gov/thredds/dodsC/ncml_aggregation/OceanTemperature/modis/terra/11um/4km/aggregate__MODIS_TERRA_L3_SST_THERMAL_DAILY_4KM_DAYTIME_V2019.0.ncml#fillmismatch\")","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"See also the issue report: https://github.com/Unidata/netcdf-c/issues/2380. The work-around is to create a .dodsrc in the current working directory with the content:","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"HTTP.COOKIEJAR=C:\\Users\\USERNAME\\AppData\\Local\\Temp\\","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"where USERNAME is your username. The directory should exist and be writable by the user. You can run pwd() to determine the current working directory. Note that the initial current working directory can be different depending you how you start julia (from the command line or from jupyter notebook for example). Julia need to be restarted after this file is placed in the your working directory.","category":"page"},{"location":"issues/#Using-non-official-julia-builds","page":"Known issues","title":"Using non-official julia builds","text":"","category":"section"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"Julia and NetCDF_jll have several common dependencies (curl, MbedTLS, zlib). Non-official julia builds will work only if they use exactly the same library version as those used to compile NetCDF. This is unlikely to be the case in general and outside of our control. Therefore non-official julia builds are not supported. Official julia builds are available at https://julialang.org/downloads/.","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"Using for example the julia package from on Arch Linux, leads to this error:","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"julia> using NCDatasets\nERROR: LoadError: InitError: could not load library \"/root/.julia/artifacts/461703969206dd426cc6b4d99f69f6ffab2a9779/lib/libnetcdf.so\"\n/usr/lib/julia/libcurl.so: version `CURL_4' not found (required by /root/.julia/artifacts/461703969206dd426cc6b4d99f69f6ffab2a9779/lib/libnetcdf.so)\nStacktrace:\n [1] macro expansion\n @ ~/.julia/packages/JLLWrappers/QpMQW/src/products/library_generators.jl:54 [inlined]\n [2] __init__()\n @ NetCDF_jll ~/.julia/packages/NetCDF_jll/BYHmI/src/wrappers/x86_64-linux-gnu.jl:12\n [3] top-level scope\n @ stdin:1\nduring initialization of module NetCDF_jll","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"You will likely have similar issues with julia installed from other package managers (like Debian/Ubuntu apt, Homebrew...). The only supported solution is to install the offical julia builds.","category":"page"},{"location":"issues/#version-CURL_4'-not-found","page":"Known issues","title":"version `CURL_4' not found","text":"","category":"section"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"If you see the following error:","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"ERROR: LoadError: InitError: could not load library \"/home/user/.julia/artifacts/461703969206dd426cc6b4d99f69f6ffab2a9779/lib/libnetcdf.so\"\n/usr/lib/x86_64-linux-gnu/libcurl.so: version `CURL_4' not found (required by /home/user/.julia/artifacts/461703969206dd426cc6b4d99f69f6ffab2a9779/lib/libnetcdf.so)","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"make sure that you are using the offical julia builds (see above) and that your LD_LIBRARY_PATH and LD_PRELOAD are empty. You can verify this by running the following commands in a terminal:","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"echo $LD_PRELOAD\necho $LD_LIBRARY_PATH","category":"page"},{"location":"issues/#Corner-cases","page":"Known issues","title":"Corner cases","text":"","category":"section"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"An attribute representing a vector with a single value (e.g. [1]) will be read back as a scalar (1) (same behavior in python netCDF4 1.3.1).\nNetCDF and Julia distinguishes between a vector of chars and a string, but both are returned as string for ease of use, in particular an attribute representing a vector of chars ['u','n','i','t','s'] will be read back as the string \"units\".\nWhile reading a NetCDF time variable, the dates are converted using the Julia's DateTime (based on the proleptic Gregorian calendar following the ISO 8601 standard) when possible. When data is written to a NetCDF file (without specifying the calendar), the dates are saved using the default calendar of the NetCDF CF convention (the mixed Julian/Gregorian calendar, called \"standard\") when possible. It is recommended that the time origin specified by the units is after 15 October 1582 in which case the mixed Julian/Gregorian calendar is identical to the proleptic Gregorian calendar.","category":"page"},{"location":"attributes/#Attributes","page":"Attributes","title":"Attributes","text":"","category":"section"},{"location":"attributes/","page":"Attributes","title":"Attributes","text":"The NetCDF dataset (as returned by NCDataset or NetCDF groups) and the NetCDF variables (as returned by getindex, variable or defVar) have the field attrib which has the type NCDatasets.Attributes and behaves like a julia dictionary.","category":"page"},{"location":"attributes/","page":"Attributes","title":"Attributes","text":"getindex(a::NCDatasets.Attributes,name::AbstractString)\nsetindex!(a::NCDatasets.Attributes,data,name::AbstractString)\nkeys(a::NCDatasets.Attributes)\ndelete!(a::NCDatasets.Attributes,name::AbstractString)","category":"page"},{"location":"attributes/#Base.getindex-Tuple{NCDatasets.Attributes, AbstractString}","page":"Attributes","title":"Base.getindex","text":"getindex(a::Attributes,name::SymbolOrString)\n\nReturn the value of the attribute called name from the attribute list a. Generally the attributes are loaded by indexing, for example:\n\nds = NCDataset(\"file.nc\")\ntitle = ds.attrib[\"title\"]\n\n\n\n\n\n","category":"method"},{"location":"attributes/#Base.setindex!-Tuple{NCDatasets.Attributes, Any, AbstractString}","page":"Attributes","title":"Base.setindex!","text":"Base.setindex!(a::Attributes,data,name::SymbolOrString)\n\nSet the attribute called name to the value data in the attribute list a. data can be a vector or a scalar. A scalar is handeld as a vector with one element in the NetCDF data model.\n\nGenerally the attributes are defined by indexing, for example:\n\nds = NCDataset(\"file.nc\",\"c\")\nds.attrib[\"title\"] = \"my title\"\nclose(ds)\n\nIf data is a string, then attribute is saved as a list of NetCDF characters (NC_CHAR) with the appropriate length. To save the attribute as a string (NC_STRING) you can use the following:\n\nds = NCDataset(\"file.nc\",\"c\")\nds.attrib[\"title\"] = [\"my title\"]\nclose(ds)\n\n\n\n\n\n","category":"method"},{"location":"attributes/#Base.keys-Tuple{NCDatasets.Attributes}","page":"Attributes","title":"Base.keys","text":"Base.keys(a::Attributes)\n\nReturn a list of the names of all attributes.\n\n\n\n\n\n","category":"method"},{"location":"attributes/#Base.delete!-Tuple{NCDatasets.Attributes, AbstractString}","page":"Attributes","title":"Base.delete!","text":"Base.delete!(a::Attributes, name)\n\nDelete the attribute name from the attribute list a.\n\n\n\n\n\n","category":"method"},{"location":"attributes/","page":"Attributes","title":"Attributes","text":"Loading all attributes as a Dict can be achieved by passing ds.attrib (where ds is the NCDataset) as argument to Dict.","category":"page"},{"location":"attributes/","page":"Attributes","title":"Attributes","text":"using NCDatasets\nncfile = download(\"https://www.unidata.ucar.edu/software/netcdf/examples/sresa1b_ncar_ccsm3-example.nc\");\nds = NCDataset(ncfile);\nattributes_as_dictionary = Dict(ds.attrib)\ntypeof(attributes_as_dictionary)\n# returns Dict{String,Any}","category":"page"},{"location":"attributes/#Possible-type-promotion-in-Julia","page":"Attributes","title":"Possible type promotion in Julia","text":"","category":"section"},{"location":"attributes/","page":"Attributes","title":"Attributes","text":"There is a subtle problem with the following code:","category":"page"},{"location":"attributes/","page":"Attributes","title":"Attributes","text":"ncv1 = defVar(ds,\"v1\", UInt8, (\"longitude\", \"latitude\", \"time\"), attrib = [\n \"add_offset\" => -1.0,\n \"scale_factor\" => 5.0,\n \"_FillValue\" => UInt8(255),\n])","category":"page"},{"location":"attributes/","page":"Attributes","title":"Attributes","text":"Julia effectively promotes the _FillValue to Float64 which leads to a \"NetCDF: Not a valid data type or _FillValue type mismatch\" as the fillvalue has to have exactly the same type as the NetCDF data type. Other parameters could be equally promoted.","category":"page"},{"location":"attributes/","page":"Attributes","title":"Attributes","text":"[\n \"add_offset\" => -1.0,\n \"scale_factor\" => 5.0,\n \"_FillValue\" => UInt8(255),\n]\n# returns\n# 3-element Array{Pair{String,Float64},1}:\n# \"add_offset\" => -1.0\n# \"scale_factor\" => 5.0\n# \"_FillValue\" => 255.0","category":"page"},{"location":"attributes/","page":"Attributes","title":"Attributes","text":"Note the type of the second element of the Pair.","category":"page"},{"location":"attributes/","page":"Attributes","title":"Attributes","text":"Using a Julia Dict does not show this behaviour:","category":"page"},{"location":"attributes/","page":"Attributes","title":"Attributes","text":"ncv1 = defVar(ds,\"v1\", UInt8, (\"longitude\", \"latitude\", \"time\"), attrib = Dict(\n \"add_offset\" => -1.0,\n \"scale_factor\" => 5.0,\n \"_FillValue\" => UInt8(255),\n))","category":"page"},{"location":"attributes/","page":"Attributes","title":"Attributes","text":"Note that Dict does not perserve the order of the attributes. Therefore an OrderedDict from the package DataStructures is preferable.","category":"page"},{"location":"attributes/","page":"Attributes","title":"Attributes","text":"Or one could use simply the fillvalue parameter of defVar.","category":"page"},{"location":"attributes/","page":"Attributes","title":"Attributes","text":"ncv1 = defVar(ds,\"v1\", UInt8, (\"longitude\", \"latitude\", \"time\"), fillvalue = UInt8(255), attrib = [\n \"add_offset\" => -1.0,\n \"scale_factor\" => 5.0,\n])","category":"page"},{"location":"dimensions/#Dimensions","page":"Dimensions","title":"Dimensions","text":"","category":"section"},{"location":"dimensions/","page":"Dimensions","title":"Dimensions","text":"In the NetCDF data model, dimensions have names and a length (but possibly an unlimited length) and are defined for a NetCDF dataset (or group). For a given Variable or CFVariable,the names of the corresponding dimensions are obtained with using dimnames.","category":"page"},{"location":"dimensions/","page":"Dimensions","title":"Dimensions","text":"keys(d::NCDatasets.Dimensions)\nhaskey(a::NCDatasets.NCIterable,name::AbstractString)\ndefDim\nunlimited(d::NCDatasets.Dimensions)\nsetindex!(d::NCDatasets.Dimensions,len,name::AbstractString)","category":"page"},{"location":"dimensions/#Base.keys-Tuple{NCDatasets.Dimensions}","page":"Dimensions","title":"Base.keys","text":"keys(d::Dimensions)\n\nReturn a list of all dimension names in NCDataset ds.\n\nExamples\n\njulia> ds = NCDataset(\"results.nc\", \"r\");\njulia> dimnames = keys(ds.dim)\n\n\n\n\n\n","category":"method"},{"location":"dimensions/#Base.haskey-Tuple{Union{NCDatasets.AbstractDimensions, NCDatasets.AbstractGroups, NCDatasets.AbstractNCDataset, NCDatasets.BaseAttributes}, AbstractString}","page":"Dimensions","title":"Base.haskey","text":"haskey(ds::NCDataset,name)\nhaskey(d::Dimensions,name)\nhaskey(ds::Attributes,name)\n\nReturn true if the NCDataset ds (or dimension/attribute list) has a variable (dimension/attribute) with the name name. For example:\n\nds = NCDataset(\"/tmp/test.nc\",\"r\")\nif haskey(ds,\"temperature\")\n println(\"The file has a variable 'temperature'\")\nend\n\nif haskey(ds.dim,\"lon\")\n println(\"The file has a dimension 'lon'\")\nend\n\nThis example checks if the file /tmp/test.nc has a variable with the name temperature and a dimension with the name lon.\n\n\n\n\n\n","category":"method"},{"location":"dimensions/#CommonDataModel.defDim","page":"Dimensions","title":"CommonDataModel.defDim","text":"defDim(ds::NCDataset,name,len)\n\nDefine a dimension in the data set ds with the given name and length len. If len is the special value Inf, then the dimension is considered as unlimited, i.e. it will grow as data is added to the NetCDF file.\n\nFor example:\n\nusing NCDatasets\nds = NCDataset(\"/tmp/test.nc\",\"c\")\ndefDim(ds,\"lon\",100)\n# [...]\nclose(ds)\n\nThis defines the dimension lon with the size 100.\n\nTo create a variable with an unlimited dimensions use for example:\n\nusing NCDatasets\nds = NCDataset(\"/tmp/test2.nc\",\"c\")\ndefDim(ds,\"lon\",10)\ndefDim(ds,\"lat\",10)\ndefDim(ds,\"time\",Inf)\ndefVar(ds,\"unlimited_variable\",Float64,(\"lon\",\"lat\",\"time\"))\n@show ds.dim[\"time\"]\n# returns 0 as no data is added\nds[\"unlimited_variable\"][:,:,:] = randn(10,10,4)\n@show ds.dim[\"time\"]\n# returns now 4 as 4 time slice have been added\nclose(ds)\n\n\n\n\n\nCommonDatamodel.defDim(ds::AbstractDataset,name::SymbolOrString,len)\n\nCreate dimension with the name name in the data set ds with the length len. len can be Inf for unlimited dimensions.\n\n\n\n\n\n","category":"function"},{"location":"dimensions/#CommonDataModel.unlimited-Tuple{NCDatasets.Dimensions}","page":"Dimensions","title":"CommonDataModel.unlimited","text":"unlimited(d::Dimensions)\nunlimited(ds::AbstractNCDataset)\n\nReturn the names of all unlimited dimensions.\n\n\n\n\n\n","category":"method"},{"location":"dimensions/#Base.setindex!-Tuple{NCDatasets.Dimensions, Any, AbstractString}","page":"Dimensions","title":"Base.setindex!","text":"Base.setindex!(d::Dimensions,len,name::AbstractString)\n\nDefines the dimension called name to the length len. Generally dimension are defined by indexing, for example:\n\nds = NCDataset(\"file.nc\",\"c\")\nds.dim[\"longitude\"] = 100\n\nIf len is the special value Inf, then the dimension is considered as unlimited, i.e. it will grow as data is added to the NetCDF file.\n\n\n\n\n\n","category":"method"},{"location":"dimensions/","page":"Dimensions","title":"Dimensions","text":"One can iterate over a list of dimensions as follows:","category":"page"},{"location":"dimensions/","page":"Dimensions","title":"Dimensions","text":"for (dimname,dim) in ds.dim\n # all dimensions\n @show (dimname,dim)\nend","category":"page"},{"location":"#NCDatasets.jl","page":"Introduction","title":"NCDatasets.jl","text":"","category":"section"},{"location":"","page":"Introduction","title":"Introduction","text":"Documentation for NCDatasets.jl, a Julia package for loading and writing NetCDF (Network Common Data Form) files.","category":"page"},{"location":"#Installation","page":"Introduction","title":"Installation","text":"","category":"section"},{"location":"","page":"Introduction","title":"Introduction","text":"Inside the Julia shell, you can download and install using the following commands:","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"using Pkg\nPkg.add(\"NCDatasets\")","category":"page"},{"location":"#Latest-development-version","page":"Introduction","title":"Latest development version","text":"","category":"section"},{"location":"","page":"Introduction","title":"Introduction","text":"If you want to try the latest development version, again go into package manager mode and simply type","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"using Pkg\nPkg.add(PackageSpec(name=\"NCDatasets\", rev=\"master\"))","category":"page"},{"location":"#Contents","page":"Introduction","title":"Contents","text":"","category":"section"},{"location":"","page":"Introduction","title":"Introduction","text":"To get started quickly see the Quickstart section. Otherwise see the following pages for details:","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"Datasets : reading/writing NetCDF datasets (including NetCDF groups) and examining their contents.\nDimensions : accessing/creating NetCDF dimensions\nVariables : accessing/examining the variables (or dimensions) stored within a NetCDF dataset.\nAttributes : accessing/creating NetCDF attributes\nSee Performance tips, Known issues, Experimental features for more information.","category":"page"},{"location":"#Quickstart","page":"Introduction","title":"Quickstart","text":"","category":"section"},{"location":"","page":"Introduction","title":"Introduction","text":"This is a quickstart guide that outlines basic loading, reading, etc. usage. For more details please see the individual pages of the documentation.","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"Explore the content of a netCDF file\nLoad a netCDF file\nCreate a netCDF file\nEdit an existing netCDF file\nCreate a netCDF file using the metadata of an existing netCDF file as template\nGet one or several variables by specifying the value of an attribute\nLoad a file with unknown structure","category":"page"},{"location":"#Explore-the-content-of-a-netCDF-file","page":"Introduction","title":"Explore the content of a netCDF file","text":"","category":"section"},{"location":"","page":"Introduction","title":"Introduction","text":"Before reading the data from a netCDF file, it is often useful to explore the list of variables and attributes defined in it.","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"For interactive use, the following commands (without ending semicolon) display the content of the file similarly to ncdump -h file.nc:","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"using NCDatasets\nds = NCDataset(\"file.nc\")","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"which produces a listing like:","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"NCDataset: file.nc\nGroup: /\n\nDimensions\n time = 115\n\nVariables\n time (115)\n Datatype: Float64\n Dimensions: time\n Attributes:\n calendar = gregorian\n standard_name = time\n units = days since 1950-01-01 00:00:00\n[...]","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"This creates the central structure of NCDatasets.jl, NCDataset, which represents the contents of the netCDF file (without immediately loading everything in memory).","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"The following displays the information just for the variable varname:","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"ds[\"varname\"]","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"To get a list of global attributes, you can use:","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"ds.attrib","category":"page"},{"location":"#Load-a-netCDF-file","page":"Introduction","title":"Load a netCDF file","text":"","category":"section"},{"location":"","page":"Introduction","title":"Introduction","text":"Loading a variable with known structure can be achieved by accessing the variables and attributes directly by their name.","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"# The mode \"r\" stands for read-only. The mode \"r\" is the default mode and the parameter can be omitted.\nds = NCDataset(\"/tmp/test.nc\",\"r\")\nv = ds[\"temperature\"]\n\n# load a subset\nsubdata = v[10:30,30:5:end]\n\n# load all data\ndata = v[:,:]\n\n# load all data ignoring attributes like scale_factor, add_offset, _FillValue and time units\ndata2 = v.var[:,:]\n\n\n# load an attribute\nunit = v.attrib[\"units\"]\nclose(ds)","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"In the example above, the subset can also be loaded with:","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"subdata = NCDataset(\"/tmp/test.nc\")[\"temperature\"][10:30,30:5:end]","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"This might be useful in an interactive session. However, the file test.nc is not closed, which can be a problem if you open many files. On Linux the number of opened files is often limited to 1024 (soft limit). If you write to a file, you should also always close the file to make sure that the data is properly written to the disk. (open files will get closed eventually when the dataset variable is finalized by julia's garbage collector).","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"An alternative way to ensure the file has been closed is to use a do block: the file will be closed automatically when leaving the block.","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"data = NCDataset(filename,\"r\") do ds\n ds[\"temperature\"][:,:]\nend # ds is closed","category":"page"},{"location":"#Create-a-netCDF-file","page":"Introduction","title":"Create a netCDF file","text":"","category":"section"},{"location":"","page":"Introduction","title":"Introduction","text":"The following gives an example of how to create a netCDF file by defining dimensions, variables and attributes.","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"using NCDatasets\n# This creates a new NetCDF file /tmp/test.nc.\n# The mode \"c\" stands for creating a new file (clobber)\nds = NCDataset(\"/tmp/test.nc\",\"c\")\n\n# Define the dimension \"lon\" and \"lat\" with the size 100 and 110 resp.\ndefDim(ds,\"lon\",100)\ndefDim(ds,\"lat\",110)\n\n# Define a global attribute\nds.attrib[\"title\"] = \"this is a test file\"\n\n# Define the variables temperature\nv = defVar(ds,\"temperature\",Float32,(\"lon\",\"lat\"))\n\n# Generate some example data\ndata = [Float32(i+j) for i = 1:100, j = 1:110]\n\n# write a single column\nv[:,1] = data[:,1]\n\n# write a the complete data set\nv[:,:] = data\n\n# write attributes\nv.attrib[\"units\"] = \"degree Celsius\"\nv.attrib[\"comments\"] = \"this is a string attribute with Unicode Ω ∈ ∑ ∫ f(x) dx\"\n\nclose(ds)","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"An equivalent way to create the previous netCDF would be the following code:","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"using NCDatasets\nusing DataStructures\ndata = [Float32(i+j) for i = 1:100, j = 1:110]\n\nDataset(\"/tmp/test2.nc\",\"c\",attrib = OrderedDict(\"title\" => \"this is a test file\")) do ds\n # Define the variable temperature. The dimension \"lon\" and \"lat\" with the\n # size 100 and 110 resp are implicetly created\n defVar(ds,\"temperature\",data,(\"lon\",\"lat\"), attrib = OrderedDict(\n \"units\" => \"degree Celsius\",\n \"comments\" => \"this is a string attribute with Unicode Ω ∈ ∑ ∫ f(x) dx\"\n ))\nend","category":"page"},{"location":"#Edit-an-existing-netCDF-file","page":"Introduction","title":"Edit an existing netCDF file","text":"","category":"section"},{"location":"","page":"Introduction","title":"Introduction","text":"When you need to modify the variables or the attributes of a netCDF, you have to open it with the \"a\" option. Here, for instance, we add a global attribute creator to the file created in the previous step.","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"ds = NCDataset(\"/tmp/test.nc\",\"a\")\nds.attrib[\"creator\"] = \"your name\"\nclose(ds);","category":"page"},{"location":"#Create-a-netCDF-file-using-the-metadata-of-an-existing-netCDF-file-as-template","page":"Introduction","title":"Create a netCDF file using the metadata of an existing netCDF file as template","text":"","category":"section"},{"location":"","page":"Introduction","title":"Introduction","text":"The utility function ncgen generates the Julia code that would produce a netCDF file with the same metadata as a template netCDF file. It is thus similar to the command line tool ncgen.","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"# download example file\nncfile = download(\"https://www.unidata.ucar.edu/software/netcdf/examples/sresa1b_ncar_ccsm3-example.nc\")\n# generate Julia code\nncgen(ncfile)","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"The produces the Julia code (only the beginning of the code is shown):","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"ds = NCDataset(\"filename.nc\",\"c\")\n# Dimensions\n\nds.dim[\"lat\"] = 128;\nds.dim[\"lon\"] = 256;\nds.dim[\"bnds\"] = 2;\nds.dim[\"plev\"] = 17;\nds.dim[\"time\"] = 1;\n\n# Declare variables\n\nncarea = defVar(ds,\"area\", Float32, (\"lon\", \"lat\"))\nncarea.attrib[\"long_name\"] = \"Surface area\";\nncarea.attrib[\"units\"] = \"meter2\";\n# ...","category":"page"},{"location":"#Get-one-or-several-variables-by-specifying-the-value-of-an-attribute","page":"Introduction","title":"Get one or several variables by specifying the value of an attribute","text":"","category":"section"},{"location":"","page":"Introduction","title":"Introduction","text":"The variable names are not always standardized. For example, for the longitude we can find: lon, LON, longitude, ...","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"The solution implemented in the function varbyattrib consists in searching for the variables that have specified value for a given attribute.","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"nclon = varbyattrib(ds, standard_name = \"longitude\");","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"will return the list of variables of the dataset ds that have \"longitude\" as standard name. To directly load the data of the first variable with the attribute standard_name equal to \"longitude\" one can do the following:","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"data = varbyattrib(ds, standard_name = \"longitude\")[1][:]","category":"page"},{"location":"#Load-a-file-with-unknown-structure","page":"Introduction","title":"Load a file with unknown structure","text":"","category":"section"},{"location":"","page":"Introduction","title":"Introduction","text":"If the structure of the netCDF file is not known before-hand, the program must check if a variable or attribute exists (with the haskey function) before loading it or alternatively place the loading in a try-catch block. It is also possible to iterate over all variables or attributes (global attributes or variable attributes) in the same syntax as iterating over a dictionary. However, unlike Julia dictionaries, the order of the attributes and variables is preserved and presented as they are stored in the netCDF file.","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"# Open a file as read-only\nds = NCDataset(\"/tmp/test.nc\",\"r\")\n\n# check if a file has a variable with a given name\nif haskey(ds,\"temperature\")\n println(\"The file has a variable 'temperature'\")\nend\n\n# get a list of all variable names\n@show keys(ds)\n\n# iterate over all variables\nfor (varname,var) in ds\n @show (varname,size(var))\nend\n\n# query size of a variable (without loading it)\nv = ds[\"temperature\"]\n@show size(v)\n\n# similar for global and variable attributes\n\nif haskey(ds.attrib,\"title\")\n println(\"The file has the global attribute 'title'\")\nend\n\n# get an list of all attribute names\n@show keys(ds.attrib)\n\n# iterate over all attributes\nfor (attname,attval) in ds.attrib\n @show (attname,attval)\nend\n\n# get the attribute \"units\" of the variable v\n# but return the default value (here \"adimensional\")\n# if the attribute does not exists\n\nunits = get(v,\"units\",\"adimensional\")\nclose(ds)","category":"page"}] +[{"location":"dataset/#Datasets","page":"Datasets","title":"Datasets","text":"","category":"section"},{"location":"dataset/","page":"Datasets","title":"Datasets","text":"This page is about loading/writing, examining and operating directly on entire NetCDF datasets. For functions regarding the variables stored in them, see the Variables page.","category":"page"},{"location":"dataset/","page":"Datasets","title":"Datasets","text":"Both variables and datasets share the functionality of the Attributes section.","category":"page"},{"location":"dataset/","page":"Datasets","title":"Datasets","text":"NCDataset","category":"page"},{"location":"dataset/#NCDatasets.NCDataset","page":"Datasets","title":"NCDatasets.NCDataset","text":"NCDataset(filename::AbstractString, mode = \"r\";\n format::Symbol = :netcdf4,\n share::Bool = false,\n diskless::Bool = false,\n persist::Bool = false,\n memory::Union{Vector{UInt8},Nothing} = nothing,\n attrib = [])\n\nLoad, create, or even overwrite a NetCDF file at filename, depending on mode\n\n\"r\" (default) : open an existing netCDF file or OPeNDAP URL in read-only mode.\n\"c\" : create a new NetCDF file at filename (an existing file with the same name will be overwritten).\n\"a\" : open filename into append mode (i.e. existing data in the netCDF file is not overwritten and a variable can be added).\n\nIf share is true, the NC_SHARE flag is set allowing to have multiple processes to read the file and one writer process. Likewise setting diskless or persist to true will enable the flags NC_DISKLESS or NC_PERSIST flag. More information is available in the NetCDF C-API.\n\nNotice that this does not close the dataset, use close on the result (or see below the do-block).\n\nThe optional parameter attrib is an iterable of attribute name and attribute value pairs, for example a Dict, DataStructures.OrderedDict or simply a vector of pairs (see example below).\n\nSupported format values:\n\n:netcdf4 (default): HDF5-based NetCDF format.\n:netcdf4_classic: Only netCDF 3 compatible API features will be used.\n:netcdf3_classic: classic netCDF format supporting only files smaller than 2GB.\n:netcdf3_64bit_offset: improved netCDF format supporting files larger than 2GB.\n:netcdf5_64bit_data: improved netCDF format supporting 64-bit integer data types.\n\nFiles can also be open and automatically closed with a do block.\n\nNCDataset(\"file.nc\") do ds\n data = ds[\"temperature\"][:,:]\nend\n\nHere is an attribute example:\n\nusing DataStructures\nNCDataset(\"file.nc\", \"c\", attrib = OrderedDict(\"title\" => \"my first netCDF file\")) do ds\n defVar(ds,\"temp\",[10.,20.,30.],(\"time\",))\nend;\n\nThe NetCDF dataset can also be a memory as a vector of bytes. A non-empty string a filename is still required, for example:\n\nusing NCDataset, HTTP\nresp = HTTP.get(\"https://www.unidata.ucar.edu/software/netcdf/examples/ECMWF_ERA-40_subset.nc\")\nds = NCDataset(\"some_string\",\"r\",memory = resp.body)\ntotal_precipitation = ds[\"tp\"][:,:,:]\nclose(ds)\n\nDataset is an alias of NCDataset.\n\n\n\n\n\nmfds = NCDataset(fnames, mode = \"r\"; aggdim = nothing, deferopen = true,\n isnewdim = false,\n constvars = [])\n\nOpens a multi-file dataset in read-only \"r\" or append mode \"a\". fnames is a vector of file names.\n\nVariables are aggregated over the first unlimited dimension or over the dimension aggdim if specified. Variables without the dimensions aggdim are not aggregated. All variables containing the dimension aggdim are aggregated. The variable who do not contain the dimension aggdim are assumed constant.\n\nIf variables should be aggregated over a new dimension (not present in the NetCDF file), one should set isnewdim to true. All NetCDF files should have the same variables, attributes and groupes. Per default, all variables will have an additional dimension unless they are marked as constant using the constvars parameter.\n\nThe append mode is only implemented when deferopen is false. If deferopen is false, all files are opened at the same time. However the operating system might limit the number of open files. In Linux, the limit can be controled with the command ulimit.\n\nAll metadata (attributes and dimension length are assumed to be the same for all NetCDF files. Otherwise reading the attribute of a multi-file dataset would be ambiguous. An exception to this rule is the length of the dimension over which the data is aggregated. This aggregation dimension can varify from file to file.\n\nSetting the experimental flag _aggdimconstant to true means that the length of the aggregation dimension is constant. This speeds up the creating of a multi-file dataset as only the metadata of the first file has to be loaded.\n\nExamples:\n\nYou can use Glob.jl to make fnames from a file pattern, e.g.\n\nusing NCDatasets, Glob\nds = NCDataset(glob(\"ERA5_monthly3D_reanalysis_*.nc\"))\n\nAggregation over a new dimension:\n\nusing NCDatasets\nfor i = 1:3\n NCDataset(\"foo$i.nc\",\"c\") do ds\n defVar(ds,\"data\",[10., 11., 12., 13.], (\"lon\",))\n end\nend\n\nds = NCDataset([\"foo$i.nc\" for i = 1:3],aggdim = \"sample\", isnewdim = true)\nsize(ds[\"data\"])\n# output\n# (4, 3)\n\n\n\n\n\n","category":"type"},{"location":"dataset/","page":"Datasets","title":"Datasets","text":"Useful functions that operate on datasets are:","category":"page"},{"location":"dataset/","page":"Datasets","title":"Datasets","text":"keys(ds::NCDataset)\nhaskey\ngetindex(ds::NCDataset,varname::AbstractString)\nvariable\ncfvariable\nsync\nclose\nNCDatasets.path\nncgen\nvarbyattrib\nwrite","category":"page"},{"location":"dataset/#Base.keys-Tuple{NCDataset}","page":"Datasets","title":"Base.keys","text":"keys(ds::NCDataset)\n\nReturn a list of all variables names in NCDataset ds.\n\n\n\n\n\n","category":"method"},{"location":"dataset/#Base.haskey","page":"Datasets","title":"Base.haskey","text":"haskey(ds::NCDataset,name)\nhaskey(d::Dimensions,name)\nhaskey(ds::Attributes,name)\n\nReturn true if the NCDataset ds (or dimension/attribute list) has a variable (dimension/attribute) with the name name. For example:\n\nds = NCDataset(\"/tmp/test.nc\",\"r\")\nif haskey(ds,\"temperature\")\n println(\"The file has a variable 'temperature'\")\nend\n\nif haskey(ds.dim,\"lon\")\n println(\"The file has a dimension 'lon'\")\nend\n\nThis example checks if the file /tmp/test.nc has a variable with the name temperature and a dimension with the name lon.\n\n\n\n\n\nBase.haskey(a::Attributes,name::SymbolOrString)\n\nCheck if name is an attribute\n\n\n\n\n\n","category":"function"},{"location":"dataset/#Base.getindex-Tuple{NCDataset, AbstractString}","page":"Datasets","title":"Base.getindex","text":"v = getindex(ds::NCDataset, varname::AbstractString)\n\nReturn the variable varname in the dataset ds as a CFVariable. The following CF convention are honored when the variable is indexed:\n\n_FillValue or missing_value (which can be a list) will be returned as missing.\nscale_factor and add_offset are applied (output = scale_factor * data_in_file + add_offset)\ntime variables (recognized by the units attribute and possibly the calendar attribute) are returned usually as DateTime object. Note that CFTime.DateTimeAllLeap, CFTime.DateTimeNoLeap and CF.TimeDateTime360Day cannot be converted to the proleptic gregorian calendar used in julia and are returned as such. (See CFTime.jl for more information about those date types.) If a calendar is defined but not among the ones specified in the CF convention, then the data in the file is not converted into a date structure.\n\nA call getindex(ds, varname) is usually written as ds[varname].\n\nIf variable represents a cell boundary, the attributes calendar and units of the related variables are used, if they are not specified. For example:\n\ndimensions:\n time = UNLIMITED; // (5 currently)\n nv = 2;\nvariables:\n double time(time);\n time:long_name = \"time\";\n time:units = \"hours since 1998-04-019 06:00:00\";\n time:bounds = \"time_bnds\";\n double time_bnds(time,nv);\n\nIn this case, the variable time_bnds uses the units and calendar of time because both variables are related thought the bounds attribute following the CF conventions.\n\nSee also cfvariable(ds, varname).\n\n\n\n\n\n","category":"method"},{"location":"dataset/#CommonDataModel.variable","page":"Datasets","title":"CommonDataModel.variable","text":"v = variable(ds::NCDataset,varname::String)\n\nReturn the NetCDF variable varname in the dataset ds as a NCDataset.Variable. No scaling or other transformations are applied when the variable v is indexed.\n\n\n\n\n\nCommonDataModel.variable(ds::AbstractDataset,variablename::SymbolOrString)\n\nReturn the variable with the name variablename from the data set ds.\n\n\n\n\n\n","category":"function"},{"location":"dataset/#CommonDataModel.cfvariable","page":"Datasets","title":"CommonDataModel.cfvariable","text":"v = cfvariable(ds::NCDataset,varname::SymbolOrString; = )\n\nReturn the variable varname in the dataset ds as a NCDataset.CFVariable. The keyword argument are the attributes (fillvalue, missing_value, scale_factor, add_offset, units and calendar) relevant to the CF conventions. By specifing the value of these attributes, the one can override the value specified in the data set. If the attribute is set to nothing, then the attribute is not loaded and the corresponding transformation is ignored. This function is similar to ds[varname] with the additional flexibility that some variable attributes can be overridden.\n\nExample:\n\nNCDataset(\"foo.nc\",\"c\") do ds\n defVar(ds,\"data\",[10., 11., 12., 13.], (\"time\",), attrib = Dict(\n \"add_offset\" => 10.,\n \"scale_factor\" => 0.2))\nend\n\n# The stored (packed) valued are [0., 5., 10., 15.]\n# since 0.2 .* [0., 5., 10., 15.] .+ 10 is [10., 11., 12., 13.]\n\nds = NCDataset(\"foo.nc\");\n\n@show ds[\"data\"].var[:]\n# returns [0., 5., 10., 15.]\n\n@show cfvariable(ds,\"data\")[:]\n# returns [10., 11., 12., 13.]\n\n# neither add_offset nor scale_factor are applied\n@show cfvariable(ds,\"data\", add_offset = nothing, scale_factor = nothing)[:]\n# returns [0, 5, 10, 15]\n\n# add_offset is applied but not scale_factor\n@show cfvariable(ds,\"data\", scale_factor = nothing)[:]\n# returns [10, 15, 20, 25]\n\n# 0 is declared as the fill value (add_offset and scale_factor are applied as usual)\n@show cfvariable(ds,\"data\", fillvalue = 0)[:]\n# return [missing, 11., 12., 13.]\n\n# Use the time units: days since 2000-01-01\n@show cfvariable(ds,\"data\", units = \"days since 2000-01-01\")[:]\n# returns [DateTime(2000,1,11), DateTime(2000,1,12), DateTime(2000,1,13), DateTime(2000,1,14)]\n\nclose(ds)\n\n\n\n\n\n","category":"function"},{"location":"dataset/#CommonDataModel.sync","page":"Datasets","title":"CommonDataModel.sync","text":"sync(ds::NCDataset)\n\nWrite all changes in NCDataset ds to the disk.\n\n\n\n\n\n","category":"function"},{"location":"dataset/#Base.close","page":"Datasets","title":"Base.close","text":"close(ds::NCDataset)\n\nClose the NCDataset ds. All pending changes will be written to the disk.\n\n\n\n\n\n","category":"function"},{"location":"dataset/#CommonDataModel.path","page":"Datasets","title":"CommonDataModel.path","text":"path(ds::NCDataset)\n\nReturn the file path (or the opendap URL) of the NCDataset ds\n\n\n\n\n\nCommonDatamodel.path(ds::AbstractDataset)\n\nFile path of the data set ds.\n\n\n\n\n\n","category":"function"},{"location":"dataset/#NCDatasets.ncgen","page":"Datasets","title":"NCDatasets.ncgen","text":"ncgen(fname; ...)\nncgen(fname,jlname; ...)\n\nGenerate the Julia code that would produce a NetCDF file with the same metadata as the NetCDF file fname. The code is placed in the file jlname or printed to the standard output. By default the new NetCDF file is called filename.nc. This can be changed with the optional parameter newfname.\n\n\n\n\n\n","category":"function"},{"location":"dataset/#CommonDataModel.varbyattrib","page":"Datasets","title":"CommonDataModel.varbyattrib","text":"varbyattrib(ds, attname = attval)\n\nReturns a list of variable(s) which has the attribute attname matching the value attval in the dataset ds. The list is empty if the none of the variables has the match. The output is a list of CFVariables.\n\nExamples\n\nLoad all the data of the first variable with standard name \"longitude\" from the NetCDF file results.nc.\n\njulia> ds = NCDataset(\"results.nc\", \"r\");\njulia> data = varbyattrib(ds, standard_name = \"longitude\")[1][:]\n\n\n\n\n\n","category":"function"},{"location":"dataset/#Base.write","page":"Datasets","title":"Base.write","text":"write(dest::AbstractDataset, src::AbstractDataset; include = keys(src), exclude = [])\n\nWrite the variables of src dataset into an empty dest dataset (which must be opened in mode \"a\" or \"c\"). The keywords include and exclude configure which variable of src should be included (by default all), or which should be excluded (by default none).\n\nIf the first argument is a file name, then the dataset is open in create mode (\"c\").\n\nThis function is useful when you want to save the dataset from a multi-file dataset.\n\nTo save a subset, one can use the view function view to virtually slice a dataset:\n\nExample\n\nNCDataset(fname_src) do ds\n write(fname_slice,view(ds, lon = 2:3))\nend\n\nAll variables in the source file fname_src with a dimension lon will be sliced along the indices 2:3 for the lon dimension. All attributes (and variables without a dimension lon) will be copied over unmodified.\n\n\n\n\n\n","category":"function"},{"location":"dataset/","page":"Datasets","title":"Datasets","text":"Notice that DateTime-structures from CFTime are used to represent time for non-standard calendars. Otherwise, we attempt to use standard structures from the Julia standard library Dates.","category":"page"},{"location":"dataset/#Groups","page":"Datasets","title":"Groups","text":"","category":"section"},{"location":"dataset/","page":"Datasets","title":"Datasets","text":"defGroup\ngetindex(g::NCDatasets.Groups,groupname::AbstractString)\nBase.keys(g::NCDatasets.Groups)","category":"page"},{"location":"dataset/#CommonDataModel.defGroup","page":"Datasets","title":"CommonDataModel.defGroup","text":"defGroup(ds::NCDataset,groupname; attrib = []))\n\nCreate the group with the name groupname in the dataset ds. attrib is a list of attribute name and attribute value pairs (see NCDataset).\n\n\n\n\n\ngroup = CommonDatamodel.defGroup(ds::AbstractDataset,name::SymbolOrString)\n\nCreate an empty sub-group with the name name in the data set ds. The group is a sub-type of AbstractDataset.\n\n\n\n\n\n","category":"function"},{"location":"dataset/#Common-methods","page":"Datasets","title":"Common methods","text":"","category":"section"},{"location":"dataset/","page":"Datasets","title":"Datasets","text":"One can iterate over a dataset, attribute list, dimensions and NetCDF groups.","category":"page"},{"location":"dataset/","page":"Datasets","title":"Datasets","text":"for (varname,var) in ds\n # all variables\n @show (varname,size(var))\nend\n\nfor (attribname,attrib) in ds.attrib\n # all attributes\n @show (attribname,attrib)\nend\n\nfor (groupname,group) in ds.groups\n # all groups\n @show (groupname,group)\nend","category":"page"},{"location":"performance/#performance_tips","page":"Performance tips","title":"Performance tips","text":"","category":"section"},{"location":"performance/","page":"Performance tips","title":"Performance tips","text":"Reading data from a file is not type-stable, because the type of the output of the read operation is dependent on the type defined in the NetCDF files and the value of various attribute (like scale_factor, add_offset and units for time conversion). All this information cannot be inferred from a static analysis of the source code. It is therefore recommended to use type annotation if the resulting type of a read operation in known:","category":"page"},{"location":"performance/","page":"Performance tips","title":"Performance tips","text":"ds = NCDataset(\"file.nc\")\nnctemp = ds[\"temp\"]\ntemp = nctemp[:,:] :: Array{Float32,2}\n\n# heavy computation using temp\n# ...","category":"page"},{"location":"performance/","page":"Performance tips","title":"Performance tips","text":"Alternatively, one can also use so-called function barriers since the function heavy_computation will be specialized based on the type its input parameters.","category":"page"},{"location":"performance/","page":"Performance tips","title":"Performance tips","text":"function heavy_computation(temp)\n# heavy computation using temp\n# ...\nend\n\nds = NCDataset(\"file.nc\")\nnctemp = ds[\"temp\"]\ntemp = nctemp[:,:]\noutput = heavy_computation(temp)","category":"page"},{"location":"performance/","page":"Performance tips","title":"Performance tips","text":"Calling the barrier function with nctemp would also be type-stable. Using the in-place NCDatasets.load! function (which is unexported, so it has to be prefixed with the module name) does also lead to type-stable code and allows to reuse a memory buffer:","category":"page"},{"location":"performance/","page":"Performance tips","title":"Performance tips","text":"ds = NCDataset(\"file.nc\")\n\ntemp = zeros(Float32,10,20)\nNCDatasets.load!(variable(ds,\"temp\"),temp,:,:)","category":"page"},{"location":"performance/","page":"Performance tips","title":"Performance tips","text":"Most julia functions (like mean, sum,... from the module Statistics) access an array element-wise. It is generally much faster to load the data in memory (if possible) to make the computation.","category":"page"},{"location":"performance/","page":"Performance tips","title":"Performance tips","text":"using NCDatasets, BenchmarkTools, Statistics\nds = NCDataset(\"file.nc\",\"c\")\ndata = randn(100,100);\ndefVar(ds,\"myvar\",data,(\"lon\",\"lat\"))\nclose(ds)\n\nds = NCDataset(\"file.nc\")\n@btime mean(ds[\"myvar\"]) # takes 107.357 ms\n@btime mean(ds[\"myvar\"][:,:]) # takes 106.873 μs, 1000 times faster\nclose(ds)","category":"page"},{"location":"performance/","page":"Performance tips","title":"Performance tips","text":"Avoid, when possible, indexing with arrays and CartesianIndex as they also result in loading the data element-wise.","category":"page"},{"location":"performance/","page":"Performance tips","title":"Performance tips","text":"ds = NCDataset(\"dataset.nc\");\nv = ds[\"v1\"][:,1:3,:]; # fast\nv = ds[\"v1\"][:,:,CartesianIndex(1)] # slow\nv = ds[\"v1\"][:,:,1] # fast\nclose(ds)","category":"page"},{"location":"experimental/#Experimental-features","page":"Experimental features","title":"Experimental features","text":"","category":"section"},{"location":"experimental/#Multi-file-support","page":"Experimental features","title":"Multi-file support","text":"","category":"section"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"Multiple files can also be aggregated over a given dimension (or the record dimension). In this example, 3 sea surface temperature fields from the 1992-01-01 to 1992-01-03 are aggregated using the OPeNDAP service from PODAAC.","category":"page"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"using NCDatasets, Printf, Dates\n\nfunction url(dt)\n doy = @sprintf(\"%03d\",Dates.dayofyear(dt))\n y = @sprintf(\"%04d\",Dates.year(dt))\n yyyymmdd = Dates.format(dt,\"yyyymmdd\")\n return \"https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/CMC/CMC0.2deg/v2/$y/$doy/$(yyyymmdd)120000-CMC-L4_GHRSST-SSTfnd-CMC0.2deg-GLOB-v02.0-fv02.0.nc\"\nend\n\nds = NCDataset(url.(DateTime(1992,1,1):Dates.Day(1):DateTime(1992,1,3)),aggdim = \"time\");\nSST2 = ds[\"analysed_sst\"][:,:,:];\nclose(ds)","category":"page"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"If there is a network or server issue, you will see an error message like NetCDF: I/O failure.","category":"page"},{"location":"experimental/#CF-Standard-Names","page":"Experimental features","title":"CF Standard Names","text":"","category":"section"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"The CF Conventions do not define how the different NetCDF variables are named, but the meaning of a variable is defined by the standard_name attribute.","category":"page"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"using NCDatasets, DataStructures\nds = NCDataset(tempname(),\"c\")\n\nnclon = defVar(ds,\"lon\", 1:10, (\"lon\",),attrib = OrderedDict(\n \"standard_name\" => \"longitude\",\n))\nnclat = defVar(ds,\"lat\", 1:11, (\"lat\",),attrib = OrderedDict(\n \"standard_name\" => \"latitude\",\n))\nncvar = defVar(ds,\"bat\", zeros(10,11), (\"lon\", \"lat\"), attrib = OrderedDict(\n \"standard_name\" => \"height\",\n))\n\nncbat = ds[CF\"height\"]\n# the same as\n# ncbat = varbyattrib(ds,standard_name = \"height\")[1]\n\nname(ncbat)\n# output\n\"bat\"","category":"page"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"If there are multiple variables with the standard_name equal to height, an error is returned because it is ambiguous which variable should be accessed.","category":"page"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"All variables whose dimensions are also dimensions of ncbat are considered as related and can also be accessed by sub-setting ncbat with their variable names of CF Standard name:","category":"page"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"nclon_of_bat = ncbat[CF\"longitude\"]\n# same as\n# nclon_of_bat = ncbat[\"lon\"]\nname(nclon_of_bat)\n# output\n\"lon\"","category":"page"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"The previous call to ncbat[CF\"longitude\"] would also worked if there are multiple variables with a standard name longitude defined in a dataset as long as they have different dimension names (which is commonly the case for model output on staggered grid such as Regional Ocean Modeling System).","category":"page"},{"location":"experimental/#Views","page":"Experimental features","title":"Views","text":"","category":"section"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"In Julia, a view of an array is a subset of an array but whose elements still point to the original parent array. If one modifies an element of a view, the corresponding element in the parent array is modified too:","category":"page"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"A = zeros(4,4)\nsubset = @view A[2:3,2:4]\n# or\n# subset = view(A,2:3,2:4)\n\nsubset[1,1] = 2\nA[2,2]\n# output\n2.0","category":"page"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"Views do not use copy of the array. The parent array and the indices of the view are obtained via the function parent and parentindices.","category":"page"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"parent(subset) == A\n# true, as both arrays are the same\n\nparentindices(subset)\n# output\n(2:3, 2:4)","category":"page"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"In NCDatasets, variables can also be sliced as a view:","category":"page"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"using NCDatasets, DataStructures\nds = NCDataset(tempname(),\"c\")\n\nnclon = defVar(ds,\"lon\", 1:10, (\"lon\",))\nnclat = defVar(ds,\"lat\", 1:11, (\"lat\",))\nncvar = defVar(ds,\"bat\", zeros(10,11), (\"lon\", \"lat\"), attrib = OrderedDict(\n \"standard_name\" => \"height\",\n))\n\nncvar_subset = @view ncvar[2:4,2:3]\n# or\n# ncvar_subset = view(ncvar,2:4,2:3)\n\nncvar_subset[1,1] = 2\n# ncvar[2,2] is now 2\n\nncvar_subset.attrib[\"standard_name\"]\n\n# output\n\"height\"","category":"page"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"This is useful for example when even the sliced array is too large to be loaded in RAM or when all attributes need to be preserved for the sliced array.","category":"page"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"The variables lon and lat are related to bat because all dimensions of the variables lon and lat are also dimensions of bat (which is commonly the case for coordinate variables). Such related variables can be retrieved by indexing the NetCDF variables with the name of the corresponding variable:","category":"page"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"lon_subset = ncvar_subset[\"lon\"]\nlon_subset[:] == [2, 3, 4]\n# output\ntrue","category":"page"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"A view of a NetCDF variable also implements the function parent and parentindices with the same meaning as for julia Arrays.","category":"page"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"A whole dataset can also be sliced using a view(ds, dim1=range1, dim2=range2...). For example:","category":"page"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"ds_subset = view(ds, lon = 2:3, lat = 2:4)\n# or\n# ds_subset = @view ds[lon = 2:3, lat = 2:4]\nds_subset.dim[\"lon\"]\n\n# output\n2","category":"page"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"Such sliced datasets can for example be saved into a new NetCDF file using write:","category":"page"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"write(\"slice.nc\",ds_subset)","category":"page"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"Any dimension not mentioned in the @view call is not sliced. While @view produces a slice based on indices, the NCDatasets.@select macro produces a slice (of an NetCDF variable or dataset) based on the values of other related variables (typically coordinates).","category":"page"},{"location":"experimental/#Data-selection-based-on-values","page":"Experimental features","title":"Data selection based on values","text":"","category":"section"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"NCDatasets.@select","category":"page"},{"location":"experimental/#CommonDataModel.@select","page":"Experimental features","title":"CommonDataModel.@select","text":"vsubset = CommonDataModel.@select(v,expression)\ndssubset = CommonDataModel.@select(ds,expression)\n\nReturn a subset of the variable v (or dataset ds) satisfying the condition expression as a view. The condition has the following form:\n\ncondition₁ && condition₂ && condition₃ ... conditionₙ\n\nEvery condition should involve a single 1D variable (typically a coordinate variable, referred as coord below). If v is a variable, the related 1D variable should have a shared dimension with the variable v. All local variables need to have a $ prefix (see examples below). This macro is experimental and subjected to change.\n\nEvery condition can either perform:\n\na nearest match: coord ≈ target_coord (for ≈ type \\approx followed by the TAB-key). Only the data corresponding to the index closest to target_coord is loaded.\na nearest match with tolerance: coord ≈ target_coord ± tolerance. As before, but if the difference between the closest value in coord and target_coord is larger (in absolute value) than tolerance, an empty array is returned.\na condition operating on scalar values. For example, a condition equal to 10 <= lon <= 20 loads all data with the longitude between 10 and 20 or abs(lat) > 60 loads all variables with a latitude north of 60° N and south of 60° S (assuming that the has the 1D variables lon and lat for longitude and latitude).\n\nOnly the data which satisfies all conditions is loaded. All conditions must be chained with an && (logical and). They should not contain additional parenthesis or other logical operators such as || (logical or).\n\nTo convert the view into a regular array one can use collect, Array or regular indexing. As in julia, views of scalars are wrapped into a zero dimensional arrays which can be dereferenced by using []. Modifying a view will modify the underlying file (if the file is opened as writable, otherwise an error is issued).\n\nAs for any view, one can use parentindices(vsubset) to get the indices matching a select query.\n\nExamples\n\nCreate a sample file with random data:\n\nusing NCDatasets, Dates\nfname = \"sample_file.nc\"\nlon = -180:180\nlat = -90:90\ntime = DateTime(2000,1,1):Day(1):DateTime(2000,1,3)\nSST = randn(length(lon),length(lat),length(time))\n\nds = NCDataset(fname,\"c\")\ndefVar(ds,\"lon\",lon,(\"lon\",));\ndefVar(ds,\"lat\",lat,(\"lat\",));\ndefVar(ds,\"time\",time,(\"time\",));\ndefVar(ds,\"SST\",SST,(\"lon\",\"lat\",\"time\"));\n\n\n# load by bounding box\nv = NCDatasets.@select(ds[\"SST\"],30 <= lon <= 60 && 40 <= lat <= 90)\n\n# substitute a local variable in condition using $\nlonr = (30,60) # longitude range\nlatr = (40,90) # latitude range\n\nv = NCDatasets.@select(ds[\"SST\"],$lonr[1] <= lon <= $lonr[2] && $latr[1] <= lat <= $latr[2])\n\n# You can also select based on `ClosedInterval`s from `IntervalSets.jl`.\n# Both 30..60 and 65 ± 25 construct `ClosedInterval`s, see their documentation for details.\n\nlon_interval = 30..60\nlat_interval = 65 ± 25\nv = NCDatasets.@select(ds[\"SST\"], lon ∈ $lon_interval && lat ∈ $lat_interval)\n\n# get the indices matching the select query\n(lon_indices,lat_indices,time_indices) = parentindices(v)\n\n# get longitude matchting the select query\nv_lon = v[\"lon\"]\n\n# find the nearest time instance\nv = NCDatasets.@select(ds[\"SST\"],time ≈ DateTime(2000,1,4))\n\n# find the nearest time instance but not earlier or later than 2 hours\n# an empty array is returned if no time instance is present\n\nv = NCDatasets.@select(ds[\"SST\"],time ≈ DateTime(2000,1,3,1) ± Hour(2))\n\nclose(ds)\n\nAny 1D variable with the same dimension name can be used in @select. For example, if we have a time series of temperature and salinity, the temperature values can also be selected based on salinity:\n\n# create a sample time series\nusing NCDatasets, Dates\nfname = \"sample_series.nc\"\ntime = DateTime(2000,1,1):Day(1):DateTime(2009,12,31)\nsalinity = randn(length(time)) .+ 35\ntemperature = randn(length(time))\n\nNCDataset(fname,\"c\") do ds\n defVar(ds,\"time\",time,(\"time\",));\n defVar(ds,\"salinity\",salinity,(\"time\",));\n defVar(ds,\"temperature\",temperature,(\"time\",));\nend\n\nds = NCDataset(fname)\n\n# load all temperature data from January where the salinity is larger than 35.\nv = NCDatasets.@select(ds[\"temperature\"],Dates.month(time) == 1 && salinity >= 35)\n\n# this is equivalent to\nv2 = ds[\"temperature\"][findall(Dates.month.(time) .== 1 .&& salinity .>= 35)]\n\n@test v == v2\nclose(ds)\n\nnote: Note\nFor optimal performance, one should try to load contiguous data ranges, in particular when the data is loaded over HTTP/OPeNDAP.\n\n\n\n\n\n","category":"macro"},{"location":"experimental/#Experimental-functions","page":"Experimental features","title":"Experimental functions","text":"","category":"section"},{"location":"experimental/","page":"Experimental features","title":"Experimental features","text":"NCDatasets.ancillaryvariables\nNCDatasets.filter","category":"page"},{"location":"experimental/#CommonDataModel.ancillaryvariables","page":"Experimental features","title":"CommonDataModel.ancillaryvariables","text":"ncvar = CommonDataModel.ancillaryvariables(ncv::CFVariable,modifier)\n\nReturn the first ancillary variables from the NetCDF (or other format) variable ncv with the standard name modifier modifier. It can be used for example to access related variable like status flags.\n\n\n\n\n\n","category":"function"},{"location":"experimental/#Base.filter","page":"Experimental features","title":"Base.filter","text":"data = CommonDataModel.filter(ncv, indices...; accepted_status_flags = nothing)\n\nLoad and filter observations by replacing all variables without an acepted status flag to missing. It is used the attribute ancillary_variables to identify the status flag.\n\n# da[\"data\"] is 2D matrix\ngood_data = NCDatasets.filter(ds[\"data\"],:,:, accepted_status_flags = [\"good_data\",\"probably_good_data\"])\n\n\n\n\n\n","category":"function"},{"location":"tutorials/#Tutorials","page":"Tutorials","title":"Tutorials","text":"","category":"section"},{"location":"tutorials/#Data-from-Copernicus-Marine","page":"Tutorials","title":"Data from Copernicus Marine","text":"","category":"section"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"This examples shows how to download sea surface temperature from the Mediterranean Sea High Resolution and Ultra High Resolution Sea Surface Temperature Analysis provided by Copernicus Marine Service. Username and password are obtained after registration.","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"The username and password can be added to the URL. For example https://example.org/path should become https://username:password@example.org/path:","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"using NCDatasets, PyPlot, Statistics, URIs\n\nusername = \"your_username\"\npassword = \"your_password\"\n\nurl = \"https://nrt.cmems-du.eu/thredds/dodsC/SST_MED_SST_L4_NRT_OBSERVATIONS_010_004_a_V2\"\n\n# add username and password to url\n# username or password can contain special characters\nusername_escaped = URIs.escapeuri(username)\npassword_escaped = URIs.escapeuri(password)\nurl2 = string(URI(URI(url),userinfo = string(username_escaped,\":\",password_escaped)))\n\nds = NCDataset(url2)\n\nncvar = ds[\"analysed_sst\"];\nSST = ncvar[:,:,1]\nlon = ds[\"lon\"][:]\nlat = ds[\"lat\"][:]\nSST_time = ds[\"time\"][1]\n\nclf()\npcolormesh(lon,lat,nomissing(SST,NaN)')\ncbar = colorbar(orientation=\"horizontal\")\ncbar.set_label(ncvar.attrib[\"units\"])\ngca().set_aspect(1/cosd(mean(lat)))\n\ntitle(\"$(ncvar.attrib[\"long_name\"]) $SST_time\")","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"Alternatively, one can also create the files .netrc and .ncrc in the your home directory with your credentials as explained for the NASA EarthData example which allows you to manage your passwords in a central configuration file and reduce the risk to accidentally share your credentials.","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"(Image: example_SST_CMEMS.png)","category":"page"},{"location":"tutorials/#Data-from-NASA-EarthData","page":"Tutorials","title":"Data from NASA EarthData","text":"","category":"section"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"This example shows show to download data via OPeNDAP from the NASA EarthData which requires a username and password.","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"You need to be registered at https://urs.earthdata.nasa.gov/users/new to get your credentials.","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"The example requires NCDatasets 0.12.5.","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"Simply adding username and password to the URL for NASA Earth Data OPeNDPAP services is unfortunately not sufficient as it uses a different authentication mechanism.","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"Create a .netrc file with the following content in your home directory:","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"machine urs.earthdata.nasa.gov\n login YOUR_USERNAME\n password YOUR_PASSWORD","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"where YOUR_USERNAME and YOUR_PASSWORD is your Earth Data username and password.","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"Create a .ncrc file with the following content in your home directory[1]:","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"HTTP.NETRC=/home/abarth/.netrc","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"where HTTP.NETRC is the full path to your new .netrc file[2]. You can test whether your configuration files are correct independently of NCDatasets by using the tool ncdump:","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"ncdump -h \"https://opendap.earthdata.nasa.gov/providers/POCLOUD/collections/GHRSST%20Level%204%20MUR%20Global%20Foundation%20Sea%20Surface%20Temperature%20Analysis%20(v4.1)/granules/20190101090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1\"","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"This should return the metadata of the OPeNDAP resource:","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"netcdf \\20190101090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04 {\ndimensions:\n\tlat = 17999 ;\n\tlon = 36000 ;\n\ttime = 1 ;\nvariables:\n\tshort analysed_sst(time, lat, lon) ;\n[...]","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"This is the typical error message which is returned when the credentials are not configured properly:","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"syntax error, unexpected WORD_WORD, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR\ncontext: HTTP^ Basic: Access denied.","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"When there is an error on the server side (500 internal server error), you might get the following error:","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"syntax error, unexpected $end, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR\ncontext: ^\nncdump: https://opendap.earthdata.nasa.gov/providers/POCLOUD/collections/GHRSST%20Level%204%20MUR%20Global%20Foundation%20Sea%20Surface%20Temperature%20Analysis%20(v4.1)/granules/20190101090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1: NetCDF: Malformed or inaccessible DAP2 DDS or DAP4 DMR response","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"Here we use the GHRSST Level 4 MUR Global Foundation Sea Surface Temperature Analysis (v4.1) dataset. In the following example, we download the data via OPeNDAP for a chosen bounding box and given time instance.","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"using NCDatasets, PyPlot, Dates, Statistics\n\nurl = \"https://opendap.earthdata.nasa.gov/providers/POCLOUD/collections/GHRSST%20Level%204%20MUR%20Global%20Foundation%20Sea%20Surface%20Temperature%20Analysis%20(v4.1)/granules/20190101090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1\"\n\nds = NCDataset(url)\n\n# range of longitude\nlonr = (-6, 37.0)\n\n# range of latitude\nlatr = (29, 45.875)\n\nds_subset = NCDatasets.@select(\n ds[\"analysed_sst\"],\n $lonr[1] <= lon <= $lonr[2] && $latr[1] <= lat <= $latr[2])\n\nncvar = ds_subset[\"analysed_sst\"]\nSST = ncvar[:,:,1]\nlon = ds_subset[\"lon\"][:]\nlat = ds_subset[\"lat\"][:]\ntime = ds_subset[\"time\"][1]\n\n\nclf()\npcolormesh(lon,lat,nomissing(SST,NaN)');\ngca().set_aspect(1/cosd(mean(lat)))\n\ncbar = colorbar(orientation=\"horizontal\")\ncbar.set_label(ncvar.attrib[\"units\"])\n\nplt.title(\"$(ncvar.attrib[\"long_name\"]) $time\")","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"This script produces the following plot:","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"(Image: example_SST.png)","category":"page"},{"location":"tutorials/#Access-data-from-S3-object-storage","page":"Tutorials","title":"Access data from S3 object storage","text":"","category":"section"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"NASA EarthData is also available from AWS S3 object storage at the AWS region us-west-2. This example assumes that you have access to an AWS instance in this region (available through e.g. Pangeo Cloud after registration thanks to funding from the NASA's ACCESS/AWS Cloud Credits for Research program and the fine folks at Pangeo).","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"The S3 access tokens available from https://archive.podaac.earthdata.nasa.gov/s3credentials are only valid for 1 hour. It is therefore preferable to get this token programmatically using your EarthData username and password. The function earthdata_s3credentials is a translation of \"Sample script to receive credentials\" to Julia:","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"using AWS, Base64, JSON3, HTTP, NCDatasets, URIs\nusing AWS: @service\n@service S3\n\nfunction earthdata_s3credentials(\n username, password;\n credentials_url = \"https://archive.podaac.earthdata.nasa.gov/s3credentials\")\n\n origin = URI(credentials_url).scheme\n\n resp = HTTP.get(credentials_url,redirect=false)\n authorize_url = Dict(resp.headers)[\"Location\"]\n\n auth = base64encode(string(username,\":\",password))\n\n resp2 = HTTP.post(authorize_url,[\"Origin\" => origin],\n \"credentials=$auth\",redirect=false)\n redirect_url=Dict(resp2.headers)[\"Location\"]\n\n # cookie jar\n jar = Dict{String, Set{HTTP.Cookie}}()\n\n # raises a 500 error, as in the shell script, but we get a cookie!\n HTTP.get(redirect_url; cookies=true,\n cookiejar = jar, redirect = false,\n status_exception = false)\n\n resp4 = HTTP.get(credentials_url; cookies=true, cookiejar = jar, redirect = false)\n\n cred = JSON3.read(resp4.body)\n return cred\nend\n\n# add your credentials here (or get it programmatically from environment variables or a file)\nusername = \"...\"\npassword = \"...\"\n\ncred = earthdata_s3credentials(username,password)\n@info \"Token expires: $(cred.expiration)\"\n\n\nENV[\"AWS_ACCESS_KEY_ID\"] = cred.accessKeyId;\nENV[\"AWS_SECRET_ACCESS_KEY\"] = cred.secretAccessKey;\nENV[\"AWS_SESSION_TOKEN\"] = cred.sessionToken;\n\nc = AWS.global_aws_config();\n\nresp = S3.list_objects(\"podaac-ops-cumulus-protected\",\n Dict(\"prefix\" => \"MODIS_TERRA_L3_SST_MID-IR_DAILY_4KM_NIGHTTIME_V2019.0/\",\n \"delimiter\" => '/'))\n\n# download the first object\ndata = S3.get_object(\"podaac-ops-cumulus-protected\",resp[\"Contents\"][1][\"Key\"]);\n\n# load the NetCDF dataset\nds = NCDataset(\"temp-memory\",\"r\",memory = data)","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"Output:","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"CDataset: temp-memory\nGroup: /\n\nDimensions\n lat = 4320\n lon = 8640\n rgb = 3\n eightbitcolor = 256\n\nVariables\n sst4 (8640 × 4320)\n Datatype: Int16\n Dimensions: lon × lat\n Attributes:\n long_name = 4um Sea Surface Temperature\n[...]","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"The example requires NCDatasets 0.12.5 which allows one to read a NetCDF dataset directly from a vector of bytes in memory.","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"To debug, it is useful to run the aws shell command to list all keys in the buckets (it requires the AWS_* environment variables to be set):","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"aws s3 ls s3://podaac-ops-cumulus-protected/MODIS_TERRA_L3_SST_THERMAL_DAILY_4KM_NIGHTTIME_V2019.0/","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"[1]: Windows users need to create a .dodsrc configuration file (instead of the .ncrc file) and place it in the current working directory or set the HOME environment variable (see https://github.com/Unidata/netcdf-c/issues/2380). This NetCDF bug is likely to be fixed in NetCDF version 4.9.1.","category":"page"},{"location":"tutorials/","page":"Tutorials","title":"Tutorials","text":"[2]: More information is available at https://docs.unidata.ucar.edu/netcdf-c/4.8.1/md_auth.html.","category":"page"},{"location":"variables/#Variables","page":"Variables","title":"Variables","text":"","category":"section"},{"location":"variables/","page":"Variables","title":"Variables","text":"Variables (like e.g. CFVariable) are the quantities contained within a NetCDF dataset. See the Datasets page on how to obtain them from a dataset.","category":"page"},{"location":"variables/","page":"Variables","title":"Variables","text":"Different type of arrays are involved when working with NCDatasets. For instance assume that test.nc is a file with a Float32 variable called variable.","category":"page"},{"location":"variables/","page":"Variables","title":"Variables","text":"using NCDatasets\nds = NCDataset(\"test.nc\")\nncvar_cf = ds[\"variable\"]","category":"page"},{"location":"variables/","page":"Variables","title":"Variables","text":"The variable ncvar_cf has the type CFVariable. No data is actually loaded from disk, but you can query its size, number of dimensions, number elements, etc., using the functions size, ndims, length as if ncvar_cf was an ordinary Julia array.","category":"page"},{"location":"variables/","page":"Variables","title":"Variables","text":"To load the variable ncvar_cf in memory you can convert it into an array with:","category":"page"},{"location":"variables/","page":"Variables","title":"Variables","text":"data = Array(ncvar_cf)\n# or\ndata = ncvar_cf |> Array\n# or if ndims(ncvar_cf) == 2\ndata = ncvar_cf[:,:]","category":"page"},{"location":"variables/","page":"Variables","title":"Variables","text":"Since NCDatasets 0.13, the syntax ncvar_cf[:] flattens the array, and is not equivalent with the above (unless ncvar_cf is a vector).","category":"page"},{"location":"variables/","page":"Variables","title":"Variables","text":"You can only load sub-parts of it in memory via indexing each dimension:","category":"page"},{"location":"variables/","page":"Variables","title":"Variables","text":"ncvar_cf[1:5, 10:20]","category":"page"},{"location":"variables/","page":"Variables","title":"Variables","text":"A scalar variable can be loaded using [], for example:","category":"page"},{"location":"variables/","page":"Variables","title":"Variables","text":"using NCDatasets\nNCDataset(\"test_scalar.nc\",\"c\") do ds\n defVar(ds,\"scalar\",42,())\nend\n\nds = NCDataset(\"test_scalar.nc\")\nvalue = ds[\"scalar\"][] # 42","category":"page"},{"location":"variables/","page":"Variables","title":"Variables","text":"To load all a variable in a NetCDF file ignoring attributes like scale_factor, add_offset, _FillValue and time units one can use the property var or the function variable for example:","category":"page"},{"location":"variables/","page":"Variables","title":"Variables","text":"using NCDatasets\nusing Dates\ndata = [DateTime(2000,1,1), DateTime(2000,1,2)]\nNCDataset(\"test_file.nc\",\"c\") do ds\n defVar(ds,\"time\",data,(\"time\",), attrib = Dict(\n \"units\" => \"days since 2000-01-01\"))\nend;\n\nds = NCDataset(\"test_file.nc\")\nncvar = ds[\"time\"].var\n# or\nncvar = variable(ds,\"time\")\ndata = ncvar[:] # here [0., 1.]","category":"page"},{"location":"variables/","page":"Variables","title":"Variables","text":"The variable ncvar can be indexed in the same way as ncvar_cf explained above.","category":"page"},{"location":"variables/","page":"Variables","title":"Variables","text":"note: Note\nNCDatasets.Variable and NCDatasets.CFVariable implement the interface of AbstractArray. It is thus possible to call any function that accepts an AbstractArray. But functions like mean, sum (and many more) would load every element individually which is very inefficient for large fields read from disk. You should instead convert such a variable to a standard Julia Array and then do computations with it. See also the performance tips for more information.","category":"page"},{"location":"variables/","page":"Variables","title":"Variables","text":"The following functions are convenient for working with variables:","category":"page"},{"location":"variables/","page":"Variables","title":"Variables","text":"Base.size(v::NCDatasets.CFVariable)\ndimnames\ndimsize\nname\nrenameVar\nNCDataset(var::NCDatasets.CFVariable)\nnomissing\nfillvalue","category":"page"},{"location":"variables/#Base.size-Tuple{CommonDataModel.CFVariable}","page":"Variables","title":"Base.size","text":"sz = size(var::CFVariable)\n\nReturn a tuple of integers with the size of the variable var.\n\nnote: Note\nNote that the size of a variable can change, i.e. for a variable with an unlimited dimension.\n\n\n\n\n\n","category":"method"},{"location":"variables/#CommonDataModel.dimnames","page":"Variables","title":"CommonDataModel.dimnames","text":"dimnames(v::Variable)\n\nReturn a tuple of strings with the dimension names of the variable v.\n\n\n\n\n\nCommonDataModel.dimnames(v::AbstractVariable)\n\nReturn an iterable of the dimension names of the variable v.\n\n\n\n\n\ndimnames(v::CFVariable)\n\nReturn a tuple of strings with the dimension names of the variable v.\n\n\n\n\n\nCommonDatamodel.dimnames(ds::AbstractDataset)\n\nReturn an iterable of all dimension names in ds. This information can also be accessed using the property ds.dim:\n\nExamples\n\nds = NCDataset(\"results.nc\", \"r\");\ndimnames = keys(ds.dim)\n\n\n\n\n\n","category":"function"},{"location":"variables/#NCDatasets.dimsize","page":"Variables","title":"NCDatasets.dimsize","text":"dimsize(v::CFVariable)\n\nGet the size of a CFVariable as a named tuple of dimension → length.\n\n\n\n\n\n","category":"function"},{"location":"variables/#CommonDataModel.name","page":"Variables","title":"CommonDataModel.name","text":"name(ds::NCDataset)\n\nReturn the group name of the NCDataset ds\n\n\n\n\n\nname(v::Variable)\n\nReturn the name of the NetCDF variable v.\n\n\n\n\n\nCommonDatamodel.name(ds::AbstractDataset)\n\nName of the group of the data set ds. For a data set containing only a single group, this will be always the root group \"/\".\n\n\n\n\n\nCommonDataModel.name(v::AbstractVariable)\n\nReturn the name of the variable v as a string.\n\n\n\n\n\n","category":"function"},{"location":"variables/#NCDatasets.renameVar","page":"Variables","title":"NCDatasets.renameVar","text":"renameVar(ds::NCDataset,oldname,newname)\n\nRename the variable called oldname to newname.\n\n\n\n\n\n","category":"function"},{"location":"variables/#NCDatasets.NCDataset-Tuple{CommonDataModel.CFVariable}","page":"Variables","title":"NCDatasets.NCDataset","text":"mfds = NCDataset(fnames, mode = \"r\"; aggdim = nothing, deferopen = true,\n isnewdim = false,\n constvars = [])\n\nOpens a multi-file dataset in read-only \"r\" or append mode \"a\". fnames is a vector of file names.\n\nVariables are aggregated over the first unlimited dimension or over the dimension aggdim if specified. Variables without the dimensions aggdim are not aggregated. All variables containing the dimension aggdim are aggregated. The variable who do not contain the dimension aggdim are assumed constant.\n\nIf variables should be aggregated over a new dimension (not present in the NetCDF file), one should set isnewdim to true. All NetCDF files should have the same variables, attributes and groupes. Per default, all variables will have an additional dimension unless they are marked as constant using the constvars parameter.\n\nThe append mode is only implemented when deferopen is false. If deferopen is false, all files are opened at the same time. However the operating system might limit the number of open files. In Linux, the limit can be controled with the command ulimit.\n\nAll metadata (attributes and dimension length are assumed to be the same for all NetCDF files. Otherwise reading the attribute of a multi-file dataset would be ambiguous. An exception to this rule is the length of the dimension over which the data is aggregated. This aggregation dimension can varify from file to file.\n\nSetting the experimental flag _aggdimconstant to true means that the length of the aggregation dimension is constant. This speeds up the creating of a multi-file dataset as only the metadata of the first file has to be loaded.\n\nExamples:\n\nYou can use Glob.jl to make fnames from a file pattern, e.g.\n\nusing NCDatasets, Glob\nds = NCDataset(glob(\"ERA5_monthly3D_reanalysis_*.nc\"))\n\nAggregation over a new dimension:\n\nusing NCDatasets\nfor i = 1:3\n NCDataset(\"foo$i.nc\",\"c\") do ds\n defVar(ds,\"data\",[10., 11., 12., 13.], (\"lon\",))\n end\nend\n\nds = NCDataset([\"foo$i.nc\" for i = 1:3],aggdim = \"sample\", isnewdim = true)\nsize(ds[\"data\"])\n# output\n# (4, 3)\n\n\n\n\n\n","category":"method"},{"location":"variables/#NCDatasets.nomissing","page":"Variables","title":"NCDatasets.nomissing","text":"a = nomissing(da)\n\nReturn the values of the array da of type Array{Union{T,Missing},N} (potentially containing missing values) as a regular Julia array a of the same element type. It raises an error if the array contains at least one missing value.\n\n\n\n\n\na = nomissing(da,value)\n\nRetun the values of the array da of type AbstractArray{Union{T,Missing},N} as a regular Julia array a by replacing all missing value by value (converted to type T). This function is identical to coalesce.(da,T(value)) where T is the element type of da.\n\nExample:\n\njulia> nomissing([missing,1.,2.],NaN)\n# returns [NaN, 1.0, 2.0]\n\n\n\n\n\n","category":"function"},{"location":"variables/#CommonDataModel.fillvalue","page":"Variables","title":"CommonDataModel.fillvalue","text":"fv = fillvalue(v::Variable)\nfv = fillvalue(v::CFVariable)\n\nReturn the fill-value of the variable v.\n\n\n\n\n\nfillvalue(::Type{Int8})\nfillvalue(::Type{UInt8})\nfillvalue(::Type{Int16})\nfillvalue(::Type{UInt16})\nfillvalue(::Type{Int32})\nfillvalue(::Type{UInt32})\nfillvalue(::Type{Int64})\nfillvalue(::Type{UInt64})\nfillvalue(::Type{Float32})\nfillvalue(::Type{Float64})\nfillvalue(::Type{Char})\nfillvalue(::Type{String})\n\nDefault fill-value for the given type from NetCDF.\n\n\n\n\n\n","category":"function"},{"location":"variables/","page":"Variables","title":"Variables","text":"loadragged\nNCDatasets.load!","category":"page"},{"location":"variables/#NCDatasets.loadragged","page":"Variables","title":"NCDatasets.loadragged","text":" data = loadragged(ncvar,index::Union{Colon,UnitRange,Integer})\n\nLoad data from ncvar in the contiguous ragged array representation as a vector of vectors. It is typically used to load a list of profiles or time series of different length each.\n\nThe indexed ragged array representation is currently not supported.\n\n\n\n\n\n","category":"function"},{"location":"variables/#CommonDataModel.load!","page":"Variables","title":"CommonDataModel.load!","text":"NCDatasets.load!(ncvar::Variable, data, indices)\n\nLoads a NetCDF variables ncvar in-place and puts the result in data along the specified indices. One can use @inbounds annotate code where bounds checking can be elided by the compiler (which typically require type-stable code).\n\nusing NCDatasets\nds = NCDataset(\"file.nc\")\nncv = ds[\"vgos\"].var;\n# data must have the right shape and type\ndata = zeros(eltype(ncv),size(ncv));\nNCDatasets.load!(ncv,data,:,:,:)\n# or\n# @inbounds NCDatasets.load!(ncv,data,:,:,:)\nclose(ds)\n\n# loading a subset\ndata = zeros(5); # must have the right shape and type\nload!(ds[\"temp\"].var,data,:,1) # loads the 1st column\n\nnote: Note\nFor a netCDF variable of type NC_CHAR, the element type of the data array must be UInt8 and cannot be the julia Char type, because the julia Char type uses 4 bytes and the NetCDF NC_CHAR only 1 byte.\n\n\n\n\n\nCommonDataModel.load!(ncvar::CFVariable, data, buffer, indices)\n\nLoads a NetCDF (or other format) variables ncvar in-place and puts the result in data (an array of eltype(ncvar)) along the specified indices. buffer is a temporary array of the same size as data but the type should be eltype(ncv.var), i.e. the corresponding type in the files (before applying scale_factor, add_offset and masking fill values). Scaling and masking will be applied to the array data.\n\ndata and buffer can be the same array if eltype(ncvar) == eltype(ncvar.var).\n\nExample:\n\n# create some test array\nDataset(\"file.nc\",\"c\") do ds\n defDim(ds,\"time\",3)\n ncvar = defVar(ds,\"vgos\",Int16,(\"time\",),attrib = [\"scale_factor\" => 0.1])\n ncvar[:] = [1.1, 1.2, 1.3]\n # store 11, 12 and 13 as scale_factor is 0.1\nend\n\n\nds = Dataset(\"file.nc\")\nncv = ds[\"vgos\"];\n# data and buffer must have the right shape and type\ndata = zeros(eltype(ncv),size(ncv)); # here Vector{Float64}\nbuffer = zeros(eltype(ncv.var),size(ncv)); # here Vector{Int16}\nNCDatasets.load!(ncv,data,buffer,:,:,:)\nclose(ds)\n\n\n\n\n\n","category":"function"},{"location":"variables/#Creating-a-variable","page":"Variables","title":"Creating a variable","text":"","category":"section"},{"location":"variables/","page":"Variables","title":"Variables","text":"defVar","category":"page"},{"location":"variables/#CommonDataModel.defVar","page":"Variables","title":"CommonDataModel.defVar","text":"defVar(ds::NCDataset,name,vtype,dimnames; kwargs...)\ndefVar(ds::NCDataset,name,data,dimnames; kwargs...)\n\nDefine a variable with the name name in the dataset ds. vtype can be Julia types in the table below (with the corresponding NetCDF type). The parameter dimnames is a tuple with the names of the dimension. For scalar this parameter is the empty tuple (). The variable is returned (of the type CFVariable).\n\nInstead of providing the variable type one can directly give also the data data which will be used to fill the NetCDF variable. In this case, the dimensions with the appropriate size will be created as required using the names in dimnames.\n\nIf data is a vector or array of DateTime objects, then the dates are saved as double-precision floats and units \"days since 1900-01-01 00:00:00\" (unless a time unit is specifed with the attrib keyword as described below). Dates are converted to the default calendar in the CF conversion which is the mixed Julian/Gregorian calendar.\n\nKeyword arguments\n\nfillvalue: A value filled in the NetCDF file to indicate missing data. It will be stored in the _FillValue attribute. NCDatasets does not use implicitely the default NetCDF fill values when reading data.\nchunksizes: Vector integers setting the chunk size. The total size of a chunk must be less than 4 GiB.\ndeflatelevel: Compression level: 0 (default) means no compression and 9 means maximum compression. Each chunk will be compressed individually.\nshuffle: If true, the shuffle filter is activated which can improve the compression ratio.\nchecksum: The checksum method can be :fletcher32 or :nochecksum (checksumming is disabled, which is the default)\nattrib: An iterable of attribute name and attribute value pairs, for example a Dict, DataStructures.OrderedDict or simply a vector of pairs (see example below)\ntypename (string): The name of the NetCDF type required for vlen arrays\n\nchunksizes, deflatelevel, shuffle and checksum can only be set on NetCDF 4 files. Compression of strings and variable-length arrays is not supported by the underlying NetCDF library.\n\nNetCDF data types\n\nNetCDF Type Julia Type\nNC_BYTE Int8\nNC_UBYTE UInt8\nNC_SHORT Int16\nNC_INT Int32\nNC_INT64 Int64\nNC_FLOAT Float32\nNC_DOUBLE Float64\nNC_CHAR Char\nNC_STRING String\n\nDimension ordering\n\nThe data is stored in the NetCDF file in the same order as they are stored in memory. As julia uses the Column-major ordering for arrays, the order of dimensions will appear reversed when the data is loaded in languages or programs using Row-major ordering such as C/C++, Python/NumPy or the tools ncdump/ncgen (NetCDF CDL). NumPy can also use Column-major ordering but Row-major order is the default. For the column-major interpretation of the dimensions (as in Julia), the CF Convention recommends the order \"longitude\" (X), \"latitude\" (Y), \"height or depth\" (Z) and \"date or time\" (T) (if applicable). All other dimensions should, whenever possible, be placed to the right of the spatiotemporal dimensions.\n\nExample:\n\nIn this example, scale_factor and add_offset are applied when the data is saved.\n\njulia> using DataStructures\njulia> data = randn(3,5)\njulia> NCDataset(\"test_file.nc\",\"c\") do ds\n defVar(ds,\"temp\",data,(\"lon\",\"lat\"), attrib = OrderedDict(\n \"units\" => \"degree_Celsius\",\n \"add_offset\" => -273.15,\n \"scale_factor\" => 0.1,\n \"long_name\" => \"Temperature\"\n ))\n end;\n\nnote: Note\nIf the attributes _FillValue, missing_value, add_offset, scale_factor, units and calendar are used, they should be defined when calling defVar by using the parameter attrib as shown in the example above.\n\n\n\n\n\nv = CommonDataModel.defVar(ds::AbstractDataset,src::AbstractVariable)\nv = CommonDataModel.defVar(ds::AbstractDataset,name::SymbolOrString,src::AbstractVariable)\n\nDefines and return the variable in the data set ds copied from the variable src. The dimension name, attributes and data are copied from src as well as the variable name (unless provide by name).\n\n\n\n\n\n","category":"function"},{"location":"variables/#Storage-parameter-of-a-variable","page":"Variables","title":"Storage parameter of a variable","text":"","category":"section"},{"location":"variables/","page":"Variables","title":"Variables","text":"chunking\ndeflate\nchecksum","category":"page"},{"location":"variables/#CommonDataModel.chunking","page":"Variables","title":"CommonDataModel.chunking","text":"storage,chunksizes = chunking(v::Variable)\n\nReturn the storage type (:contiguous or :chunked) and the chunk sizes of the varable v. Note that chunking reports the same information as nc_inq_var_chunking and therefore considers variables with unlimited dimension as :contiguous.\n\n\n\n\n\nstorage,chunksizes = chunking(v::MFVariable)\n\nReturn the storage type (:contiguous or :chunked) and the chunk sizes of the varable v corresponding to the first file. If the first file in the collection is chunked then this storage attributes are returns. If not the first file is not contiguous, then multi-file variable is still reported as chunked with chunk size equal to the variable size.\n\n\n\n\n\n","category":"function"},{"location":"variables/#CommonDataModel.deflate","page":"Variables","title":"CommonDataModel.deflate","text":"isshuffled,isdeflated,deflate_level = deflate(v::Variable)\n\nReturn compression information of the variable v. If shuffle is true, then shuffling (byte interlacing) is activated. If deflate is true, then the data chunks (see chunking) are compressed using the compression level deflate_level (0 means no compression and 9 means maximum compression).\n\n\n\n\n\n","category":"function"},{"location":"variables/#CommonDataModel.checksum","page":"Variables","title":"CommonDataModel.checksum","text":"checksummethod = checksum(v::Variable)\n\nReturn the checksum method of the variable v which can be either be :fletcher32 or :nochecksum.\n\n\n\n\n\n","category":"function"},{"location":"variables/#Coordinate-variables-and-cell-boundaries","page":"Variables","title":"Coordinate variables and cell boundaries","text":"","category":"section"},{"location":"variables/","page":"Variables","title":"Variables","text":"coord\nbounds","category":"page"},{"location":"variables/#CommonDataModel.coord","page":"Variables","title":"CommonDataModel.coord","text":"cv = coord(v::Union{CFVariable,Variable},standard_name)\n\nFind the coordinate of the variable v by the standard name standard_name or some standardized heuristics based on units. If the heuristics fail to detect the coordinate, consider to modify the file to add the standard_name attribute. All dimensions of the coordinate must also be dimensions of the variable v.\n\nExample\n\nusing NCDatasets\nds = NCDataset(\"file.nc\")\nncv = ds[\"SST\"]\nlon = coord(ncv,\"longitude\")[:]\nlat = coord(ncv,\"latitude\")[:]\nv = ncv[:]\nclose(ds)\n\n\n\n\n\n","category":"function"},{"location":"variables/#CommonDataModel.bounds","page":"Variables","title":"CommonDataModel.bounds","text":"b = bounds(ncvar::NCDatasets.CFVariable)\n\nReturn the CFVariable corresponding to the bounds attribute of the variable ncvar. The time units and calendar from the ncvar are used but not the attributes controling the packing of data scale_factor, add_offset and _FillValue.\n\n\n\n\n\n","category":"function"},{"location":"issues/#Known-issues","page":"Known issues","title":"Known issues","text":"","category":"section"},{"location":"issues/#NetCDF:-Not-a-valid-data-type-or-_FillValue-type-mismatch","page":"Known issues","title":"NetCDF: Not a valid data type or _FillValue type mismatch","text":"","category":"section"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"Trying to define the _FillValue, produces the following error:","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"ERROR: LoadError: NCDatasets.NetCDFError(-45, \"NetCDF: Not a valid data type or _FillValue type mismatch\")","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"The error could be generated by a code like this:","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"using NCDatasets, DataStructures\n# ...\ntempvar = defVar(ds,\"temp\",Float32,(\"lonc\",\"latc\",\"time\"), attrib = OrderedDict(\n \"_FillValue\" => -9999.))","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"or","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"using NCDatasets\n# ...\ntempvar = defVar(ds,\"temp\",Float32,(\"lonc\",\"latc\",\"time\"), fillvalue = -9999.)","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"In fact, _FillValue must have the same data type as the corresponding variable. In the case above, tempvar is a 32-bit float and the number -9999. is a 64-bit float (aka double, which is the default floating point type in Julia). It is sufficient to convert the value -9999. to a 32-bit float -9999.f0 (or Float32(-9999.)).","category":"page"},{"location":"issues/#Defining-the-attributes-_FillValue,-add_offset,-scale_factor,-units-and-calendar","page":"Known issues","title":"Defining the attributes _FillValue, add_offset, scale_factor, units and calendar","text":"","category":"section"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"An error like Cannotconvertan object of type Missing (or similar) is generated by code like this:","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"v = defVar(ds,\"var_with_all_missing_data\",Float32,(\"lon\",))\nv.attrib[\"_FillValue\"] = fv\nv[1] = missing","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"This produces the following error:","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"ERROR: LoadError: MethodError: Cannot `convert` an object of type Missing to an object of type Float32\nClosest candidates are:\n convert(::Type{T}, ::T) where T<:Number at number.jl:6\n convert(::Type{T}, ::Number) where T<:Number at number.jl:7\n convert(::Type{T}, ::Base.TwicePrecision) where T<:Number at twiceprecision.jl:250\n ...\nStacktrace:\n [1] fill!(::SubArray{Float32,1,NCDatasets.CFVariable{Float32,1,NCDatasets.Variable{Float32,1},NCDatasets.Attributes},Tuple{UnitRange{Int64}},false}, ::Missing) at ./multidimensional.jl:865\n [2] copyto! at ./broadcast.jl:871 [inlined]\n [3] materialize!(::SubArray{Float32,1,NCDatasets.CFVariable{Float32,1,NCDatasets.Variable{Float32,1},NCDatasets.Attributes},Tuple{UnitRange{Int64}},false}, ::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{0},Nothing,typeof(identity),Tuple{Base.RefValue{Missing}}}) at ./broadcast.jl:822","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"One should use define the _FillValue (and similar attributes like add_offset, scale_factor, units and calendar affecting the type of the data) in the call of defVar:","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"using DataStructures\nv = defVar(ds,\"var_with_all_missing_data\",Int32,(\"lon\",), fillvalue = fv, attrib = OrderedDict(\n \"scale_factor\" => 0.1,\n \"add_offset\" => 0.1\n ))","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"This change was introduced in NCDatasets version 0.10","category":"page"},{"location":"issues/#Multiple-versions-of-HDF5-or-NetCDF-libraries","page":"Known issues","title":"Multiple versions of HDF5 or NetCDF libraries","text":"","category":"section"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"Having outdated versions of HDF5 or NetCDF libraries installed can be an issue on Windows if they are included in the system PATH environment variable. It is advised to adapt the system PATH to remove the locations containing these libraries.","category":"page"},{"location":"issues/#Using-a-custom-NetCDF-library","page":"Known issues","title":"Using a custom NetCDF library","text":"","category":"section"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"The NetCDF library libnetcdf.so is installed as an artifact via the package NetCDF_jll. You can override which libnetcdf.so gets loaded through the Preferences package, as follows:","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"# install these packages if necessary\nusing Preferences, NetCDF_jll\n\nset_preferences!(NetCDF_jll, \"libnetcdf_path\" => \"/path/to/libnetcdf.so.xyz\")","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"where /path/to/libnetcdf.so.xyz is the full path to the NetCDF library. This will create a LocalPreferences.toml file in your top-level project with the following content:","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"[NetCDF_jll]\nlibnetcdf_path = \"/path/to/libnetcdf.so.xyz\"","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"However, the dependencies of the library version libnetcdf.so.xyz (in particular libcurl.so and libmbedtls.so) should be compatible with the dependencies of julia (in the folder .../julia-x.y.z/lib/julia). On Linux, you can list the library dependencies with the shell command ldd, for example:","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"ldd /path/to/libnetcdf.so.xyz","category":"page"},{"location":"issues/#OPeNDAP-on-Windows-fails-with-Assertion-failed:-ocpanic","page":"Known issues","title":"OPeNDAP on Windows fails with Assertion failed: ocpanic","text":"","category":"section"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"On windows, NetCDF 4.7.4 can fail with this error:","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"Assertion failed: ocpanic((\"state->auth.curlflags.cookiejar != NULL\")), file ocinternal.c, line 566","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"when accessing OPeNDAP URLs, like these:","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"using NCDatasets\nds = NCDataset(\"https://thredds.jpl.nasa.gov/thredds/dodsC/ncml_aggregation/OceanTemperature/modis/terra/11um/4km/aggregate__MODIS_TERRA_L3_SST_THERMAL_DAILY_4KM_DAYTIME_V2019.0.ncml#fillmismatch\")","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"See also the issue report: https://github.com/Unidata/netcdf-c/issues/2380. The work-around is to create a .dodsrc in the current working directory with the content:","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"HTTP.COOKIEJAR=C:\\Users\\USERNAME\\AppData\\Local\\Temp\\","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"where USERNAME is your username. The directory should exist and be writable by the user. You can run pwd() to determine the current working directory. Note that the initial current working directory can be different depending you how you start julia (from the command line or from jupyter notebook for example). Julia need to be restarted after this file is placed in the your working directory.","category":"page"},{"location":"issues/#Using-non-official-julia-builds","page":"Known issues","title":"Using non-official julia builds","text":"","category":"section"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"Julia and NetCDF_jll have several common dependencies (curl, MbedTLS, zlib). Non-official julia builds will work only if they use exactly the same library version as those used to compile NetCDF. This is unlikely to be the case in general and outside of our control. Therefore non-official julia builds are not supported. Official julia builds are available at https://julialang.org/downloads/.","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"Using for example the julia package from on Arch Linux, leads to this error:","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"julia> using NCDatasets\nERROR: LoadError: InitError: could not load library \"/root/.julia/artifacts/461703969206dd426cc6b4d99f69f6ffab2a9779/lib/libnetcdf.so\"\n/usr/lib/julia/libcurl.so: version `CURL_4' not found (required by /root/.julia/artifacts/461703969206dd426cc6b4d99f69f6ffab2a9779/lib/libnetcdf.so)\nStacktrace:\n [1] macro expansion\n @ ~/.julia/packages/JLLWrappers/QpMQW/src/products/library_generators.jl:54 [inlined]\n [2] __init__()\n @ NetCDF_jll ~/.julia/packages/NetCDF_jll/BYHmI/src/wrappers/x86_64-linux-gnu.jl:12\n [3] top-level scope\n @ stdin:1\nduring initialization of module NetCDF_jll","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"You will likely have similar issues with julia installed from other package managers (like Debian/Ubuntu apt, Homebrew...). The only supported solution is to install the offical julia builds.","category":"page"},{"location":"issues/#version-CURL_4'-not-found","page":"Known issues","title":"version `CURL_4' not found","text":"","category":"section"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"If you see the following error:","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"ERROR: LoadError: InitError: could not load library \"/home/user/.julia/artifacts/461703969206dd426cc6b4d99f69f6ffab2a9779/lib/libnetcdf.so\"\n/usr/lib/x86_64-linux-gnu/libcurl.so: version `CURL_4' not found (required by /home/user/.julia/artifacts/461703969206dd426cc6b4d99f69f6ffab2a9779/lib/libnetcdf.so)","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"make sure that you are using the offical julia builds (see above) and that your LD_LIBRARY_PATH and LD_PRELOAD are empty. You can verify this by running the following commands in a terminal:","category":"page"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"echo $LD_PRELOAD\necho $LD_LIBRARY_PATH","category":"page"},{"location":"issues/#Corner-cases","page":"Known issues","title":"Corner cases","text":"","category":"section"},{"location":"issues/","page":"Known issues","title":"Known issues","text":"An attribute representing a vector with a single value (e.g. [1]) will be read back as a scalar (1) (same behavior in python netCDF4 1.3.1).\nNetCDF and Julia distinguishes between a vector of chars and a string, but both are returned as string for ease of use, in particular an attribute representing a vector of chars ['u','n','i','t','s'] will be read back as the string \"units\".\nWhile reading a NetCDF time variable, the dates are converted using the Julia's DateTime (based on the proleptic Gregorian calendar following the ISO 8601 standard) when possible. When data is written to a NetCDF file (without specifying the calendar), the dates are saved using the default calendar of the NetCDF CF convention (the mixed Julian/Gregorian calendar, called \"standard\") when possible. It is recommended that the time origin specified by the units is after 15 October 1582 in which case the mixed Julian/Gregorian calendar is identical to the proleptic Gregorian calendar.","category":"page"},{"location":"attributes/#Attributes","page":"Attributes","title":"Attributes","text":"","category":"section"},{"location":"attributes/","page":"Attributes","title":"Attributes","text":"The NetCDF dataset (as returned by NCDataset or NetCDF groups) and the NetCDF variables (as returned by getindex, variable or defVar) have the field attrib which has the type NCDatasets.Attributes and behaves like a julia dictionary.","category":"page"},{"location":"attributes/","page":"Attributes","title":"Attributes","text":"getindex(a::NCDatasets.Attributes,name::AbstractString)\nsetindex!(a::NCDatasets.Attributes,data,name::AbstractString)\nkeys(a::NCDatasets.Attributes)\ndelete!(a::NCDatasets.Attributes,name::AbstractString)","category":"page"},{"location":"attributes/#Base.getindex-Tuple{CommonDataModel.Attributes, AbstractString}","page":"Attributes","title":"Base.getindex","text":"getindex(a::Attributes,name::SymbolOrString)\n\nReturn the value of the attribute called name from the attribute list a. Generally the attributes are loaded by indexing, for example:\n\nusing NCDatasets\nds = NCDataset(\"file.nc\")\ntitle = ds.attrib[\"title\"]\n\n\n\n\n\n","category":"method"},{"location":"attributes/#Base.setindex!-Tuple{CommonDataModel.Attributes, Any, AbstractString}","page":"Attributes","title":"Base.setindex!","text":"Base.setindex!(a::Attributes,data,name::SymbolOrString)\n\nSet the attribute called name to the value data in the attribute list a. data can be a vector or a scalar. A scalar is handeld as a vector with one element in the NetCDF data model.\n\nGenerally the attributes are defined by indexing, for example:\n\nds = NCDataset(\"file.nc\",\"c\")\nds.attrib[\"title\"] = \"my title\"\nclose(ds)\n\n\n\n\n\n","category":"method"},{"location":"attributes/#Base.keys-Tuple{CommonDataModel.Attributes}","page":"Attributes","title":"Base.keys","text":"Base.keys(a::Attributes)\n\nReturn a list of the names of all attributes.\n\n\n\n\n\n","category":"method"},{"location":"attributes/#Base.delete!-Tuple{CommonDataModel.Attributes, AbstractString}","page":"Attributes","title":"Base.delete!","text":"Base.delete!(a::Attributes, name)\n\nDelete the attribute name from the attribute list a.\n\n\n\n\n\n","category":"method"},{"location":"attributes/","page":"Attributes","title":"Attributes","text":"Loading all attributes as a Dict can be achieved by passing ds.attrib (where ds is the NCDataset) as argument to Dict.","category":"page"},{"location":"attributes/","page":"Attributes","title":"Attributes","text":"using NCDatasets\nncfile = download(\"https://www.unidata.ucar.edu/software/netcdf/examples/sresa1b_ncar_ccsm3-example.nc\");\nds = NCDataset(ncfile);\nattributes_as_dictionary = Dict(ds.attrib)\ntypeof(attributes_as_dictionary)\n# returns Dict{String,Any}","category":"page"},{"location":"attributes/#Possible-type-promotion-in-Julia","page":"Attributes","title":"Possible type promotion in Julia","text":"","category":"section"},{"location":"attributes/","page":"Attributes","title":"Attributes","text":"There is a subtle problem with the following code:","category":"page"},{"location":"attributes/","page":"Attributes","title":"Attributes","text":"ncv1 = defVar(ds,\"v1\", UInt8, (\"longitude\", \"latitude\", \"time\"), attrib = [\n \"add_offset\" => -1.0,\n \"scale_factor\" => 5.0,\n \"_FillValue\" => UInt8(255),\n])","category":"page"},{"location":"attributes/","page":"Attributes","title":"Attributes","text":"Julia effectively promotes the _FillValue to Float64 which leads to a \"NetCDF: Not a valid data type or _FillValue type mismatch\" as the fillvalue has to have exactly the same type as the NetCDF data type. Other parameters could be equally promoted.","category":"page"},{"location":"attributes/","page":"Attributes","title":"Attributes","text":"[\n \"add_offset\" => -1.0,\n \"scale_factor\" => 5.0,\n \"_FillValue\" => UInt8(255),\n]\n# returns\n# 3-element Array{Pair{String,Float64},1}:\n# \"add_offset\" => -1.0\n# \"scale_factor\" => 5.0\n# \"_FillValue\" => 255.0","category":"page"},{"location":"attributes/","page":"Attributes","title":"Attributes","text":"Note the type of the second element of the Pair.","category":"page"},{"location":"attributes/","page":"Attributes","title":"Attributes","text":"Using a Julia Dict does not show this behaviour:","category":"page"},{"location":"attributes/","page":"Attributes","title":"Attributes","text":"ncv1 = defVar(ds,\"v1\", UInt8, (\"longitude\", \"latitude\", \"time\"), attrib = Dict(\n \"add_offset\" => -1.0,\n \"scale_factor\" => 5.0,\n \"_FillValue\" => UInt8(255),\n))","category":"page"},{"location":"attributes/","page":"Attributes","title":"Attributes","text":"Note that Dict does not perserve the order of the attributes. Therefore an OrderedDict from the package DataStructures is preferable.","category":"page"},{"location":"attributes/","page":"Attributes","title":"Attributes","text":"Or one could use simply the fillvalue parameter of defVar.","category":"page"},{"location":"attributes/","page":"Attributes","title":"Attributes","text":"ncv1 = defVar(ds,\"v1\", UInt8, (\"longitude\", \"latitude\", \"time\"), fillvalue = UInt8(255), attrib = [\n \"add_offset\" => -1.0,\n \"scale_factor\" => 5.0,\n])","category":"page"},{"location":"dimensions/#Dimensions","page":"Dimensions","title":"Dimensions","text":"","category":"section"},{"location":"dimensions/","page":"Dimensions","title":"Dimensions","text":"In the NetCDF data model, dimensions have names and a length (but possibly an unlimited length) and are defined for a NetCDF dataset (or group). For a given Variable or CFVariable,the names of the corresponding dimensions are obtained with using dimnames.","category":"page"},{"location":"dimensions/","page":"Dimensions","title":"Dimensions","text":"keys(d::NCDatasets.Dimensions)\nhaskey(a::NCDatasets.NCIterable,name::AbstractString)\ndefDim\nunlimited(d::NCDatasets.Dimensions)\nsetindex!(d::NCDatasets.Dimensions,len,name::AbstractString)","category":"page"},{"location":"dimensions/#Base.haskey-Tuple{NCDatasets.AbstractNCDataset, AbstractString}","page":"Dimensions","title":"Base.haskey","text":"haskey(ds::NCDataset,name)\nhaskey(d::Dimensions,name)\nhaskey(ds::Attributes,name)\n\nReturn true if the NCDataset ds (or dimension/attribute list) has a variable (dimension/attribute) with the name name. For example:\n\nds = NCDataset(\"/tmp/test.nc\",\"r\")\nif haskey(ds,\"temperature\")\n println(\"The file has a variable 'temperature'\")\nend\n\nif haskey(ds.dim,\"lon\")\n println(\"The file has a dimension 'lon'\")\nend\n\nThis example checks if the file /tmp/test.nc has a variable with the name temperature and a dimension with the name lon.\n\n\n\n\n\n","category":"method"},{"location":"dimensions/#CommonDataModel.defDim","page":"Dimensions","title":"CommonDataModel.defDim","text":"defDim(ds::NCDataset,name,len)\n\nDefine a dimension in the data set ds with the given name and length len. If len is the special value Inf, then the dimension is considered as unlimited, i.e. it will grow as data is added to the NetCDF file.\n\nFor example:\n\nusing NCDatasets\nds = NCDataset(\"/tmp/test.nc\",\"c\")\ndefDim(ds,\"lon\",100)\n# [...]\nclose(ds)\n\nThis defines the dimension lon with the size 100.\n\nTo create a variable with an unlimited dimensions use for example:\n\nusing NCDatasets\nds = NCDataset(\"/tmp/test2.nc\",\"c\")\ndefDim(ds,\"lon\",10)\ndefDim(ds,\"lat\",10)\ndefDim(ds,\"time\",Inf)\ndefVar(ds,\"unlimited_variable\",Float64,(\"lon\",\"lat\",\"time\"))\n@show ds.dim[\"time\"]\n# returns 0 as no data is added\nds[\"unlimited_variable\"][:,:,:] = randn(10,10,4)\n@show ds.dim[\"time\"]\n# returns now 4 as 4 time slice have been added\nclose(ds)\n\n\n\n\n\nCommonDatamodel.defDim(ds::AbstractDataset,name::SymbolOrString,len)\n\nCreate dimension with the name name in the data set ds with the length len. len can be Inf for unlimited dimensions.\n\n\n\n\n\n","category":"function"},{"location":"dimensions/","page":"Dimensions","title":"Dimensions","text":"One can iterate over a list of dimensions as follows:","category":"page"},{"location":"dimensions/","page":"Dimensions","title":"Dimensions","text":"for (dimname,dim) in ds.dim\n # all dimensions\n @show (dimname,dim)\nend","category":"page"},{"location":"#NCDatasets.jl","page":"Introduction","title":"NCDatasets.jl","text":"","category":"section"},{"location":"","page":"Introduction","title":"Introduction","text":"Documentation for NCDatasets.jl, a Julia package for loading and writing NetCDF (Network Common Data Form) files.","category":"page"},{"location":"#Installation","page":"Introduction","title":"Installation","text":"","category":"section"},{"location":"","page":"Introduction","title":"Introduction","text":"Inside the Julia shell, you can download and install using the following commands:","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"using Pkg\nPkg.add(\"NCDatasets\")","category":"page"},{"location":"#Latest-development-version","page":"Introduction","title":"Latest development version","text":"","category":"section"},{"location":"","page":"Introduction","title":"Introduction","text":"If you want to try the latest development version, again go into package manager mode and simply type","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"using Pkg\nPkg.add(PackageSpec(name=\"NCDatasets\", rev=\"master\"))","category":"page"},{"location":"#Contents","page":"Introduction","title":"Contents","text":"","category":"section"},{"location":"","page":"Introduction","title":"Introduction","text":"To get started quickly see the Quickstart section. Otherwise see the following pages for details:","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"Datasets : reading/writing NetCDF datasets (including NetCDF groups) and examining their contents.\nDimensions : accessing/creating NetCDF dimensions\nVariables : accessing/examining the variables (or dimensions) stored within a NetCDF dataset.\nAttributes : accessing/creating NetCDF attributes\nSee Performance tips, Known issues, Experimental features for more information.","category":"page"},{"location":"#Quickstart","page":"Introduction","title":"Quickstart","text":"","category":"section"},{"location":"","page":"Introduction","title":"Introduction","text":"This is a quickstart guide that outlines basic loading, reading, etc. usage. For more details please see the individual pages of the documentation.","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"Explore the content of a netCDF file\nLoad a netCDF file\nCreate a netCDF file\nEdit an existing netCDF file\nCreate a netCDF file using the metadata of an existing netCDF file as template\nGet one or several variables by specifying the value of an attribute\nLoad a file with unknown structure","category":"page"},{"location":"#Explore-the-content-of-a-netCDF-file","page":"Introduction","title":"Explore the content of a netCDF file","text":"","category":"section"},{"location":"","page":"Introduction","title":"Introduction","text":"Before reading the data from a netCDF file, it is often useful to explore the list of variables and attributes defined in it.","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"For interactive use, the following commands (without ending semicolon) display the content of the file similarly to ncdump -h file.nc:","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"using NCDatasets\nds = NCDataset(\"file.nc\")","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"which produces a listing like:","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"NCDataset: file.nc\nGroup: /\n\nDimensions\n time = 115\n\nVariables\n time (115)\n Datatype: Float64\n Dimensions: time\n Attributes:\n calendar = gregorian\n standard_name = time\n units = days since 1950-01-01 00:00:00\n[...]","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"This creates the central structure of NCDatasets.jl, NCDataset, which represents the contents of the netCDF file (without immediately loading everything in memory).","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"The following displays the information just for the variable varname:","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"ds[\"varname\"]","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"To get a list of global attributes, you can use:","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"ds.attrib","category":"page"},{"location":"#Load-a-netCDF-file","page":"Introduction","title":"Load a netCDF file","text":"","category":"section"},{"location":"","page":"Introduction","title":"Introduction","text":"Loading a variable with known structure can be achieved by accessing the variables and attributes directly by their name.","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"# The mode \"r\" stands for read-only. The mode \"r\" is the default mode and the parameter can be omitted.\nds = NCDataset(\"/tmp/test.nc\",\"r\")\nv = ds[\"temperature\"]\n\n# load a subset\nsubdata = v[10:30,30:5:end]\n\n# load all data\ndata = v[:,:]\n\n# load all data ignoring attributes like scale_factor, add_offset, _FillValue and time units\ndata2 = v.var[:,:]\n\n\n# load an attribute\nunit = v.attrib[\"units\"]\nclose(ds)","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"In the example above, the subset can also be loaded with:","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"subdata = NCDataset(\"/tmp/test.nc\")[\"temperature\"][10:30,30:5:end]","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"This might be useful in an interactive session. However, the file test.nc is not closed, which can be a problem if you open many files. On Linux the number of opened files is often limited to 1024 (soft limit). If you write to a file, you should also always close the file to make sure that the data is properly written to the disk. (open files will get closed eventually when the dataset variable is finalized by julia's garbage collector).","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"An alternative way to ensure the file has been closed is to use a do block: the file will be closed automatically when leaving the block.","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"data = NCDataset(filename,\"r\") do ds\n ds[\"temperature\"][:,:]\nend # ds is closed","category":"page"},{"location":"#Create-a-netCDF-file","page":"Introduction","title":"Create a netCDF file","text":"","category":"section"},{"location":"","page":"Introduction","title":"Introduction","text":"The following gives an example of how to create a netCDF file by defining dimensions, variables and attributes.","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"using NCDatasets\n# This creates a new NetCDF file /tmp/test.nc.\n# The mode \"c\" stands for creating a new file (clobber)\nds = NCDataset(\"/tmp/test.nc\",\"c\")\n\n# Define the dimension \"lon\" and \"lat\" with the size 100 and 110 resp.\ndefDim(ds,\"lon\",100)\ndefDim(ds,\"lat\",110)\n\n# Define a global attribute\nds.attrib[\"title\"] = \"this is a test file\"\n\n# Define the variables temperature\nv = defVar(ds,\"temperature\",Float32,(\"lon\",\"lat\"))\n\n# Generate some example data\ndata = [Float32(i+j) for i = 1:100, j = 1:110]\n\n# write a single column\nv[:,1] = data[:,1]\n\n# write a the complete data set\nv[:,:] = data\n\n# write attributes\nv.attrib[\"units\"] = \"degree Celsius\"\nv.attrib[\"comments\"] = \"this is a string attribute with Unicode Ω ∈ ∑ ∫ f(x) dx\"\n\nclose(ds)","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"An equivalent way to create the previous netCDF would be the following code:","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"using NCDatasets\nusing DataStructures\ndata = [Float32(i+j) for i = 1:100, j = 1:110]\n\nDataset(\"/tmp/test2.nc\",\"c\",attrib = OrderedDict(\"title\" => \"this is a test file\")) do ds\n # Define the variable temperature. The dimension \"lon\" and \"lat\" with the\n # size 100 and 110 resp are implicetly created\n defVar(ds,\"temperature\",data,(\"lon\",\"lat\"), attrib = OrderedDict(\n \"units\" => \"degree Celsius\",\n \"comments\" => \"this is a string attribute with Unicode Ω ∈ ∑ ∫ f(x) dx\"\n ))\nend","category":"page"},{"location":"#Edit-an-existing-netCDF-file","page":"Introduction","title":"Edit an existing netCDF file","text":"","category":"section"},{"location":"","page":"Introduction","title":"Introduction","text":"When you need to modify the variables or the attributes of a netCDF, you have to open it with the \"a\" option. Here, for instance, we add a global attribute creator to the file created in the previous step.","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"ds = NCDataset(\"/tmp/test.nc\",\"a\")\nds.attrib[\"creator\"] = \"your name\"\nclose(ds);","category":"page"},{"location":"#Create-a-netCDF-file-using-the-metadata-of-an-existing-netCDF-file-as-template","page":"Introduction","title":"Create a netCDF file using the metadata of an existing netCDF file as template","text":"","category":"section"},{"location":"","page":"Introduction","title":"Introduction","text":"The utility function ncgen generates the Julia code that would produce a netCDF file with the same metadata as a template netCDF file. It is thus similar to the command line tool ncgen.","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"# download example file\nncfile = download(\"https://www.unidata.ucar.edu/software/netcdf/examples/sresa1b_ncar_ccsm3-example.nc\")\n# generate Julia code\nncgen(ncfile)","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"The produces the Julia code (only the beginning of the code is shown):","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"ds = NCDataset(\"filename.nc\",\"c\")\n# Dimensions\n\nds.dim[\"lat\"] = 128;\nds.dim[\"lon\"] = 256;\nds.dim[\"bnds\"] = 2;\nds.dim[\"plev\"] = 17;\nds.dim[\"time\"] = 1;\n\n# Declare variables\n\nncarea = defVar(ds,\"area\", Float32, (\"lon\", \"lat\"))\nncarea.attrib[\"long_name\"] = \"Surface area\";\nncarea.attrib[\"units\"] = \"meter2\";\n# ...","category":"page"},{"location":"#Get-one-or-several-variables-by-specifying-the-value-of-an-attribute","page":"Introduction","title":"Get one or several variables by specifying the value of an attribute","text":"","category":"section"},{"location":"","page":"Introduction","title":"Introduction","text":"The variable names are not always standardized. For example, for the longitude we can find: lon, LON, longitude, ...","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"The solution implemented in the function varbyattrib consists in searching for the variables that have specified value for a given attribute.","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"nclon = varbyattrib(ds, standard_name = \"longitude\");","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"will return the list of variables of the dataset ds that have \"longitude\" as standard name. To directly load the data of the first variable with the attribute standard_name equal to \"longitude\" one can do the following:","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"data = varbyattrib(ds, standard_name = \"longitude\")[1][:]","category":"page"},{"location":"#Load-a-file-with-unknown-structure","page":"Introduction","title":"Load a file with unknown structure","text":"","category":"section"},{"location":"","page":"Introduction","title":"Introduction","text":"If the structure of the netCDF file is not known before-hand, the program must check if a variable or attribute exists (with the haskey function) before loading it or alternatively place the loading in a try-catch block. It is also possible to iterate over all variables or attributes (global attributes or variable attributes) in the same syntax as iterating over a dictionary. However, unlike Julia dictionaries, the order of the attributes and variables is preserved and presented as they are stored in the netCDF file.","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"# Open a file as read-only\nds = NCDataset(\"/tmp/test.nc\",\"r\")\n\n# check if a file has a variable with a given name\nif haskey(ds,\"temperature\")\n println(\"The file has a variable 'temperature'\")\nend\n\n# get a list of all variable names\n@show keys(ds)\n\n# iterate over all variables\nfor (varname,var) in ds\n @show (varname,size(var))\nend\n\n# query size of a variable (without loading it)\nv = ds[\"temperature\"]\n@show size(v)\n\n# similar for global and variable attributes\n\nif haskey(ds.attrib,\"title\")\n println(\"The file has the global attribute 'title'\")\nend\n\n# get an list of all attribute names\n@show keys(ds.attrib)\n\n# iterate over all attributes\nfor (attname,attval) in ds.attrib\n @show (attname,attval)\nend\n\n# get the attribute \"units\" of the variable v\n# but return the default value (here \"adimensional\")\n# if the attribute does not exists\n\nunits = get(v,\"units\",\"adimensional\")\nclose(ds)","category":"page"}] } diff --git a/dev/tutorials/index.html b/dev/tutorials/index.html index ddbb7f71..76731664 100644 --- a/dev/tutorials/index.html +++ b/dev/tutorials/index.html @@ -18,7 +18,7 @@ SST = ncvar[:,:,1] lon = ds["lon"][:] lat = ds["lat"][:] -time = ds["time"][1] +SST_time = ds["time"][1] clf() pcolormesh(lon,lat,nomissing(SST,NaN)') @@ -26,7 +26,7 @@ cbar.set_label(ncvar.attrib["units"]) gca().set_aspect(1/cosd(mean(lat))) -title("$(ncvar.attrib["long_name"]) $time")

      Alternatively, one can also create the files .netrc and .ncrc in the your home directory with your credentials as explained for the NASA EarthData example which allows you to manage your passwords in a central configuration file and reduce the risk to accidentally share your credentials.

      example_SST_CMEMS.png

      Data from NASA EarthData

      This example shows show to download data via OPeNDAP from the NASA EarthData which requires a username and password.

      You need to be registered at https://urs.earthdata.nasa.gov/users/new to get your credentials.

      The example requires NCDatasets 0.12.5.

      Simply adding username and password to the URL for NASA Earth Data OPeNDPAP services is unfortunately not sufficient as it uses a different authentication mechanism.

      Create a .netrc file with the following content in your home directory:

      machine urs.earthdata.nasa.gov
+title("$(ncvar.attrib["long_name"]) $SST_time")

      Alternatively, one can also create the files .netrc and .ncrc in the your home directory with your credentials as explained for the NASA EarthData example which allows you to manage your passwords in a central configuration file and reduce the risk to accidentally share your credentials.

      example_SST_CMEMS.png

      Data from NASA EarthData

      This example shows show to download data via OPeNDAP from the NASA EarthData which requires a username and password.

      You need to be registered at https://urs.earthdata.nasa.gov/users/new to get your credentials.

      The example requires NCDatasets 0.12.5.

      Simply adding username and password to the URL for NASA Earth Data OPeNDPAP services is unfortunately not sufficient as it uses a different authentication mechanism.

      Create a .netrc file with the following content in your home directory:

      machine urs.earthdata.nasa.gov
     login YOUR_USERNAME
     password YOUR_PASSWORD

      where YOUR_USERNAME and YOUR_PASSWORD is your Earth Data username and password.

      Create a .ncrc file with the following content in your home directory[1]:

      HTTP.NETRC=/home/abarth/.netrc

      where HTTP.NETRC is the full path to your new .netrc file[2]. You can test whether your configuration files are correct independently of NCDatasets by using the tool ncdump:

      ncdump -h "https://opendap.earthdata.nasa.gov/providers/POCLOUD/collections/GHRSST%20Level%204%20MUR%20Global%20Foundation%20Sea%20Surface%20Temperature%20Analysis%20(v4.1)/granules/20190101090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1"

      This should return the metadata of the OPeNDAP resource:

      netcdf \20190101090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04 {
 dimensions:
@@ -138,4 +138,4 @@
     Dimensions:  lon × lat
     Attributes:
      long_name            = 4um Sea Surface Temperature
-[...]

      The example requires NCDatasets 0.12.5 which allows one to read a NetCDF dataset directly from a vector of bytes in memory.

      To debug, it is useful to run the aws shell command to list all keys in the buckets (it requires the AWS_* environment variables to be set):

      aws s3 ls s3://podaac-ops-cumulus-protected/MODIS_TERRA_L3_SST_THERMAL_DAILY_4KM_NIGHTTIME_V2019.0/
      +[...]

      The example requires NCDatasets 0.12.5 which allows one to read a NetCDF dataset directly from a vector of bytes in memory.

      To debug, it is useful to run the aws shell command to list all keys in the buckets (it requires the AWS_* environment variables to be set):

      aws s3 ls s3://podaac-ops-cumulus-protected/MODIS_TERRA_L3_SST_THERMAL_DAILY_4KM_NIGHTTIME_V2019.0/
      diff --git a/dev/variables/index.html b/dev/variables/index.html index c161b882..88a7cd06 100644 --- a/dev/variables/index.html +++ b/dev/variables/index.html @@ -23,7 +23,8 @@ ncvar = ds["time"].var # or ncvar = variable(ds,"time") -data = ncvar[:] # here [0., 1.]

      The variable ncvar can be indexed in the same way as ncvar_cf explained above.

      Note

      NCDatasets.Variable and NCDatasets.CFVariable implement the interface of AbstractArray. It is thus possible to call any function that accepts an AbstractArray. But functions like mean, sum (and many more) would load every element individually which is very inefficient for large fields read from disk. You should instead convert such a variable to a standard Julia Array and then do computations with it. See also the performance tips for more information.

      The following functions are convenient for working with variables:

      Base.size — Method
      sz = size(var::CFVariable)

      Return a tuple of integers with the size of the variable var.

      Note

      Note that the size of a variable can change, i.e. for a variable with an unlimited dimension.

      CommonDataModel.dimnames — Function
      dimnames(v::Variable)

      Return a tuple of strings with the dimension names of the variable v.

      source
      dimnames(v::CFVariable)

      Return a tuple of strings with the dimension names of the variable v.

      source
      CommonDataModel.dimnames(v::AbstractVariable)

      Return an iterable of the dimension names of the variable v.

      CommonDatamodel.dimnames(ds::AbstractDataset)

      Return an iterable of all dimension names in ds.

      NCDatasets.dimsize — Function
      dimsize(v::CFVariable)

      Get the size of a CFVariable as a named tuple of dimension → length.

      source
      CommonDataModel.name — Function
      name(ds::NCDataset)

      Return the group name of the NCDataset ds

      source
      name(v::Variable)

      Return the name of the NetCDF variable v.

      source
      CommonDatamodel.name(ds::AbstractDataset)

      Name of the group of the data set ds. For a data set containing only a single group, this will be always the root group "/".

      CommonDataModel.name(v::AbstractVariable)

      Return the name of the variable v as a string.

      NCDatasets.NCDataset — Method
      mfds = NCDataset(fnames, mode = "r"; aggdim = nothing, deferopen = true,
+data = ncvar[:] # here [0., 1.]

      The variable ncvar can be indexed in the same way as ncvar_cf explained above.

      Note

      NCDatasets.Variable and NCDatasets.CFVariable implement the interface of AbstractArray. It is thus possible to call any function that accepts an AbstractArray. But functions like mean, sum (and many more) would load every element individually which is very inefficient for large fields read from disk. You should instead convert such a variable to a standard Julia Array and then do computations with it. See also the performance tips for more information.

      The following functions are convenient for working with variables:

      Base.size — Method
      sz = size(var::CFVariable)

      Return a tuple of integers with the size of the variable var.

      Note

      Note that the size of a variable can change, i.e. for a variable with an unlimited dimension.

      CommonDataModel.dimnames — Function
      dimnames(v::Variable)

      Return a tuple of strings with the dimension names of the variable v.

      source
      CommonDataModel.dimnames(v::AbstractVariable)

      Return an iterable of the dimension names of the variable v.

      dimnames(v::CFVariable)

      Return a tuple of strings with the dimension names of the variable v.

      CommonDatamodel.dimnames(ds::AbstractDataset)

      Return an iterable of all dimension names in ds. This information can also be accessed using the property ds.dim:

      Examples

      ds = NCDataset("results.nc", "r");
+dimnames = keys(ds.dim)
      NCDatasets.dimsize — Function
      dimsize(v::CFVariable)

      Get the size of a CFVariable as a named tuple of dimension → length.

      source
      CommonDataModel.name — Function
      name(ds::NCDataset)

      Return the group name of the NCDataset ds

      source
      name(v::Variable)

      Return the name of the NetCDF variable v.

      source
      CommonDatamodel.name(ds::AbstractDataset)

      Name of the group of the data set ds. For a data set containing only a single group, this will be always the root group "/".

      CommonDataModel.name(v::AbstractVariable)

      Return the name of the variable v as a string.

      NCDatasets.NCDataset — Method
      mfds = NCDataset(fnames, mode = "r"; aggdim = nothing, deferopen = true,
               isnewdim = false,
               constvars = [])

      Opens a multi-file dataset in read-only "r" or append mode "a". fnames is a vector of file names.

      Variables are aggregated over the first unlimited dimension or over the dimension aggdim if specified. Variables without the dimensions aggdim are not aggregated. All variables containing the dimension aggdim are aggregated. The variable who do not contain the dimension aggdim are assumed constant.

      If variables should be aggregated over a new dimension (not present in the NetCDF file), one should set isnewdim to true. All NetCDF files should have the same variables, attributes and groupes. Per default, all variables will have an additional dimension unless they are marked as constant using the constvars parameter.

      The append mode is only implemented when deferopen is false. If deferopen is false, all files are opened at the same time. However the operating system might limit the number of open files. In Linux, the limit can be controled with the command ulimit.

      All metadata (attributes and dimension length are assumed to be the same for all NetCDF files. Otherwise reading the attribute of a multi-file dataset would be ambiguous. An exception to this rule is the length of the dimension over which the data is aggregated. This aggregation dimension can varify from file to file.

      Setting the experimental flag _aggdimconstant to true means that the length of the aggregation dimension is constant. This speeds up the creating of a multi-file dataset as only the metadata of the first file has to be loaded.

      Examples:

      You can use Glob.jl to make fnames from a file pattern, e.g.

      using NCDatasets, Glob
 ds = NCDataset(glob("ERA5_monthly3D_reanalysis_*.nc"))

      Aggregation over a new dimension:

      using NCDatasets
@@ -36,8 +37,9 @@
 ds = NCDataset(["foo$i.nc" for i = 1:3],aggdim = "sample", isnewdim = true)
 size(ds["data"])
 # output
-# (4, 3)
      source
      NCDatasets.nomissing — Function
      a = nomissing(da)

      Return the values of the array da of type Array{Union{T,Missing},N} (potentially containing missing values) as a regular Julia array a of the same element type. It raises an error if the array contains at least one missing value.

      source
      a = nomissing(da,value)

      Retun the values of the array da of type AbstractArray{Union{T,Missing},N} as a regular Julia array a by replacing all missing value by value (converted to type T). This function is identical to coalesce.(da,T(value)) where T is the element type of da.

      Example:

      julia> nomissing([missing,1.,2.],NaN)
-# returns [NaN, 1.0, 2.0]
      source
      NCDatasets.nomissing — Function
      a = nomissing(da)

      Return the values of the array da of type Array{Union{T,Missing},N} (potentially containing missing values) as a regular Julia array a of the same element type. It raises an error if the array contains at least one missing value.

      source
      a = nomissing(da,value)

      Retun the values of the array da of type AbstractArray{Union{T,Missing},N} as a regular Julia array a by replacing all missing value by value (converted to type T). This function is identical to coalesce.(da,T(value)) where T is the element type of da.

      Example:

      julia> nomissing([missing,1.,2.],NaN)
+# returns [NaN, 1.0, 2.0]
      source
      CommonDataModel.fillvalue — Function
      fv = fillvalue(v::Variable)
+fv = fillvalue(v::CFVariable)

      Return the fill-value of the variable v.

      source
      fillvalue(::Type{Int8})
 fillvalue(::Type{UInt8})
 fillvalue(::Type{Int16})
 fillvalue(::Type{UInt16})
@@ -48,8 +50,7 @@
 fillvalue(::Type{Float32})
 fillvalue(::Type{Float64})
 fillvalue(::Type{Char})
-fillvalue(::Type{String})

      Default fill-value for the given type.

      source
      fv = fillvalue(v::Variable)
-fv = fillvalue(v::CFVariable)

      Return the fill-value of the variable v.

      source
      NCDatasets.load! — Function
      NCDatasets.load!(ncvar::Variable, data, indices)

      Loads a NetCDF variables ncvar in-place and puts the result in data along the specified indices. One can use @inbounds annotate code where bounds checking can be elided by the compiler (which typically require type-stable code).

      using NCDatasets
+fillvalue(::Type{String})

      Default fill-value for the given type from NetCDF.

      CommonDataModel.load! — Function
      NCDatasets.load!(ncvar::Variable, data, indices)

      Loads a NetCDF variables ncvar in-place and puts the result in data along the specified indices. One can use @inbounds annotate code where bounds checking can be elided by the compiler (which typically require type-stable code).

      using NCDatasets
 ds = NCDataset("file.nc")
 ncv = ds["vgos"].var;
 # data must have the right shape and type
@@ -61,7 +62,7 @@
 
 # loading a subset
 data = zeros(5); # must have the right shape and type
-load!(ds["temp"].var,data,:,1) # loads the 1st column
      Note

      For a netCDF variable of type NC_CHAR, the element type of the data array must be UInt8 and cannot be the julia Char type, because the julia Char type uses 4 bytes and the NetCDF NC_CHAR only 1 byte.

      source
      NCDatasets.load!(ncvar::CFVariable, data, buffer, indices)

      Loads a NetCDF variables ncvar in-place and puts the result in data (an array of eltype(ncvar)) along the specified indices. buffer is a temporary array of the same size as data but the type should be eltype(ncv.var), i.e. the corresponding type in the NetCDF files (before applying scale_factor, add_offset and masking fill values). Scaling and masking will be applied to the array data.

      data and buffer can be the same array if eltype(ncvar) == eltype(ncvar.var).

      Example:

      # create some test array
+load!(ds["temp"].var,data,:,1) # loads the 1st column
      Note

      For a netCDF variable of type NC_CHAR, the element type of the data array must be UInt8 and cannot be the julia Char type, because the julia Char type uses 4 bytes and the NetCDF NC_CHAR only 1 byte.

      source
      CommonDataModel.load!(ncvar::CFVariable, data, buffer, indices)

      Loads a NetCDF (or other format) variables ncvar in-place and puts the result in data (an array of eltype(ncvar)) along the specified indices. buffer is a temporary array of the same size as data but the type should be eltype(ncv.var), i.e. the corresponding type in the files (before applying scale_factor, add_offset and masking fill values). Scaling and masking will be applied to the array data.

      data and buffer can be the same array if eltype(ncvar) == eltype(ncvar.var).

      Example:

      # create some test array
 Dataset("file.nc","c") do ds
     defDim(ds,"time",3)
     ncvar = defVar(ds,"vgos",Int16,("time",),attrib = ["scale_factor" => 0.1])
@@ -76,8 +77,8 @@
 data = zeros(eltype(ncv),size(ncv)); # here Vector{Float64}
 buffer = zeros(eltype(ncv.var),size(ncv)); # here Vector{Int16}
 NCDatasets.load!(ncv,data,buffer,:,:,:)
-close(ds)
      source

      Creating a variable

      CommonDataModel.defVar — Function
      defVar(ds::NCDataset,name,vtype,dimnames; kwargs...)
-defVar(ds::NCDataset,name,data,dimnames; kwargs...)

      Define a variable with the name name in the dataset ds. vtype can be Julia types in the table below (with the corresponding NetCDF type). The parameter dimnames is a tuple with the names of the dimension. For scalar this parameter is the empty tuple (). The variable is returned (of the type CFVariable).

      Instead of providing the variable type one can directly give also the data data which will be used to fill the NetCDF variable. In this case, the dimensions with the appropriate size will be created as required using the names in dimnames.

      If data is a vector or array of DateTime objects, then the dates are saved as double-precision floats and units "days since 1900-01-01 00:00:00" (unless a time unit is specifed with the attrib keyword as described below). Dates are converted to the default calendar in the CF conversion which is the mixed Julian/Gregorian calendar.

      Keyword arguments

      • fillvalue: A value filled in the NetCDF file to indicate missing data. It will be stored in the _FillValue attribute.
      • chunksizes: Vector integers setting the chunk size. The total size of a chunk must be less than 4 GiB.
      • deflatelevel: Compression level: 0 (default) means no compression and 9 means maximum compression. Each chunk will be compressed individually.
      • shuffle: If true, the shuffle filter is activated which can improve the compression ratio.
      • checksum: The checksum method can be :fletcher32 or :nochecksum (checksumming is disabled, which is the default)
      • attrib: An iterable of attribute name and attribute value pairs, for example a Dict, DataStructures.OrderedDict or simply a vector of pairs (see example below)
      • typename (string): The name of the NetCDF type required for vlen arrays

      chunksizes, deflatelevel, shuffle and checksum can only be set on NetCDF 4 files. Compression of strings and variable-length arrays is not supported by the underlying NetCDF library.

      NetCDF data types

      NetCDF TypeJulia Type
      NC_BYTEInt8
      NC_UBYTEUInt8
      NC_SHORTInt16
      NC_INTInt32
      NC_INT64Int64
      NC_FLOATFloat32
      NC_DOUBLEFloat64
      NC_CHARChar
      NC_STRINGString

      Dimension ordering

      The data is stored in the NetCDF file in the same order as they are stored in memory. As julia uses the Column-major ordering for arrays, the order of dimensions will appear reversed when the data is loaded in languages or programs using Row-major ordering such as C/C++, Python/NumPy or the tools ncdump/ncgen (NetCDF CDL). NumPy can also use Column-major ordering but Row-major order is the default. For the column-major interpretation of the dimensions (as in Julia), the CF Convention recommends the order "longitude" (X), "latitude" (Y), "height or depth" (Z) and "date or time" (T) (if applicable). All other dimensions should, whenever possible, be placed to the right of the spatiotemporal dimensions.

      Example:

      In this example, scale_factor and add_offset are applied when the data is saved.

      julia> using DataStructures
+close(ds)

      Creating a variable

      CommonDataModel.defVar — Function
      defVar(ds::NCDataset,name,vtype,dimnames; kwargs...)
+defVar(ds::NCDataset,name,data,dimnames; kwargs...)

      Define a variable with the name name in the dataset ds. vtype can be Julia types in the table below (with the corresponding NetCDF type). The parameter dimnames is a tuple with the names of the dimension. For scalar this parameter is the empty tuple (). The variable is returned (of the type CFVariable).

      Instead of providing the variable type one can directly give also the data data which will be used to fill the NetCDF variable. In this case, the dimensions with the appropriate size will be created as required using the names in dimnames.

      If data is a vector or array of DateTime objects, then the dates are saved as double-precision floats and units "days since 1900-01-01 00:00:00" (unless a time unit is specifed with the attrib keyword as described below). Dates are converted to the default calendar in the CF conversion which is the mixed Julian/Gregorian calendar.

      Keyword arguments

      • fillvalue: A value filled in the NetCDF file to indicate missing data. It will be stored in the _FillValue attribute. NCDatasets does not use implicitely the default NetCDF fill values when reading data.
      • chunksizes: Vector integers setting the chunk size. The total size of a chunk must be less than 4 GiB.
      • deflatelevel: Compression level: 0 (default) means no compression and 9 means maximum compression. Each chunk will be compressed individually.
      • shuffle: If true, the shuffle filter is activated which can improve the compression ratio.
      • checksum: The checksum method can be :fletcher32 or :nochecksum (checksumming is disabled, which is the default)
      • attrib: An iterable of attribute name and attribute value pairs, for example a Dict, DataStructures.OrderedDict or simply a vector of pairs (see example below)
      • typename (string): The name of the NetCDF type required for vlen arrays

      chunksizes, deflatelevel, shuffle and checksum can only be set on NetCDF 4 files. Compression of strings and variable-length arrays is not supported by the underlying NetCDF library.

      NetCDF data types

      NetCDF TypeJulia Type
      NC_BYTEInt8
      NC_UBYTEUInt8
      NC_SHORTInt16
      NC_INTInt32
      NC_INT64Int64
      NC_FLOATFloat32
      NC_DOUBLEFloat64
      NC_CHARChar
      NC_STRINGString

      Dimension ordering

      The data is stored in the NetCDF file in the same order as they are stored in memory. As julia uses the Column-major ordering for arrays, the order of dimensions will appear reversed when the data is loaded in languages or programs using Row-major ordering such as C/C++, Python/NumPy or the tools ncdump/ncgen (NetCDF CDL). NumPy can also use Column-major ordering but Row-major order is the default. For the column-major interpretation of the dimensions (as in Julia), the CF Convention recommends the order "longitude" (X), "latitude" (Y), "height or depth" (Z) and "date or time" (T) (if applicable). All other dimensions should, whenever possible, be placed to the right of the spatiotemporal dimensions.

      Example:

      In this example, scale_factor and add_offset are applied when the data is saved.

      julia> using DataStructures
 julia> data = randn(3,5)
 julia> NCDataset("test_file.nc","c") do ds
           defVar(ds,"temp",data,("lon","lat"), attrib = OrderedDict(
@@ -86,10 +87,11 @@
              "scale_factor" => 0.1,
              "long_name" => "Temperature"
           ))
-       end;
      Note

      If the attributes _FillValue, missing_value, add_offset, scale_factor, units and calendar are used, they should be defined when calling defVar by using the parameter attrib as shown in the example above.

      source
      v = CommonDataModel.defVar(ds::AbstractDataset,src::AbstractVariable)

      Defines and return the variable in the data set ds copied from the variable src. The variable name, dimension name, attributes and data are copied from src.

      Storage parameter of a variable

      NCDatasets.chunking — Function
      storage,chunksizes = chunking(v::Variable)

      Return the storage type (:contiguous or :chunked) and the chunk sizes of the varable v. Note that chunking reports the same information as nc_inq_var_chunking and therefore considers variables with unlimited dimension as :contiguous.

      source
      storage,chunksizes = chunking(v::MFVariable)

      Return the storage type (:contiguous or :chunked) and the chunk sizes of the varable v corresponding to the first NetCDF file. If the first NetCDF file in the collection is chunked then this storage attributes are returns. If not the first NetCDF file is not contiguous, then multi-file variable is still reported as chunked with chunk size equal to the variable size.

      source
      NCDatasets.deflate — Function
      isshuffled,isdeflated,deflate_level = deflate(v::Variable)

      Return compression information of the variable v. If shuffle is true, then shuffling (byte interlacing) is activated. If deflate is true, then the data chunks (see chunking) are compressed using the compression level deflate_level (0 means no compression and 9 means maximum compression).

      source
      NCDatasets.checksum — Function
      checksummethod = checksum(v::Variable)

      Return the checksum method of the variable v which can be either be :fletcher32 or :nochecksum.

      source

      Coordinate variables and cell boundaries

      NCDatasets.coord — Function
      cv = coord(v::Union{CFVariable,Variable},standard_name)

      Find the coordinate of the variable v by the standard name standard_name or some standardized heuristics based on units. If the heuristics fail to detect the coordinate, consider to modify the netCDF file to add the standard_name attribute. All dimensions of the coordinate must also be dimensions of the variable v.

      Example

      using NCDatasets
+       end;
      Note

      If the attributes _FillValue, missing_value, add_offset, scale_factor, units and calendar are used, they should be defined when calling defVar by using the parameter attrib as shown in the example above.

      source
      v = CommonDataModel.defVar(ds::AbstractDataset,src::AbstractVariable)
+v = CommonDataModel.defVar(ds::AbstractDataset,name::SymbolOrString,src::AbstractVariable)

      Defines and return the variable in the data set ds copied from the variable src. The dimension name, attributes and data are copied from src as well as the variable name (unless provide by name).

      Storage parameter of a variable

      CommonDataModel.chunking — Function
      storage,chunksizes = chunking(v::Variable)

      Return the storage type (:contiguous or :chunked) and the chunk sizes of the varable v. Note that chunking reports the same information as nc_inq_var_chunking and therefore considers variables with unlimited dimension as :contiguous.

      source
      storage,chunksizes = chunking(v::MFVariable)

      Return the storage type (:contiguous or :chunked) and the chunk sizes of the varable v corresponding to the first file. If the first file in the collection is chunked then this storage attributes are returns. If not the first file is not contiguous, then multi-file variable is still reported as chunked with chunk size equal to the variable size.

      CommonDataModel.deflate — Function
      isshuffled,isdeflated,deflate_level = deflate(v::Variable)

      Return compression information of the variable v. If shuffle is true, then shuffling (byte interlacing) is activated. If deflate is true, then the data chunks (see chunking) are compressed using the compression level deflate_level (0 means no compression and 9 means maximum compression).

      source
      CommonDataModel.checksum — Function
      checksummethod = checksum(v::Variable)

      Return the checksum method of the variable v which can be either be :fletcher32 or :nochecksum.

      source

      Coordinate variables and cell boundaries

      CommonDataModel.coord — Function
      cv = coord(v::Union{CFVariable,Variable},standard_name)

      Find the coordinate of the variable v by the standard name standard_name or some standardized heuristics based on units. If the heuristics fail to detect the coordinate, consider to modify the file to add the standard_name attribute. All dimensions of the coordinate must also be dimensions of the variable v.

      Example

      using NCDatasets
 ds = NCDataset("file.nc")
 ncv = ds["SST"]
 lon = coord(ncv,"longitude")[:]
 lat = coord(ncv,"latitude")[:]
 v = ncv[:]
-close(ds)
      source
      NCDatasets.bounds — Function
      b = bounds(ncvar::NCDatasets.CFVariable)

      Return the CFVariable corresponding to the bounds attribute of the variable ncvar. The time units and calendar from the ncvar are used but not the attributes controling the packing of data scale_factor, add_offset and _FillValue.

      source
      +close(ds)
      CommonDataModel.bounds — Function
      b = bounds(ncvar::NCDatasets.CFVariable)

      Return the CFVariable corresponding to the bounds attribute of the variable ncvar. The time units and calendar from the ncvar are used but not the attributes controling the packing of data scale_factor, add_offset and _FillValue.