diff --git a/dev/.documenter-siteinfo.json b/dev/.documenter-siteinfo.json index 2523d8b..8a51a0a 100644 --- a/dev/.documenter-siteinfo.json +++ b/dev/.documenter-siteinfo.json @@ -1 +1 @@ -{"documenter":{"julia_version":"1.11.1","generation_timestamp":"2024-12-06T14:11:25","documenter_version":"1.8.0"}} \ No newline at end of file +{"documenter":{"julia_version":"1.11.2","generation_timestamp":"2025-01-06T13:50:49","documenter_version":"1.8.0"}} \ No newline at end of file diff --git a/dev/index.html b/dev/index.html index 5f14881..06e44fe 100644 --- a/dev/index.html +++ b/dev/index.html @@ -1,2 +1,2 @@ -Home · Zarr.jl
GitHub

Zarr.jl

Reading and Writing Zarr Datasets from Julia

Package features

This is a currently incomplete implementation of the Zarr specs v2. It is possible to read an write (compressed) chunked n-dimensional arrays to disk, memory and cloud storage backends. Have a look at the Tutorial for a quick start.

Manual Outline

Index

+Home · Zarr.jl
GitHub

Zarr.jl

Reading and Writing Zarr Datasets from Julia

Package features

This is a currently incomplete implementation of the Zarr specs v2. It is possible to read an write (compressed) chunked n-dimensional arrays to disk, memory and cloud storage backends. Have a look at the Tutorial for a quick start.

Manual Outline

Index

diff --git a/dev/missings/index.html b/dev/missings/index.html index 967bdc3..0576add 100644 --- a/dev/missings/index.html +++ b/dev/missings/index.html @@ -39,4 +39,4 @@ missing 5 5 -

The fill_as_missing option is also available on array construction with zcreate, zopen or zzeros. Note also that one can also write missings into arrays opened with fill_as_missing=true. This means that every missing entry will be converted to a fillvalue in the zarr array and will appear as fill values in other software that opens the same array.

+

The fill_as_missing option is also available on array construction with zcreate, zopen or zzeros. Note also that one can also write missings into arrays opened with fill_as_missing=true. This means that every missing entry will be converted to a fillvalue in the zarr array and will appear as fill values in other software that opens the same array.

diff --git a/dev/operations/index.html b/dev/operations/index.html index 0aa7a69..9cbd008 100644 --- a/dev/operations/index.html +++ b/dev/operations/index.html @@ -31,4 +31,4 @@ 14.614388350826788 [:, :, 1980] = - 13.913361540597469 + 13.913361540597469 diff --git a/dev/reference/index.html b/dev/reference/index.html index badd732..2d25d8a 100644 --- a/dev/reference/index.html +++ b/dev/reference/index.html @@ -1,2 +1,2 @@ -API Reference · Zarr.jl
GitHub

API reference

Arrays

Zarr.zcreate — Function

zcreate(T, dims...;kwargs)

Creates a new empty zarr array with element type T and array dimensions dims. The following keyword arguments are accepted:

  • path="" directory name to store a persistent array. If left empty, an in-memory array will be created
  • name="" name of the zarr array, defaults to the directory name
  • storagetype determines the storage to use, current options are DirectoryStore or DictStore
  • chunks=dims size of the individual array chunks, must be a tuple of length length(dims)
  • fill_value=nothing value to represent missing values
  • fill_as_missing=false set to true shall fillvalue s be converted to missings
  • filters=filters to be applied
  • compressor=BloscCompressor() compressor type and properties
  • attrs=Dict() a dict containing key-value pairs with metadata attributes associated to the array
  • writeable=true determines if the array is opened in read-only or write mode
  • indent_json=false determines if indents are added to format the json files .zarray and .zattrs. This makes them more readable, but increases file size.
source

Create a new subarray of the group g

source
Zarr.zzeros — Function

zzeros(T, dims...; kwargs... )

Creates a zarr array and initializes all values with zero. Accepts the same keyword arguments as zcreate

source

Group hierarchy

Zarr.zgroup — Function
zgroup(s::AbstractStore; attrs=Dict())

Create a new zgroup in the store s

source
Zarr.zopen — Function
zopen(s::AbstractStore, mode="r"; consolidated = false, path = "", lru = 0)

Opens a zarr Array or Group at Store s. If consolidated is set to "true", Zarr will search for a consolidated metadata field as created by the python zarr consolidate_metadata function. This can substantially speed up metadata parsing of large zarr groups. Setting lru to a value > 0 means that chunks that have been accessed before will be cached and consecutive reads will happen from the cache. Here, lru denotes the number of chunks that remain in memory.

source
Zarr.zopen — Function
zopen(p::String, mode="r")

Open a zarr Array or group at disc path p.

source
Zarr.zopen_noerr — Function
zopen_noerr(AbstractStore, mode = "r"; consolidated = false)

Works like zopen with the single difference that no error is thrown when the path or store does not point to a valid zarr array or group, but nothing is returned instead.

source

Compressors

Zarr.Compressor — Type
abstract type Compressor

The abstract supertype for all Zarr compressors.

Interface

All subtypes of Compressor SHALL implement the following methods:

  • zcompress(a, c::Compressor): compress the array a using the compressor c.
  • zuncompress(a, c::Compressor, T): uncompress the array a using the compressor c and return an array of type T.
  • JSON.lower(c::Compressor): return a JSON representation of the compressor c, which follows the Zarr specification for that compressor.
  • getCompressor(::Type{<:Compressor}, d::Dict): return a compressor object from a given dictionary d which contains the compressor's parameters according to the Zarr spec.

Subtypes of Compressor MAY also implement the following methods:

  • zcompress!(compressed, data, c::Compressor): compress the array data using the compressor c and store the result in the array compressed.
  • zuncompress!(data, compressed, c::Compressor): uncompress the array compressed using the compressor c and store the result in the array data.

Finally, an entry MUST be added to the compressortypes dictionary for each compressor type. This must also follow the Zarr specification's name for that compressor. The name of the compressor is the key, and the value is the compressor type (e.g. BloscCompressor or NoCompressor).

For example, the Blosc compressor is named "blosc" in the Zarr spec, so the entry for BloscCompressor must be added to compressortypes as compressortypes["blosc"] = BloscCompressor.

source
Zarr.NoCompressor — Type
NoCompressor()

Creates an object that can be passed to ZArray constructors without compression.

source
Zarr.BloscCompressor — Method
BloscCompressor(;blocksize=0, clevel=5, cname="lz4", shuffle=1)

Returns a BloscCompressor struct that can serve as a Zarr array compressor. Keyword arguments are:

  • clevel=5 the compression level, number between 0 (no compression) and 9 (max compression)
  • cname="lz4" compressor name, can be one of "blosclz", "lz4", and "lz4hc"
  • shuffle=1 Either NOSHUFFLE (0), SHUFFLE (1), BITSHUFFLE (2) or AUTOSHUFFLE (-1). If AUTOSHUFFLE, bit-shuffle will be used for buffers with itemsize 1, and byte-shuffle will be used otherwise. The default is SHUFFLE.
source
Zarr.ZlibCompressor — Type
ZlibCompressor(clevel=-1)

Returns a ZlibCompressor struct that can serve as a Zarr array compressor. Keyword arguments are:

  • clevel=-1 the compression level, number between -1 (Default), 0 (no compression) and 9 (max compression)
  • default is -1 compromise between speed and compression (currently equivalent to level 6).
source
+API Reference · Zarr.jl
GitHub

API reference

Arrays

Zarr.zcreate — Function

zcreate(T, dims...;kwargs)

Creates a new empty zarr array with element type T and array dimensions dims. The following keyword arguments are accepted:

  • path="" directory name to store a persistent array. If left empty, an in-memory array will be created
  • name="" name of the zarr array, defaults to the directory name
  • storagetype determines the storage to use, current options are DirectoryStore or DictStore
  • chunks=dims size of the individual array chunks, must be a tuple of length length(dims)
  • fill_value=nothing value to represent missing values
  • fill_as_missing=false set to true shall fillvalue s be converted to missings
  • filters=filters to be applied
  • compressor=BloscCompressor() compressor type and properties
  • attrs=Dict() a dict containing key-value pairs with metadata attributes associated to the array
  • writeable=true determines if the array is opened in read-only or write mode
  • indent_json=false determines if indents are added to format the json files .zarray and .zattrs. This makes them more readable, but increases file size.
source

Create a new subarray of the group g

source
Zarr.zzeros — Function

zzeros(T, dims...; kwargs... )

Creates a zarr array and initializes all values with zero. Accepts the same keyword arguments as zcreate

source

Group hierarchy

Zarr.zgroup — Function
zgroup(s::AbstractStore; attrs=Dict())

Create a new zgroup in the store s

source
Zarr.zopen — Function
zopen(s::AbstractStore, mode="r"; consolidated = false, path = "", lru = 0)

Opens a zarr Array or Group at Store s. If consolidated is set to "true", Zarr will search for a consolidated metadata field as created by the python zarr consolidate_metadata function. This can substantially speed up metadata parsing of large zarr groups. Setting lru to a value > 0 means that chunks that have been accessed before will be cached and consecutive reads will happen from the cache. Here, lru denotes the number of chunks that remain in memory.

source
Zarr.zopen — Function
zopen(p::String, mode="r")

Open a zarr Array or group at disc path p.

source
Zarr.zopen_noerr — Function
zopen_noerr(AbstractStore, mode = "r"; consolidated = false)

Works like zopen with the single difference that no error is thrown when the path or store does not point to a valid zarr array or group, but nothing is returned instead.

source

Compressors

Zarr.Compressor — Type
abstract type Compressor

The abstract supertype for all Zarr compressors.

Interface

All subtypes of Compressor SHALL implement the following methods:

  • zcompress(a, c::Compressor): compress the array a using the compressor c.
  • zuncompress(a, c::Compressor, T): uncompress the array a using the compressor c and return an array of type T.
  • JSON.lower(c::Compressor): return a JSON representation of the compressor c, which follows the Zarr specification for that compressor.
  • getCompressor(::Type{<:Compressor}, d::Dict): return a compressor object from a given dictionary d which contains the compressor's parameters according to the Zarr spec.

Subtypes of Compressor MAY also implement the following methods:

  • zcompress!(compressed, data, c::Compressor): compress the array data using the compressor c and store the result in the array compressed.
  • zuncompress!(data, compressed, c::Compressor): uncompress the array compressed using the compressor c and store the result in the array data.

Finally, an entry MUST be added to the compressortypes dictionary for each compressor type. This must also follow the Zarr specification's name for that compressor. The name of the compressor is the key, and the value is the compressor type (e.g. BloscCompressor or NoCompressor).

For example, the Blosc compressor is named "blosc" in the Zarr spec, so the entry for BloscCompressor must be added to compressortypes as compressortypes["blosc"] = BloscCompressor.

source
Zarr.NoCompressor — Type
NoCompressor()

Creates an object that can be passed to ZArray constructors without compression.

source
Zarr.BloscCompressor — Method
BloscCompressor(;blocksize=0, clevel=5, cname="lz4", shuffle=1)

Returns a BloscCompressor struct that can serve as a Zarr array compressor. Keyword arguments are:

  • clevel=5 the compression level, number between 0 (no compression) and 9 (max compression)
  • cname="lz4" compressor name, can be one of "blosclz", "lz4", and "lz4hc"
  • shuffle=1 Either NOSHUFFLE (0), SHUFFLE (1), BITSHUFFLE (2) or AUTOSHUFFLE (-1). If AUTOSHUFFLE, bit-shuffle will be used for buffers with itemsize 1, and byte-shuffle will be used otherwise. The default is SHUFFLE.
source
Zarr.ZlibCompressor — Type
ZlibCompressor(clevel=-1)

Returns a ZlibCompressor struct that can serve as a Zarr array compressor. Keyword arguments are:

  • clevel=-1 the compression level, number between -1 (Default), 0 (no compression) and 9 (max compression)
  • default is -1 compromise between speed and compression (currently equivalent to level 6).
source
diff --git a/dev/s3examples/index.html b/dev/s3examples/index.html index f8056c5..337ebaf 100644 --- a/dev/s3examples/index.html +++ b/dev/s3examples/index.html @@ -101,4 +101,4 @@ 4.0 10.0 16.0 - 22.0

`````

+ 22.0

`````

diff --git a/dev/storage/index.html b/dev/storage/index.html index 9aaa70f..9c05cdd 100644 --- a/dev/storage/index.html +++ b/dev/storage/index.html @@ -1,2 +1,2 @@ -Storage Backends · Zarr.jl
GitHub

Developing new storage backends

One advantage of the zarr data model is that it can be used in combination with a variety of storage backends. Currently in this package there is support for a DictStore (keeping data in memory), DirectoryStore (writing data to a local disk) and an S3Store for S3-compatible object store which is currently read-only. In order to implement a new storage backend, you would have to create a subtype of Zarr.AbstractStore and implement the following methods:

Zarr.storagesize — Function

storagesize(d::AbstractStore, p::AbstractString)

This function shall return the size of all data files in a store at path p.

source
Base.getindex — Method
Base.getindex(d::AbstractStore,i::String)

Returns the data stored in the given key as a Vector{UInt8}

source
Base.setindex! — Method
Base.setindex!(d::AbstractStore,v,i::String)

Writes the values in v to the given store and key.

source
Zarr.subdirs — Function
subdirs(d::AbstractStore, p)

Returns a list of keys for children stores in the given store at path p.

source

You can get some inspiration on how to implement this by looking at the source code of existing storage backends.

+Storage Backends · Zarr.jl
GitHub

Developing new storage backends

One advantage of the zarr data model is that it can be used in combination with a variety of storage backends. Currently in this package there is support for a DictStore (keeping data in memory), DirectoryStore (writing data to a local disk) and an S3Store for S3-compatible object store which is currently read-only. In order to implement a new storage backend, you would have to create a subtype of Zarr.AbstractStore and implement the following methods:

Zarr.storagesize — Function

storagesize(d::AbstractStore, p::AbstractString)

This function shall return the size of all data files in a store at path p.

source
Base.getindex — Method
Base.getindex(d::AbstractStore,i::String)

Returns the data stored in the given key as a Vector{UInt8}

source
Base.setindex! — Method
Base.setindex!(d::AbstractStore,v,i::String)

Writes the values in v to the given store and key.

source
Zarr.subdirs — Function
subdirs(d::AbstractStore, p)

Returns a list of keys for children stores in the given store at path p.

source

You can get some inspiration on how to implement this by looking at the source code of existing storage backends.

diff --git a/dev/tutorial/index.html b/dev/tutorial/index.html index 3200a4d..82745fb 100644 --- a/dev/tutorial/index.html +++ b/dev/tutorial/index.html @@ -122,4 +122,4 @@ [1, 3, 5] [4] [7, 9, 14] - [] + []