Migration to zarr 3

.github/workflows/tests.yml

@@ -16,7 +16,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["3.10", "3.11", "3.12"]
+        python-version: ["3.12", "3.13", "3.14"]
 
     steps:
     - uses: actions/checkout@v3

.gitignore

@@ -1,5 +1,6 @@
 testdata/
 site/
+examples/outputs/
 
 # Byte-compiled / optimized / DLL files
 __pycache__/
@@ -162,5 +163,8 @@ cython_debug/
 #  option (not recommended) you can uncomment the following to ignore the entire idea folder.
 .idea/
 
+# Visual Studio Code
+.vscode/
+
 # Downloaded assets
 docs/assets/downloaded/

CHANGES.md

@@ -1,5 +1,7 @@
 ## Version 0.8.1 (in development)
 
+* Migrated to the Zarr v3 Python API. Supports writing Zarr v2 and v3 stores, 
+  with Zarr v2 as the default. (#111)
 * Replaced dev tools black and flake8 by isort and ruff. 
 * Reformatted code and fixed linter issues.
 * Added `environment.yml` to setup a conda/mamba development environment. 

README.md

@@ -28,7 +28,8 @@ awesome Python packages [xarray](https://docs.xarray.dev/) and [zarr](https://za
 The objective of `zappend` is enabling geodata scientists and developers to 
 robustly create large data cubes. The tool performs transaction-based dataset 
 appends to existing data cubes in the 
-[Zarr format](https://zarr.readthedocs.io/en/stable/spec/v2.html). If an error 
+[Zarr format](https://zarr.readthedocs.io/en/stable/). Targets use Zarr v2 by
+default and may opt into Zarr v3. If an error
 occurs during an append step — typically due to I/O problems or out-of-memory 
 conditions — `zappend` will automatically roll back the operation, ensuring that 
 the existing data cube maintains its structural integrity. The design drivers 

docs/config.md

@@ -76,6 +76,10 @@ Variable metadata.
                 Its value is `null`.
           - Disable chunking in all dimensions.
             Its value is `null`.
+      - `shards`:
+        Zarr v3 shard sizes for each dimension of the variable. Each shard size
+        must be a multiple of the corresponding chunk size. The items of the
+        list are positive integers.
       - `fill_value`:
         Must be one of the following:
           - Type _number_.
@@ -134,8 +138,7 @@ Defaults to `"keep"`.
 
 ### `zarr_version`
 
-The Zarr version to be used.
-Its value is `2`.
+The Zarr version to be used. Allowed values are `2` and `3`.
 Defaults to `2`.
 
 ## Data I/O - Target

docs/guide.md

@@ -31,10 +31,11 @@ one, if their paths are given as URIs prefixed with a filesystem protocol such a
 dedicated configuration settings. More on this is given
 in section [_Data I/O_](#data-io) below.
 
-!!! note "Zarr Format v2"
+!!! note "Zarr formats"
     By default, `zappend` uses the [Zarr storage specification 2](https://zarr.readthedocs.io/en/stable/spec/v2.html)
-    and has only been tested with this version. The `zarr_version` setting can be used 
-    to change it, e,g, to `3`, but any other value than `2` is currently unsupported.
+    for compatibility with existing clients. Set `zarr_version` to `3` to write
+    Zarr v3. The target's format is fixed when it is created; use the same
+    setting for all later appends.
 
 The tool takes care of generating the target dataset from slice datasets, but doesn't 
 care how the slice datasets are created. Hence, when using the Python `zappend()` 
@@ -336,6 +337,26 @@ using the `chunks` setting in the encoding of each variable.
 The value of `chunks` can also be `null`, which means no chunking is 
 desired and the variable's data array will be persisted as one block. 
 
+For Zarr v3, `shards` may additionally be configured for a variable. A shard
+contains an integer number of regular chunks, reducing object counts while
+retaining smaller read/write chunks. Each shard size must be at least its
+corresponding chunk size and an exact multiple of it. Shards are not available
+for Zarr v2.
+
+```json
+{
+    "zarr_version": 3,
+    "variables": {
+        "chl": {
+            "encoding": {
+                "chunks": [1, 512, 512],
+                "shards": [8, 2048, 2048]
+            }
+        }
+    }
+}
+```
+
 By default, the chunking of the coordinate variable corresponding to the append 
 dimension will be its dimension size in the first slice dataset. Often, the size 
 will be `1` or another small number. Since `xarray` loads coordinates eagerly 

docs/index.md

@@ -11,7 +11,8 @@ awesome Python packages [xarray](https://docs.xarray.dev/) and [zarr](https://za
 The objective of `zappend` is enabling geodata scientists and developers to 
 robustly create large data cubes. The tool performs transaction-based dataset 
 appends to existing data cubes in the 
-[Zarr format](https://zarr.readthedocs.io/en/stable/spec/v2.html). If an error 
+[Zarr format](https://zarr.readthedocs.io/en/stable/). Targets use Zarr v2 by
+default and may opt into Zarr v3. If an error
 occurs during an append step — typically due to I/O problems or out-of-memory 
 conditions — `zappend` will automatically roll back the operation, ensuring that 
 the existing data cube maintains its structural integrity. The design drivers 

environment.yml

@@ -11,8 +11,8 @@ dependencies:
   - numcodecs
   - numpy
   - pyyaml
-  - xarray <2026.4 # constrain to <2026.4 to retain compatibility with zarr 2.
-  - zarr >=2.18,<3
+  - xarray <2026.4
+  - zarr >=3,<4
   # Dev Dependencies
   - isort
   - pytest
@@ -32,3 +32,6 @@ dependencies:
   - h5netcdf
   - s3fs
   - scipy
+  # Development Dependencies - Demos
+  - jupyterlab
+  - matplotlib

examples/create_inputs.py

@@ -0,0 +1,92 @@
+#  Copyright © 2024-2026 Brockmann Consult and contributors
+#  Permissions are hereby granted under the terms of the MIT License:
+#  https://opensource.org/licenses/MIT.
+
+from pathlib import Path
+
+import numpy as np
+import xarray as xr
+
+INPUT_DIR = Path(__file__).parent / "inputs"
+START_TIME = np.datetime64("2024-01-01", "D")
+
+
+def make_slice(time: np.datetime64, index: int) -> xr.Dataset:
+    """Return one small daily slice with a predictable value pattern."""
+    lon = np.arange(-179.95, 180, 0.1, dtype="float32")
+    lat = np.arange(-89.95, 90, 0.1, dtype="float32")
+    data = np.full((1, len(lat), len(lon)), index, dtype="uint8")
+    return xr.Dataset(
+        data_vars={
+            "temperature": (
+                ("time", "lat", "lon"),
+                data.astype("float32"),
+                {"units": "degC", "long_name": "example sea temperature"},
+            ),
+            "quality_flag": (
+                ("time", "lat", "lon"),
+                data,
+                {"long_name": "example quality flag"},
+            ),
+        },
+        coords={"time": [time], "lat": lat, "lon": lon},
+        attrs={"title": "Zappend example slice"},
+    ).chunk({"time": 1, "lat": 450, "lon": 450})
+
+
+def make_zarr_v2_slices(n_slices: int) -> None:
+    """Create *n_slices* daily Zarr v2 inputs from :func:`make_slice`."""
+    times = START_TIME + np.arange(n_slices) * np.timedelta64(1, "D")
+    directory = INPUT_DIR / "zarr-v2"
+    directory.mkdir(parents=True, exist_ok=True)
+    for index, time in enumerate(times):
+        dataset = make_slice(time, index)
+        dataset.to_zarr(
+            directory / f"slice-{time!s}.zarr",
+            mode="w",
+            zarr_format=2,
+            consolidated=True,
+        )
+
+
+def make_zarr_v3_slices(n_slices: int) -> None:
+    """Create *n_slices* daily, sharded Zarr v3 inputs from :func:`make_slice`."""
+    times = START_TIME + np.arange(n_slices) * np.timedelta64(1, "D")
+    directory = INPUT_DIR / "zarr-v3"
+    directory.mkdir(parents=True, exist_ok=True)
+
+    for index, time in enumerate(times):
+        # Xarray requires Dask write chunks to align with the outer Zarr v3
+        # shard grid. The encoding below still defines 450 by 450 inner
+        # chunks inside each 1800 by 3600 shard.
+        dataset = make_slice(time, index).chunk({"time": 1, "lat": 1800, "lon": 3600})
+        encoding = {
+            name: {
+                "chunks": (1, 450, 450),
+                "shards": (1, 1800, 3600),
+            }
+            for name in dataset.data_vars
+        }
+        dataset.to_zarr(
+            directory / f"slice-{time!s}.zarr",
+            mode="w",
+            encoding=encoding,
+            zarr_format=3,
+            consolidated=False,
+        )
+
+
+def make_netcdf_slices(n_slices: int) -> None:
+    """Create *n_slices* daily NetCDF from :func:`make_slice`."""
+    times = START_TIME + np.arange(n_slices) * np.timedelta64(1, "D")
+    directory = INPUT_DIR / "netcdf"
+    directory.mkdir(parents=True, exist_ok=True)
+
+    for index, time in enumerate(times):
+        dataset = make_slice(time, index)
+        encoding = {name: {"chunksizes": (1, 450, 450)} for name in dataset.data_vars}
+        dataset.to_netcdf(
+            directory / f"slice-{time!s}.nc",
+            engine="h5netcdf",
+            encoding=encoding,
+        )