Add test to illustrate change in behaviour between rasterio/gdal verions

[alexgleith]

See issue raised in odc-stac at opendatacube/odc-stac#259

This test is intended to illustrate the change in behavior between older and newer versions of rasterio/gdal.

Note that in versions of Python that are supported by rasterio version 1.5.0, we see this warning twice:

WARNING  rasterio._env:__init__.py:377 CPLE_NotSupported in 4bcf1d22-2884-4b6d-b83a-a2c7e247745b.tif: PREDICTOR option is ignored for COMPRESS=None. Only valid for DEFLATE, LZW, LZMA or ZSTD
WARNING  rasterio._err:warp.py:385 CPLE_AppDefined:Value 1 in the source dataset has been changed to 0 in the destination dataset to avoid being treated as NoData. To avoid this, select a different NoData value for the destination dataset.

[pjonsson]

It's the workflows that use Rasterio 1.5 that fail and I recognize the second warning message, so my guess is that this is one of the bugs we hit in datacube-core.

If it is the same bug, it should disappear if one rebuilds Rasterio to use GDAL 3.12.2 instead of the GDAL version the binary wheels of Rasterio are built with. (It's possible some earlier version of GDAL will fix this specific problem too, I only remember that all the bugs I was tracking were fixed for 3.12.2.)

[alexgleith]

it should disappear if one rebuilds Rasterio to use GDAL 3.12.2

I built locally against 3.12.2 and still experienced the bug. I'm pretty sure that 1.5.0 of Rasterio packages 3.12.1 though, so these tests aren't against that newer GDAL.

[Kirill888]

@alexgleith I'm confused about this whole nodata for qa band of Landsat 9 issue. Can you please confirm/correct my understanding so far

1. nodata=1 attribute is configured in STAC metadata
2. nodata is not set in the actual COGs stored on S3
3. In this context there are differences in the behaviour of odc.stac.load based on what version of GDAL/rasterio combo is installed in the Python environment under test.
4. Difference is only observed when using non-native projection at load
5. Given that nodata is missing from COGs we do not expect fusing to work correctly, yet it does in some circumstances and it seems to depend on versions of GDAL/rasterio

Sounds to me like a change in how src_nodata=None, dst_nodata=<something> is handled by rasterio/GDAL.

[alexgleith]

Yes, @Kirill888, I think you have it captured.

Note that on point 5, fusing works as expected when not reprojecting on both versions.

The warning from GDAL seems to imply it's being more strict with destination nodata definitions, but I don't know about that.

I'm reporting the issues I've seen using the tooling and the change (unexpected/bad behaviour) in new versions of rasterio/gdal.

[alexgleith]

At the risk of being redundant, here's the GDAL warning:

Value 1 in the source dataset has been changed to 0 in the destination dataset to avoid being treated as NoData. To avoid this, select a different NoData value for the destination dataset.

[Kirill888]

Let's be even more redundant and document versions of GDAL you tested that differ in behaviour for this case.

[Kirill888]

In our case dst_nodata is coming from STAC, but can be also supplied by the user at call time (,nodata=...). While src_nodata should come from COG (so should be not set, i.e. src_nodata=None), but might be resolved to src_nodata=dst_nodata or maybe src_nodata=0, we need to verify what actually ends up being passed to GDAL in actuality.

[alexgleith]

Let's be even more redundant and document versions of GDAL you tested that differ in behaviour for this case.

This configuration produces the error:

Rasterio: 1.5.0
GDAL: 3.12.2
ODC Loader: 0.6.4
ODC Geo: 0.5.0
ODC STAC: 0.5.2

This configuration works:

Rasterio: 1.4.3
GDAL: 3.9.3
ODC Loader: 0.6.4
ODC Geo: 0.5.0
ODC STAC: 0.5.2

There's more info in the original issue opendatacube/odc-stac#259