added DecodingResult struct, changed decoding and predictor API to allow in-place

[feefladder]

closes #87 also #86 is somewhat relevant.

makes the output of a decoding operation a typed array.

The goals:

• output data is properly aligned and typed
• less-copy: try to copy the resulting data as little as possible
  the problem:

1. data comes in as Bytes from the underlying network layer -> 1 copy is needed for typing+alignment (without extensive spec-reading and changing the api even more if we ever wanted to directly cast the Bytes to aligned+typed arrays)
2. some decompressions allow for reading into a buffer (deflate, lzw), others (lzw, jpeg) not. uncompressed copies because of 1
3. some predictors allow in-place (none: endianness-fixing and horizontal), others (float) not

The typed array can be filled in different places:

1. during decoding (breaking change, this PR)
2. during unpredicting (less breaking, leaves DecoderRegistry API untouched)
3. after unpredicting (current, the user does that here)

this PR does it during the decoding step, which:

• 6allows decompression algorithms to directly decompress into the buffer, saving a copy
• allows users to create their own typed buffers and decode into those using decode_into.

It changes quite some public api, so open to discussion. more comments coming on the changes

sidenote: there is a redundant copy with no compression and floating point prediction, but which sensible person would ever have that configuration?

[feefladder]

oof it seems putting the change at the decoder step introduces some Python headaches. this pyo3 issue seems relevant. Mainly that result_buffer: &mut[u8] which is now passed to the decoder(s) is not python-friendly.... have to go now

[feefladder]

Made it a draft, because currently the Python side would need changes as well and I'm as of yet inexperienced with Rust-Python things. Can look into it further after wednesday. For the moment, any thoughts @weiji14? especially if changing to typed array at the unpredicting step is the way to go? (would be slightly sad because of the extra copy, but that's ok and a good compromise I think - if the python stuff turns out to be a lot/difficult or too breaking)

alternatively, this PR could be split into:

• one adding the DecodingResult enum, creation logic and its Pythonic equivalents
• one that changes the actual decoding steps

[weiji14]

alternatively, this PR could be split into:

* one adding the DecodingResult enum, creation logic and its Pythonic equivalents
* one that changes the actual decoding steps

Yes please, might be good to split this into two (I'm generally in favour of reviewing smaller PRs). I'm wondering if there's an opportunity ot use num-trait's ToBytes trait here somewhere, but haven't thought about it too much yet.

[feefladder]

alternatively, this PR could be split into:

* one adding the DecodingResult enum, creation logic and its Pythonic equivalents
* one that changes the actual decoding steps

Yes please, might be good to split this into two (I'm generally in favour of reviewing smaller PRs).

will do! I'll keep it in one step for now, because most changes are coherent and basically I won't know what is needed before I see things working together. So the plan:

1. make everything work
2. split it in two
3. add nice unit tests
4. make non-draft PRs

I'm wondering if there's an opportunity ot use num-trait's ToBytes trait here somewhere, but haven't thought about it too much yet.

Is num-trait's ToBytes Bytes the same as bytes::Bytes? If so that would be good

I've given the whole Python part a bit more thought. passing a &mut[u8] across the boundary adds a lot of caveats on the python side or requires implementing the buffer protocol for DecodingResult RasterData (I think that's a better name), which is IMO a no-go (the added complexity is absolutely not worth it). Should be as simple as:

impl Decoder for PyDecoder {
    fn decode_tile(
        &self,
        compressed_buffer: Bytes,
        result_buffer: &mut[u8],
        _photometric_interpretation: PhotometricInterpretation,
        _jpeg_tables: Option<&[u8]>,
    ) -> AsyncTiffResult<()> {
        let decoded_buffer = Python::with_gil(|py| self.call(py, buffer))
            .map_err(|err| AsyncTiffError::General(err.to_string()))?;
        result_buffer.copy_from_slice(&decoded_buffer.into_inner()); //only add this extra copy
        Ok(())
    }
}

then there is a bit of sadness (needless copy) in the case of a user-implemented decoder and floating point predictor, but solving that would also be unnecessarily ugly for saving only a copy.

[kylebarron]

I'm wondering if there's an opportunity ot use num-trait's ToBytes trait here somewhere

No, that ToBytes is for converting a single number to a byte array. Like converting a single u32 to the four bytes that make it up. So it's irrelevant for our needs I think.

[kylebarron]

I'm a pretty strong 👎 on having any unsafe in our code, especially for a transmute like this.

[kylebarron]

It's not clear to me that passing in a result buffer is a meaningful improvement. I'd much rather use bytes::Bytes (i.e. an Arc<Vec<u8>>) that provides cheap cloning. (I.e. keep the existing API)

[kylebarron]

oof it seems putting the change at the decoder step introduces some Python headaches. this pyo3 issue seems relevant. Mainly that result_buffer: &mut[u8] which is now passed to the decoder(s) is not python-friendly.... have to go now

This is another reason why we should use bytes::Bytes everywhere, so that we can easily reuse free buffer protocol interop with https://docs.rs/pyo3-bytes/0.3.0/pyo3_bytes/

[feefladder]

oof it seems putting the change at the decoder step introduces some Python headaches. this pyo3 issue seems relevant. Mainly that result_buffer: &mut[u8] which is now passed to the decoder(s) is not python-friendly.... have to go now

This is another reason why we should use bytes::Bytes everywhere, so that we can easily reuse free buffer protocol interop with https://docs.rs/pyo3-bytes/0.3.0/pyo3_bytes/

So my final snippet did not change anything Python-API side and solved the issue I raised here. To keep everything Bytes, one would:

impl Decoder for PyDecoder {
    fn decode_tile(
        &self,
        compressed_buffer: Bytes,
        result_buffer: &mut[u8],
        _photometric_interpretation: PhotometricInterpretation,
        _jpeg_tables: Option<&[u8]>,
    ) -> AsyncTiffResult<()> {
        let decoded_buffer = Python::with_gil(|py| self.call(py, buffer))
            .map_err(|err| AsyncTiffError::General(err.to_string()))?;
        result_buffer.copy_from_slice(&decoded_buffer.into_inner()); //only add this extra copy
        Ok(())
    }
}

...I guess it could indeed have some more thought