-
Notifications
You must be signed in to change notification settings - Fork 97
Can't pickle numcodecs codecs #744
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
is it OK if I transfer this issue to numcodecs? |
Probably need to use Or is this coming from some other Zarr code path? Can't comment on Numcodecs Zarr 3 code as I haven't been involved in this |
@jakirkham the cause is this function: def _make_bytes_bytes_codec(codec_name: str, cls_name: str) -> type[_NumcodecsBytesBytesCodec]:
# rename for class scope
_codec_name = CODEC_PREFIX + codec_name
class _Codec(_NumcodecsBytesBytesCodec):
codec_name = _codec_name
def __init__(self, **codec_config: JSON) -> None:
super().__init__(**codec_config)
_Codec.__name__ = cls_name
return _Codec It defines a class inside function scope, but this approach is not needed to solve the underlying problem. As this function gets called fewer than 10 times, we could instead have fewer than 10 separate normal class definitions. That would be my recommended short-term fix. |
we could also use |
Yes I understand. Read the same code. This is the kind of thing That said, Idk why we went with that design choice. This is what I mean by I cannot comment on why that was added that way |
@TomNicholas i transferred this to numcodecs, because it requires changes to numcodecs code. |
I'm happy to submit a PR to do this - I had to make a janky branch to get what I'm working on to work anyway - I might as well turn it into a proper PR. |
that would be great @TomNicholas, feel free to reach out if you want any pointers. |
Zarr version
3.0.6
Numcodecs version
0.16.0
Python Version
3.12.2
Operating System
Mac
Installation
conda
Description
The numcodecs codecs aren't pickleable.
For me it's a problem because I can't use AWS lambda functions with the lithops serverless framework. My understanding is that pickleability is something that
zarr-python
tests it can do, so it's inconsistent thatnumcodecs
can't.The problem seems to be this weird dynamic creation of codec classes. Presumably if
numcodecs
just defined these classes in a more conentional static way then they would be pickleable.Steps to reproduce
Additional output
No response
The text was updated successfully, but these errors were encountered: