Change method: statistic from Property to Method to allow eg active.mean(axis=(0, 1)) and instriduce comprehensive testing of Reductionist axis implementation#300
Merged
valeriupredoi merged 59 commits intomainfrom Feb 17, 2026
Conversation
… the headers such as x-activestorage-count in favour of returning their data as part of the JSON payload.
Collaborator
Author
|
@maxstack many thanks for looking into this! I think I found the issue at hand - in the current Reductionist, the response is a dict that has a "bytes" key eg: but that value comes in as raw bytes and needs to be decoded at end pount by the Client; this explains a few things:
You can see the 503 from the failed test: -> that's a json decoder error (which also, incidentally, completely destroyed the memory on my local machine). We should not get Reductionist to return raw bytes, we need actual data that can not risk corruption and the Client being unable to decode it and use. Is this something doable? Cheers 🍺 |
Collaborator
Author
|
@maxstack I added a test that runs with a small file, run it like so: the test uses a small file with data 15x143x144, and looks like this: def test_small_file_axis_0_1():
"""Fails: activestorage.reductionist.ReductionistError: Reductionist error: HTTP 502: -"""
active = build_active_small_file()
result = active.min(axis=(0, 1))[:]
print("Reductionist final result", result)
assert result == [[[[164.8125]]]]the result that Reductionist returns looks like: so it looks like Reductionist is not applying a final statistic, by the looks of it - no idea why there are 101 elements in that array though; at any rate, the full payload sent by Reductionist looks like this: bytes is 576 long, so that's a bit weird too. Anyway, at least this test runs 🍺 |
… "url" with accompanying "interface_type" to denote either "s3" or "http" / "https" backend
The following example would work, tested with TLS cert and authentication:
storage_options = {
'username': "admin",
'password': "admin",
'client_kwargs': {'endpoint_url': "https://example.net:8000"}, # final proxy
}
active_storage_url = "https://reductionist.example.net/" # Reductionist
active = Active("file_path/file.nc", 'm01s30i111', storage_type="https",
storage_options=storage_options,
active_storage_url=active_storage_url)
…e_type as this is auto-detected
Collaborator
Author
|
@maxstack great many thanks for the good work done, mate! This is all spiffy and I'm merging now - big leaps with our work here 🍻 |
This was referenced Feb 17, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Big One
API change
meanis not a property anymore, but a method, soactive.mean[...]becomesactive.mean()[...]so we can pass args and kwargs, so now you canactive.mean(axis=(0, 1))[...]axisObsolete points, mainly from Max-V conversations
Main test case for Reductionist with axis
https://github.com/NCAS-CMS/PyActiveStorage/blob/axis_api/tests/test_real_s3_with_axes.py
Test Case 1
Test Case 2
Test Case 3
These fails are here https://github.com/NCAS-CMS/PyActiveStorage/actions/runs/20272446127/job/58211728980?pr=300