Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make "combine cubes" function public. #82

Draft
wants to merge 5 commits into
base: load_refactor
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 61 additions & 0 deletions lib/iris/cube.py
Original file line number Diff line number Diff line change
Expand Up @@ -634,6 +634,67 @@ def concatenate(
check_derived_coords=check_derived_coords,
)

def combine(self, options=None, merge_require_unique=False, **kwargs):
"""Combine cubes according to :class:`iris~.io.loading.LoadPolicy` options.
Copy link

@stephenworsley stephenworsley Oct 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if there may be some confusion caused by the link between CubeList.combine and LoadPolicy as it may not be intuitive that changing the latter would affect the former. There may be a way of naming this method so that this connection is clearer. Something like "apply_load_policy" or "combine_like_load".

I suppose there is an idea of having separate objects controlling loading and combining (i.e. a CombinePolicy object), but I think that would probably overcomplicate things.

Copy link
Owner Author

@pp-mo pp-mo Oct 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if there may be some confusion caused by the link between CubeList.combine and LoadPolicy ... a way of naming this method ... like "apply_load_policy" or "combine_like_load".

As discussed today in @SciTools/peloton :

@pp-mo yes I do agree that this is a bit mixed up.
TBH I think it's a result of my hurrying to get LOAD_POLICY + friends into the Iris 3.11 release, while not including combine_cubes, or referring to it, because there was not time (!)
In retrospect, I wish the loading controls referred to combine_cubes directly as in a former draft. And since @stephenworsley suggested making 'combine_cubes' a public thing, making all this "one thing" would have been preferable!
So with some hindsight, I would now favour :

  • rename LoadPolicy as CombinePolicy
  • include the "merge_unique" keyword as an option (attribute) of CombinePolicy
  • explicitly reference combine_cubes in the docs for LOAD_POLICY

@trexfeathers pointed out, a desire to rename 'LoadPolicy' is not worth making an rc1 release for. But we can still do all the above, subsequent to 3.11, while leaving LoadPolicy in the top-level Iris API, as an (effective) alias of CombinePolicy.

Where from here ?

I'll set about making those changes here + we can see what it looks like


Applies a :meth:`combine`.

Parameters
----------
options : dict or str
Settings, as described for :meth:`iris.LOAD_POLICY.set`.
Defaults to current :meth:`iris.LOAD_POLICY.settings`.
merge_require_unique : bool
Value for the 'unique' keyword in any merge operations.
kwargs : dict
Individual settings, as described for :meth:`iris.LOAD_POLICY.set`.

Returns
-------
CubeList of :class:`~iris.cube.Cube`

.. Note::
The ``support_multiple_references`` keyword/property has no effect on the
:func:`combine_cubes` operation : it only takes effect during a load operation.

"""
from iris.io.loading import combine_cubes

return combine_cubes(
self, options=options, merge_require_unique=merge_require_unique, **kwargs
)

def combine_cube(self, options=None, merge_require_unique=False, **kwargs):
"""Combine cubes as for :meth:`combine`, expecting a single cube result.

Parameters
----------
options : dict or str
Settings, as for :meth:`combine`.
merge_require_unique : bool
Value for the 'unique' keyword in any merge operations.
kwargs : dict
Individual settings, as for :meth:`combine`.

Returns
-------
CubeList of :class:`~iris.cube.Cube`

"""
from iris.io.loading import combine_cubes

result = combine_cubes(
self, options=options, merge_require_unique=merge_require_unique, **kwargs
)
n_cubes = len(result)
if n_cubes == 1:
(result,) = result
elif n_cubes == 0:
raise ValueError("'combine' returned no cubes.")
else:
raise ValueError(f"'combine' result is not a single cube : {result!r}.")
return result

def realise_data(self):
"""Fetch 'real' data for all cubes, in a shared calculation.

Expand Down
61 changes: 45 additions & 16 deletions lib/iris/io/loading.py
Original file line number Diff line number Diff line change
Expand Up @@ -391,42 +391,71 @@ def context(self, settings=None, **kwargs):
LOAD_POLICY = LoadPolicy()


def _combine_cubes(cubes, options, merge_require_unique):
def combine_cubes(cubes, options=None, merge_require_unique=False, **kwargs):
"""Combine cubes as for load, according to "loading policy" options.

This provides a facility like merge/concatenate, but with more flexibiity.
See :class:`iris.LoadPolicy` for details.

Applies :meth:`~iris.cube.CubeList.merge`/:meth:`~iris.cube.CubeList.concatenate`
steps to the given cubes, as determined by the 'settings'.

Parameters
----------
cubes : list of :class:`~iris.cube.Cube`
A list of cubes to combine.
options : dict
Settings, as described for :meth:`iris.LOAD_POLICY.set`.
options : dict or str, optional
Operation settings, as described for :meth:`iris.LOAD_POLICY.set`.
Defaults to current :meth:`iris.LOAD_POLICY.settings`.
merge_require_unique : bool
Value for the 'unique' keyword in any merge operations.
merge_require_unique : bool, default False
Value applied as the 'unique' keyword in any merge operations.
See :meth:`~iris.cube.CubeList.merge`.
kwargs : dict
Individual option values. These take precedence over the 'options' arg, as
described for :meth:`iris.LOAD_POLICY.set`.

Returns
-------
:class:`~iris.cube.CubeList`

.. Note::
The ``support_multiple_references`` keyword/property has no effect on the
:func:`_combine_cubes` operation : it only takes effect during a load operation.

Notes
-----
TODO: make this public API in future.
At that point, change the API to support (options=None, **kwargs) + add testing of
those modes (notably arg type = None / str / dict).

A ``support_multiple_references`` keyword/property is a valid setting, but it
has no effect on a :func:`combine_cubes` call, because it only acts during a
load operation.
"""
from iris.cube import CubeList

if not isinstance(cubes, CubeList):
cubes = CubeList(cubes)

err = None
if options is None:
options = LOAD_POLICY.settings()
elif isinstance(options, str):
if options in LoadPolicy.SETTINGS:
options = LoadPolicy.SETTINGS[options]
else:
err = (
"Unrecognised settings name : expected one of "
f"{tuple(LoadPolicy.SETTINGS)}."
)
elif not isinstance(options, Mapping):
err = (
f"Unexpected 'options' arg {options!r} : "
"expected a None, Mapping or string."
)

if err:
raise ValueError(err)

if kwargs:
options = options.copy() # do not modify the baseline settings !
options.update(kwargs)

return _combine_cubes_inner(cubes, options, merge_require_unique)


def _combine_cubes_inner(cubes, options, merge_require_unique):
while True:
n_original_cubes = len(cubes)
sequence = options["merge_concat_sequence"]
Expand All @@ -449,7 +478,7 @@ def _combine_cubes(cubes, options, merge_require_unique):


def _combine_load_cubes(cubes, merge_require_unique=False):
# A special version to call _combine_cubes while also implementing the
# A special version to call combine_cubes while also implementing the
# _MULTIREF_DETECTION behaviour
options = LOAD_POLICY.settings()
if (
Expand All @@ -462,7 +491,7 @@ def _combine_load_cubes(cubes, merge_require_unique=False):
if _MULTIREF_DETECTION.found_multiple_refs:
options["merge_concat_sequence"] += "c"

return _combine_cubes(cubes, options, merge_require_unique=merge_require_unique)
return combine_cubes(cubes, options, merge_require_unique=merge_require_unique)


def load(uris, constraints=None, callback=None):
Expand Down
Loading
Loading