Create Method to Return the values of all options used, including default values.

[richardhathaway-8451]

Create a method to return the values of all the options used to evaluate an Evaluatable, including default values. This will help with tracking parameters and showing users of an Evaluatable all of the parameters going into calculating the result.

@dataset
def foo(x = Option('X',1)):
    return x

@dataset
def bar(
    _foo = foo,
    y = Option('Y',2),
    z = Option('Z',3)
):
    return _foo + y + z

#something like this:
>>> bar.get_all_options({'Y': 100})
{'X':1, 'Y':100, 'Z':3}

[jjmarchewitz]

I've started looking into this. I am new to this project so I am still learning the ropes :)

It seems that we would need to partially evaluate the Evaluatable, up until the point where all of the Options are resolved from Switches, Overloads, etc. Right now, this evaluation is part of the .evaluate() call tree. I think the best way to accomplish this would be to pull out this logic into a .evaluate_options() or .evaluate_params() method that is part of the Evaluatable base class.

Ultimately we would be trying to get an instance of Arguments that results from evaluating these Switches, Overloads, etc. We could implement __dict__() for Arguments as the output of this new .evaluate_options() method so that the end user would get back {'X':1, 'Y':100, 'Z':3} as per your example.

@austinwarner-8451 thoughts?

[jjmarchewitz]

After thinking about it further, this could probably be used to implement #14 as well

[austinwarner-8451]

@jjmarchewitz Sorry, been a busy few weeks at 84.51 and forgot to respond to this.

I agree though, this would probably require a new method in the Evaluatable base class. One thing I've been struggling with is that adding a new method to the base class is technically a breaking change. The intention is that third-party libraries should be able to implement their own Evaluatable types (in fact, we have an internal package at 84.51 that does exactly this), and even if a default implementation is provided (which might not be possible), we could cause a name conflict with a third-party subclass (e.g. we name the method .evaluate_options and there is a subclass that already has a method called .evaluate_options for another purpose). I can think of a couple different ways to handle this.

First, we could just make another base class called like CanEvaluteOptions that Evaluatable does not inherit from by default. That would mean third party subclasses could opt-in. The downside is we cannot guarantee that every implementor of Evaluatable has an implementation, so the results of calling .evaluate_options are not guaranteed to be complete when using third-party extensions to labrea.

Second, we could just name the method something that we can reasonably expect to not be implemented by subclasses (something like .__labrea_evaluate_options__?). In this case it might be beneficial to explicitly state in the docs that .__labrea* should be reserved for hooks provided by the labrea framework.

Third, this is the kind of thing that traits in Rust are perfect at. Unfortunately, there is no such construct in Python, but I have for a while thought about writing a package that attempts to recreate this behavior in Python. In that world, we wouldn't need to worry about namespace collisions, since the implementation would live outside of the class body. For example:

@trait
class CanEvaluteOptions:
    def evaluate_options(self, options: Options) -> Options:
        return NotImplemented


class CustomEvaluatable(Evaluatable):
    """Main ABC implementation lives here"""


@CanEvaluateOptions.implement
class _(CustomEvaluatable):
    def evaluate_options(self, options: Options) -> Options:
        """Trait implementation lives here"""
        ...


obj = CustomEvaluatable()


evaluated_options = CanEvaluateOptions.evaluate_options(obj)

This would require the biggest lift, but I think it would give us the most flexibility around having as many opt-in traits as we want and not needing to worry about the names of the methods. The biggest downside is the up-front cost of creating a framework for traits, and then anyone who wants to implement it downstream needs to understand this sort of non-pythonic way of doing thins.

[jjmarchewitz]

@austinwarner-8451 no worries, I got busy over the last month myself. I did manage to get a base case working! Would you prefer that I fork this repo and work from there, or would you mind giving me limited write permissions (just to create branches and MRs)?

Just for now to get it working, I added .evaluate_options to the Evaluatable base class. I think the idea to add a CanEvaluateOptions mixin would be hard to implement because it would depend on the subclass also implementing Evaluatable. Either we can trust/check that Evaluatable is implemented, or we could make CanEvaluateOptions inherit from Evaluatable. The problem with the latter is that now Evaluatable shows up in two places in the inheritance tree, which causes method resolution problems.

Purely for my own understanding I want to ask - what would be the impact to 84.51 of the breaking change to Evaluatable? Roughly how many classes implement Evaluatable?

P.S. I added test checks for .evaluate_options to all the Options test cases and they are all passing now :D It took some experimenting to get Templates to work properly