What should the metrics API look like?

[hildeweerts]

To get the conversation going on what the metrics API should look like and better explain my own view, I created a quick mock-up: metrics.txt. Please feel free to comment!

An issue @MiroDudik brought up is that this design does not provide a way to retrieve both "group" scores and "overall" scores, something that was possible with the previous Summary object. I am not sure what would be a nice way to incorporate this pattern, so if people have thoughts: please share!

Tagging @romanlutz @riedgar-ms @adrinjalali

[romanlutz]

Thanks for getting this started @hildeweerts !

I have some concerns about the naming but I'll focus on the functionality since we can always adjust the naming. I'll walk through an example based on your file to figure out whether I understand it right.

Let's say I care about accuracy. Then I can instantiate score_groups as

score_groups(y_true, y_pred, sensitive_features, metric='accuracy')

which would mean we need to map those metric strings to actual metric functions. Maybe it's easier to pass the function itself?

score_groups(y_true, y_pred, sensitive_features, metric=sklearn.metrics.accuracy_score)

Regardless of that detail, I'd get a Scores object back. I'm not 100% sure I understand what that one would do, but I'll try:

scores = ... # my Scores object
scores.disparity("difference")  # returns diff between min and max?
scores.difference()  # identical to disparity("difference")
scores.min()
scores.max()
scores.to_dict()  # would that be similar to what the group summary is right now? or would this just be the individual ones for all groups?
scores.to_dataframe()  # that's probably just the scores per group, which makes me think that to_dict is also just per group (?)

aggregate seems to be a helper function that can give you difference, ratio, max, min etc., right?

This reminds me very much of our existing group_summary by the way! That one looks like this:

group_summary(metric_function, y_true, y_pred, *, sensitive_features, indexed_params=None, **metric_params)

and there are individual ones per metric, e.g. accuracy_score_group_summary(y_true, y_pred,....). The only odd part there is that you need to pass this group_summary object to functions like difference_from_summary(...) to get the values out of there, whereas the scores.difference() way of writing it is more natural. Perhaps even scores.difference ?

For each individual metric there's the <metric>_grouped way of using it.

false_positive_rate_grouped(y_true, y_pred, sensitive_features, disparity="difference", aggregation="max")

What I'm wondering is how that's different from

score_groups(y_true, y_pred, sensitive_features, metric='false_positive_rate').disparity('difference')

I remember that the complex object (here called Scores) is not necessarily what we want in every situation, so perhaps redundancy is intended (?).

[hildeweerts]

which would mean we need to map those metric strings to actual metric functions. Maybe it's easier to pass the function itself?

Perhaps both should be possible? I don't think we'll be adding a lot of "base" metrics (we should probably think of a good name for those) on a regular basis, so we can just use scikit-learn's predefined values in addition to others that are not directly accessible from scikit-learn (e.g. false positive rate). For first time users, I can imagine having strings to choose from is a bit less intimidating than having to find (or even define) the appropriate (scikit-learn) function yourself? But this is just speculation from my side :)

aggregate seems to be a helper function that can give you difference, ratio, max, min etc., right?

This is a good question - I think my naming is not super clear. I was actually thinking that there should be one type of function for getting the "disparity" between all groups (i.e. ratio or difference); i.e. scores.disparity("difference") would return all differences between groups. In the case of two sensitive groups, this is just a single number, but in the case of multiple sensitive groups, you still need to aggregate it somehow. At the moment, Fairlearn only allows for one possible aggregation, which is the difference between min and max. IMO, given that fairness can have so many different meanings in different contexts, it is important allow for some flexibility in defining a fairness metric. However, I'm not sure whether having a separate argument/function does more harm than good - i.e. is the increase in flexibility worth the increase in complexity?

This reminds me very much of our existing group_summary by the way! [...] and there are individual ones per metric, e.g. accuracy_score_group_summary(y_true, y_pred,....). The only odd part there is that you need to pass this group_summary object to functions like difference_from_summary(...) to get the values out of there, whereas the scores.difference() way of writing it is more natural. Perhaps even scores.difference ?

From what you describe I think it's very similar indeed! I agree that writing something like scores.difference() is more natural (pandas style), especially because this pattern allows you to easily access all possible methods in most IDE's. If you use an attribute rather than a method, does that mean you have to precompute all of them? In that case, I'd be in favor of a method.

I remember that the complex object (here called Scores) is not necessarily what we want in every situation, so perhaps redundancy is intended (?).

Yes, this redundancy is intended! Mostly because you can use something like false_positive_rate_grouped() directly as a scorer function in things like scikit-learn's grid search.

[riedgar-ms]

I would also be in favour of sticking to just passing around functions, rather than allowing for 'string or callable' all over the place. This may reflect the fact that I'm far more used to typed languages, and I'm seeing a lot of tests being required to make sure the string->fn mappings are set up correctly. I agree that score_groups() looks rather like the existing group_summary(), minus the support for passing extra arguments down into the underlying metric.

Can you clarify what the result of calling scores.to_dict() or scores.do_dataframe() would look like?

Supporting different aggregation methods (beyond simple max-min) is something I'm sure would be useful. I think that the aggregation method might look something like:

class Scores:
...
   def aggregate(self, f):
       return f(self)

We could provide things such as difference() and ratio() for f(), and if users needed some other aggregation, they'd be able to write it themselves fairly easily.

A note on attributes in Python... they're actually methods, and we used to use them in the GroupMetricResult class (removed in favour of Bunches a few months ago):
https://github.com/fairlearn/fairlearn/blob/29448633c776fbcb4f697d11c3354ef36b4cfd89/fairlearn/metrics/_group_metric_result.py#L45
In use, they look like fields, but they are actually method calls.

[hildeweerts]

I would also be in favour of sticking to just passing around functions, rather than allowing for 'string or callable' all over the place. This may reflect the fact that I'm far more used to typed languages, and I'm seeing a lot of tests being required to make sure the string->fn mappings are set up correctly.

From a developer perspective, I agree. But from a user perspective, I do think strings can be much easier to use. Particularly if you're familiar with scikit-learn.

Can you clarify what the result of calling scores.to_dict() or scores.do_dataframe() would look like?

It depends a bit on what we choose as an underlying data structure of the scores object. But it would allow (particularly novice) users to easily convert the current object to a data structure they are comfortable working with. E.g. if a user has called scores.difference() they can convert the result to a pandas dataframe and visualize it using e.g. matploblib.

To clarify further: in my head 'disparity' and 'aggregation' are two different steps; in the disparity step you define what a disparity between two groups is (i.e. either 'ratio' or 'difference') and the other how you aggregate those disparities between more than two groups (e.g. the min-max difference, average difference, max difference with one particular group of interest etc.). Would it be clearer if we would have something like scores.difference(aggregate="maxmin") or would that be even more confusing? I am also not sure how to convert that nicely to single scorer functions such as false_positive_rate_grouped(). And it would probably result in even more 'string or callable' patters ;)

I didn't know that about attributes - interesting!

[riedgar-ms]

I would also be in favour of sticking to just passing around functions, rather than allowing for 'string or callable' all over the place. This may reflect the fact that I'm far more used to typed languages, and I'm seeing a lot of tests being required to make sure the string->fn mappings are set up correctly.

From a developer perspective, I agree. But from a user perspective, I do think strings can be much easier to use. Particularly if you're familiar with scikit-learn.

You're probably right.... it just feels dirty and uncomfortable to me.

Can you clarify what the result of calling scores.to_dict() or scores.do_dataframe() would look like?

It depends a bit on what we choose as an underlying data structure of the scores object. But it would allow (particularly novice) users to easily convert the current object to a data structure they are comfortable working with. E.g. if a user has called scores.difference() they can convert the result to a pandas dataframe and visualize it using e.g. matploblib.

Could you propose a specific implementation? Right now, the returned object has an overall field, and a by_groups dictionary for the return (this was a bit more obvious in the days of the GroupMetricResult). I did look into flattening that to a dataframe (since one of the notebooks did that), but I always got hung up on how to do it. Specifically, what to call the rows when there's a danger that one of the subgroups could be called 'overall'.

To clarify further: in my head 'disparity' and 'aggregation' are two different steps; in the disparity step you define what a disparity between two groups is (i.e. either 'ratio' or 'difference') and the other how you aggregate those disparities between more than two groups (e.g. the min-max difference, average difference, max difference with one particular group of interest etc.). Would it be clearer if we would have something like scores.difference(aggregate="maxmin") or would that be even more confusing? I am also not sure how to convert that nicely to single scorer functions such as false_positive_rate_grouped(). And it would probably result in even more 'string or callable' patters ;)

Perhaps something like scores.aggregated_disparity(disparity='difference', aggregate='max') where both arguments can also be functions? The disparity function should probably just take two floats. The aggregate function would need something more elaborate, in case users wanted all their measurements to be relative to a particular subgroup (which they would smuggle into their callable).

[hildeweerts]

Could you propose a specific implementation? Right now, the returned object has an overall field, and a by_groups dictionary for the return (this was a bit more obvious in the days of the GroupMetricResult). I did look into flattening that to a dataframe (since one of the notebooks did that), but I always got hung up on how to do it. Specifically, what to call the rows when there's a danger that one of the subgroups could be called 'overall'.

Ah yes, I forgot about the overall/by_group pattern. Off the top of my head, I can see two possibilities. Either perform the call on the groups attribute only, i.e. you'd have something like scores.by_groups.to_dataframe(). If we only allow for one metric at the time, scores.overall will only contain one number, so I'd say it makes more sense to print it separately anyways.

The second option would be to use a multi-index dataframe. I think this would make the most sense if we allow for multiple metrics in the same summary. Which, if I think about it, would actually be quite useful. Is passing multiple metrics possible in the current API? The resulting dataframe could look something like this:

              fpr   fnr
overall NaN  0.10  0.20
group   A    0.05  0.10
        B    0.20  0.30
        C    0.01  0.05

Perhaps something like scores.aggregated_disparity(disparity='difference', aggregate='max') where both arguments can also be functions? The disparity function should probably just take two floats. The aggregate function would need something more elaborate, in case users wanted all their measurements to be relative to a particular subgroup (which they would smuggle into their callable).

I like this idea! 👍

[riedgar-ms]

I have seen MultiIndexes a bit (they appear in the Moment code), but I'm not fully familiar with their details. If that sort of layout would be useful to data scientists, it seems reasonable, though.

One thing which has just occurred to me: are disparities always symmetric (i.e. would you always define the function such that disparity(a, b) == disparity(b, a)) or are there times when that's not the case? There would be some significant effects on the required aggregate() function if not.

[romanlutz]

@riedgar-ms disparities are not always symmetric. Think of the FPR ratio of two groups, for example.

@adrinjalali and @MiroDudik should probably chime in here as well.

[riedgar-ms]

disparities are not always symmetric. Think of the FPR ratio of two groups, for example

Agreed. For that particular case, one could adjust the specification of the ratio, much like we specify differences to be absolute values. However, I'm sure that there will be other cases not amenable to something like that (especially if someone tries defining disparities on confusion matrices).

Per a separate discussion which @MiroDudik and I had about a presentation I was putting together, rather than calling it disparity(a, b) we might want to call it fairness_comparator(a, b). Thoughts on that?

The aggregate() function is more complicated. In addition to the matrix of fairness_comparator() results (which may not be symmetric), we also need to supply comparisons to the 'overall' metric (evaluated on the whole dataset) - I only understood some of the kinks on the GridSearch() pareto fronts when @MiroDudik pointed out that the 'dominant model' function worked relative to the overall metric, while the plot was for difference between subgroups.

We could include these in the matrix of fairness_comparator() results, but we'd have to be rather careful about indexing. There's always a danger that any name we might pick for 'overall' could also be a subgroup name (I've alluded to this issue above, I believe).

[hildeweerts]

I'm a bit confused where the fairness_comparator(a,b) would come to play exactly. Would it be a replacement for the aggregated_disparity() you suggested, for the function argument disparity, or something else?

If it is the former, I agree something like scores.fairness_comparator(disparity='difference', aggregate='max') would be nice, although I'd probably go for a more active verb like compare. But it depends a bit on what we're going to call the summary/scores object.

I think we should probably raise an error if 'overall' is used as a subgroup name. I think most issues can be dealt with by proper indexing, but there's always the issue of comparing the actual overall to subgroup 'overall'; i.e. in a table with all comparisons you'll have (overall, overall) twice and it's impossible to tell which 'direction' the disparity is going without looking up the original scores.

[MiroDudik]

Let me comment on a couple of issues:

Base metrics

I don't mind allowing both strings as well as callable objects. This is a common pattern around metrics in sklearn, where you can pass a string or a scorer object to crossvalidation routines. In fact our metrics engine already has some dictionaries of that kind, so it would be just a matter of adding another dict.

Aggregators

I am worried about over-engineering things. I don't think we should try to come up with generic aggregators (yet). I was thinking about using the following pattern for the Scores object

# the current behavior
scores.difference()

# some alternatives could be enabled with keyword arguments
scores.difference(relative_to='overall', aggregate='max_flip')
scores.difference(relative_to='fixed', fixed_group='male', aggregate='max')
scores.difference(relative_to='min', aggregate='max_flip') # default

# similarly for ratio:
scores.ratio(relative_to='max', aggregate='min_flip') # default
scores.ratio(relative_to='fixed', fixed_group='male', aggregate='min')
scores.ratio(relative_to='overall', aggregate='min_flip') 

The flip above means that you consider both a-b and b-a when taking the difference and a/b and b/a when taking the ratio. So the difference is the "absolute" difference. If it's confusing we could gave a separate scores.difference() and separate scores.signed_difference() and maybe similarly for the ratio.

It's not clear to me that we need to have a mechanism for custom aggregators. My sense is that for any custom behavior, it should be enough for the users to one of the following:

1. define their own functions that operate on Scores object (kinda like our "difference_from_summary")
2. extend the Scores class
3. export Scores into a pandas DataFrame and take advantage of pandas aggregators

Exporting

I'm not yet sure here. There are a few other patterns that might be handy, e.g.,

by_group, overall = scores.to_frames()
difference = (by_group-overall).abs().max()
difference_to_male = (by_group-by_group['male']).min()

This is the one I was thinking about to avoid custom aggregators.

[hildeweerts]

I agree. It's probably best to keep things as simple as possible and allow for flexibility by easy exports to pandas.

I like your suggested pattern to avoid generic aggregators. My hard-wired instinct to make things complicated says we might want to add None options. That allows you to better understand/debug weird results (e.g. due to accidentally flipping the binary target outcome).

I'm wondering what would be a good way to translate this to the fairness metrics, e.g. false_positive_rate_grouped().

[riedgar-ms]

Fair enough on avoiding the custom aggregators - especially if the DataFrame export can reasonably allow users to define things to their hearts' content.

My main concern is the complexity of the 'dispatch' logic at the top of the difference() and ratio() methods. That's going to be an interesting set of tests to write, to make sure we catch all the invalid combinations (unless I'm just thinking too much like a typed programmer again....).

I'm not a huge fan of the flip suffix, but I can't think of anything better.