Avoid log(0) in KL divergence

[bz-e]

…denominator and added a test case

Describe your change:

Fixes #12233
Added type NONE to make it pass type checking, and added a small constant to the kullback_leibler_divergence method to fix the bug of numerator and denominator being 0, and also added a test case.

• Add an algorithm?
• Fix a bug or typo in an existing algorithm?
• Add or change doctests? -- Note: Please avoid changing both code and tests in a single pull request.
• Documentation change?

Checklist:

• I have read CONTRIBUTING.md.
• This pull request is all my own work -- I have not plagiarized.
• I know that pull requests will not be merged if they fail the automated tests.
• This PR only changes one algorithm file. To ease review, please open separate PRs for separate algorithms.
• All new Python files are placed inside an existing directory.
• All filenames are in all lowercase characters with no spaces or dashes.
• All functions and variable names follow Python naming conventions.
• All function parameters and return values are annotated with Python type hints.
• All functions have doctests that pass the automated testing.
• All new algorithms include at least one URL that points to Wikipedia or another similar explanation.
• If this pull request resolves one or more open issues then the description above includes the issue number(s) with a closing keyword: "Fixes Avoid log(0) in KL divergence #12233 ".

[imSanko]

what do you actually want to change ?

[bz-e]

what do you actually want to change ?

To be precise, I want to change the kullback_leibler_divergence method to fix the bug that the final return value is INF when the numerator or denominator of np.log(y_true / y_pred) is zero. The loss of precision after the change will not affect the calculation of machine learning model parameters. However, strictly following the latest contribution guidelines, I need to fix a type error in the previous version before I can submit it.

[kevin1kevin1k]

@bz-e I think we do not intend to change the default behavior when all y_true are nonzero.
A better way might be to mask out all the zero entries and only sum them.

[bz-e]

@bz-e I think we do not intend to change the default behavior when all y_true are nonzero. A better way might be to mask out all the zero entries and only sum them.

I think this is the lowest time complexity solution to the issue #12233 .

[kevin1kevin1k]

I think this is the lowest time complexity solution to the issue #12233 .

I meant you could do something like the following (need not be identical, just for demo), and the complexity stays linear.
Also IMHO since this repo is more for educational purpose than for practical/production usage, correctness is more favorable compared to efficiency

    mask = y_true != 0
    y_true_filtered = y_true[mask]
    y_pred_filtered = y_pred[mask]
    kl_loss = y_true_filtered * np.log(y_true_filtered / y_pred_filtered)

[bz-e]

I meant you could do something like the following (need not be identical, just for demo), and the complexity stays linear. Also IMHO since this repo is more for educational purpose than for practical/production usage, correctness is more favorable compared to efficiency

    mask = y_true != 0
    y_true_filtered = y_true[mask]
    y_pred_filtered = y_pred[mask]
    kl_loss = y_true_filtered * np.log(y_true_filtered / y_pred_filtered)

For educational purposes I think you are right, this is the method with the least changes.