Setting per schema 'severity' not working

[MrChocolateMoose]

Bug Report

I have a schemas.json that has various schemas in it. I am not quite ready to make some of them violations (which I would then escalate to errors). I would like to keep those particular ones as "informational" for now.

In the document, it makes it seem like that you can accomplish this:
https://sphinx-needs.readthedocs.io/en/latest/schema/index.html#id2

However, I am not able to achieve that result

Expected Results

I have several schema items with severity "violation" and several with severity "info".

Simple Stripped Down Example

{
    "schemas": [
        {
            "id": "field-1-enum",
            "severity": "violation",
            "message": "field must be one of A, B, C",
            "select": {
                "required": [
                    "my_field_1"
                ],
                "properties": {
                    "type": {
                        "const": "my_type"
                    }
                }
            },
            "validate": {
                "local": {
                    "required": [
                        "my_field_1"
                    ],
                    "properties": {
                        "my_field_1": {
                            "type": "string",
                            "enum": [
                                "A",
                                "B",
                                "C",
                            ]
                        }
                    }
                }
            }
        },
        {
            "id": "field-2-enum",
            "severity": "info",
            "message": "field must be one of D, E, F",
            "select": {
                "required": [
                    "my_field_2"
                ],
                "properties": {
                    "type": {
                        "const": "my_type"
                    }
                }
            },
            "validate": {
                "local": {
                    "required": [
                        "my_field_2"
                    ],
                    "properties": {
                        "my_field_2": {
                            "type": "string",
                            "enum": [
                                "D",
                                "E",
                                "F",
                            ]
                        }
                    }
                }
            }
        },

I expect if I violate one of type Info that I see it displayed as an informational item in the build process, otherwise as a violation (which would be categorized as a warning?).

Actual Results

When I violate informational items, it is showing that the severity is "violation" (Default).

I have found only having "info" also fails. It's like it isn't applying at all.

Library Details

I am on sphinx-needs 6.0.1

[ubmarco]

Please try the latest master commit, I pushed a fix for another bug report that might help you.
Just have not released that yet.

[ubmarco]

Reported issue #1513 and PR #1535 that I mentioned.
Please check whether your report is a duplicate / the PR fixes your issue.

The minimum needs_schema_severity, together with the given rule severity, decides about whether a rule is reported or not. This is true for both the Sphinx warning and for for schema_violations.json.

That should be working, I also created additional tests in above PR. Please confirm this, then I will release a new version.

[MrChocolateMoose]

Whoops, I'll try to keep up on the delta changes a bit better.

I will updated my sphinx-needs library to HEAD of master and retest and confirm by the end of next week.

Thanks!

[ubmarco]

@MrChocolateMoose any feedback on this? I have more people complaining about the severity not working.
I want to release that bugfix.

[MrChocolateMoose]

Hey @ubmarco,
I was on vacation, but I am back now. I saw your message and tried it out:

e.g.

WARNING: Need 'MY_COMPONENT' has validation errors:
  Severity:       info
  Field:          
  Need path:      My Component
  Schema path:    my-field-enum[17] > local > required
  User message:   my-field must be one of A, B, C
  Schema message: 'my-field' is a required property [sn_schema.local_fail]

1. ✅ I am correctly seeing the severity being reflected:
2. ⚠️ ❓ Shouldn't "info" messages be regarded as NON-warnings? My sphinx project is now set up to treat warnings as errors as you suggested. I can't really use the feature as intended because all of my schema violations of severity "info" would cause build errors. (e.g. build finished with problems, 510 warnings (with warnings treated as errors). )

[ubmarco]

Hi @MrChocolateMoose, how dare you go to vacation. Nah good for you :)

Thanks a lot for the feedback. So this comes down to specification.
I have to admit the setting needs_schema_severity is not 100% clear on that. The idea is to set the minimum severity required for a report. For Sphinx a report is currently emitted as warning. Any report below that threshold is not reported at all. The same applies to schema_violations.json.

I am rethinking the handling now, due to your feedback. I revisited the SHACL spec which was my inspiration for the severity implementation. I like what they are doing, the severities are just reported and downstream tools can do what they want with it.

I will change the implementation to this:

• Violations are always reported, irrespective of the severity. Both in Sphinx and schema_violations.json.
• The needs_schema_severity determines the min level for a Sphinx warning, so users can react to those with a different exit code using -W. needs_schema_severity behaves like a split between logger.info and logger.warning.
• Any report lower than needs_schema_severity is logged with info.
• Any report higher or equal than needs_schema_severity is logged with warning with type and subtype.
• schema_violations.json always has the complete info, and the log_lvl field will have info or warning, according to what Sphinx did.

Does this make sense to you? Will prepare a PR.