Support checking exception order in `RaisesGroup`

[gschaffner]

What's the problem this feature will solve?

Some tests need to assert that an exception group was raised with multiple exceptions in a particular order. Currently, making such an assertion about the group must be done somewhat manually, e.g. like

def assert_matches(
    exception: BaseException, raises: _pytest.raises.AbstractRaises[BaseException]
) -> None:
    assert raises.matches(exception), raises.fail_reason


def check_group(exc_grp: ExceptionGroup[Exception], /) -> bool:
    assert len(exc_grp.exceptions) == 2
    assert_matches(
        exc_grp.exceptions[0],
        pytest.RaisesExc(RuntimeError, check=check_body_exc),
    )
    assert_matches(
        exc_grp.exceptions[1],
        pytest.RaisesExc(RuntimeError, check=check_aclose_exc),
    )
    return True

with pytest.raises(ExceptionGroup, check=check_group):
    ...

or similarly via exc_info:

with pytest.raises(ExceptionGroup) as exc_info:
    ...
assert many_properties_of(exc_info.value) == ...

Describe the solution you'd like

raises and RaisesGroup have made asserting expected exceptions much more convenient and less mistake-prone than the old

try:
    ...
except ExpectedExceptionType as exc:
    assert some_property(exc) == ...
else:
    assert False

and

with pytest.raises(ExceptionGroup) as exc_info:
    ...
assert enough_properties_about_every_exc_in(exc_info.value) == ...

It would be convenient if RaisesGroup also supported cases where the group has an expected order.

In more detail, the proposal is: add an ordered: bool = True parameter to RaisesGroup. Setting ordered=False would cause RaisesGroup to use a simpler matching algorithm that does not attempt to reorder exceptions and retry AbstractRaises.matches on other children when a .matches on a child fails.

For example, two real-world use cases:

• We test a class that builds a BaseExceptionGroup containing any exception from the body of its async with and an exception raised during a cleanup operation in __aexit__. This assertion was being written as

  with pytest.RaisesGroup(
      pytest.RaisesExc(RuntimeError, check=check_body_exc),
      pytest.RaisesExc(RuntimeError, check=check_aclose_exc),
  ):
      ...

  until we realized that this wasn't asserting the order and we needed to assert the order explicitly.

• We test various error, exception, and shutdown semantics of a de/multiplexer class that has a task group containing two service tasks, can propagate an exception from the user (the body of the object's async with), can raise socket, TLS, and protocol state machine exceptions in the _serve_outgoing and _serve_incoming service tasks, and can raise another exception during a final shutdown operation in __aexit__ after the service task group exits earlier in multiplexer.__aexit__. There are various test cases that assert that async with multiplexer: stuff() raises the correct exceptions in the correct order for that case.

Alternative solutions

It's perfectly possible to make the necessary assertions manually without using RaisesGroup, similar to how expected ExceptionGroups were more often asserted before Trio added RaisesGroup, e.g.

with pytest.raises(BaseExceptionGroup) as exc_info:
    ...
assert len(exc_info.value.exceptions) == 3
assert isinstance(exc_info.value.exceptions[0], BrokenResourceError)
assert isinstance(exc_info.value.exceptions[0].__cause__, ...)
assert isinstance(exc_info.value.exceptions[1], RuntimeError)
assert property_a(exc_info.value.exceptions[1]) == ...
assert isinstance(exc_info.value.exceptions[2], RuntimeError)
assert property_b(exc_info.value.exceptions[2]) == ...

assert_matches(exc_info.value.exceptions[i], pytest.RaisesExc(...)) can sometimes simplify this a bit, but it's still a bit tedious and too easy to forget to assert the length and/or to forget to make assertions about one or more of the exc_info.value.exceptions, allowing them to slip through. It's a somewhat similar issue to group_contains: #11538 (comment).

Aside: while it is true that when using with AbstractRaises, using a shorter body of the with is preferrable (see: flake8-bugbear B908), in cases where the exception group gets raised out of a class's __aexit__ there isn't much of another choice in order to test the behavior. flake8-bugbear won't even yell about with big_AbstractRaises: async with thing: a_bunch_of_stuff(), because there is technically only one top-level statement within big_AbstractRaises's context.