From reading the chromium implementation I believe the problem is that step 9.4.4 in "topmost popover ancestor" is missing the condition that okNesting is also true if isPopover is false (source), without that condition non-popover elements (i.e. dialog elements) are treated as if they were "auto" popovers and are disallowed from being descendants of "hint" popovers.
Another problem is that "nearest inclusive open popover" seemingly hadn't been updated to allow "hint" popovers.

cc @mfreed7 @nt1m @josepharhar

Are you sure the problem doesn't all just boil down to "nearest inclusive open popover" not including hint at step 2.1? The "topmost popover ancesor" algorithm does seem to handle hints, e.g. step 9.4.4.

@mfreed7

Are you sure the problem doesn't all just boil down to "nearest inclusive open popover" not including hint at step 2.1? The "topmost popover ancesor" algorithm does seem to handle hints, e.g. step 9.4.4.

My comment about "topmost popover ancestor" isn't about it handling hints but specifically about it not allowing dialog elements to be nested within hint popover ancestors.

Step 9.4.4 currently only sets okNesting if the descendant is "hint" or the ancestor is "auto", this translates to "anything can be nested within auto popovers but only hints can be nested within hints".
Chromium's implementation also sets okNesting if the ancestor is not a popover (i.e. a non-popover dialog), this means dialogs can be nested within either type.

See the following table for a more visual reference, the "topmost popover ancestor" steps say the "?" should be "No" while Chromium and WPT say it should be "Yes".

The note above those steps also suggests it should be a "Yes":

If the provided element is a top layer element such as a dialog which is not showing as a popover, then topmost popover ancestor will only look in the node tree to find the first popover.

For some extra motivation: this non-spec behaviour is implemented in Chromium, Webkit, Firefox and Ladybird, in addition to being expected by WPT

"nearest inclusive target popover for invoker" also needs to be updated to accept "hint" popovers to match existing light dismissal implementations.

On the one hand, it's probably not a good idea for developers to open a modal dialog inside of a hint popover. I imagine that would be quite disruptive. On the other hand, without this provision, should developers do such a thing we'd be back to the same type of issues described in #9998. So I think the change @Gingeh describes is warranted.

Though a small aside; I think Webkit and Firefox pass this WPT due to a lack of support for "hint" popovers. So it is perhaps coincidental for them.