Skip to content

tornado.gen.multi() results in mypy error #3142

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
exajuk opened this issue May 18, 2022 · 4 comments · Fixed by #3478
Closed

tornado.gen.multi() results in mypy error #3142

exajuk opened this issue May 18, 2022 · 4 comments · Fixed by #3478
Labels

Comments

@exajuk
Copy link

exajuk commented May 18, 2022

Hi!

I've converted some old code from tornado.gen based coroutines to native coroutines, in order to be able to add static type checking using mypy (version 0.950). However, I've got some problems after replacing an old yield [Dict[int, Future]] call with tornado.gen.multi:

old call:

response_dict = yield req_dict

new call:

response_dict = await tornado.gen.multi(req_dict)

with req_dict being built like this:

req_dict = {}
for nid, val in nodes.items ():
    req = HTTPRequest (uri, method = 'POST',
                       headers = {'Authorization' : self._token},
                       body = body, **http_params)
    req_dict.update ({nid : self._http_client.fetch (req)})

I thought that should work (according to the type annotations in the code), but I'm getting this mypy error:

error: Argument 1 to "multi" has incompatible type "Dict[Any, Future[HTTPResponse]]"; expected "Union[List[Union[None, Awaitable[Any], List[Awaitable[Any]], Dict[Any, Awaitable[Any]], Future[Any]]], Dict[Any, Union[None, Awaitable[Any], List[Awaitable[Any]], Dict[Any, Awaitable[Any]], Future[Any]]]]"

Am I doing sth. wrong or is this issue similar to #3093? Maybe _Yieldable should also allow Future[Any]? 🤔

@exajuk exajuk changed the title usage of tornado.gen.multi() results in mypy error tornado.gen.multi() results in mypy error May 18, 2022
@bdarnell
Copy link
Member

I think this may be related to covariance (see https://mypy.readthedocs.io/en/stable/generics.html#variance-of-generic-types), like #3119. I think some uses of Dict in type annotations may need to be changed to Mapping (just like some uses of List should be replaced by Sequence). Annotating the type of req_dict as Dict[Any, Awaitable] may work as a workaround.

@exajuk
Copy link
Author

exajuk commented May 24, 2022

Thanks for the workaround, but it didn't work for me. However, I've solved the problem by implementing my own version of multi based on native functions:

async def multi_coro(self, coro: Dict[Any, Awaitable], return_exceptions: bool = False) -> Dict[Any, Any]:
    res_list = await asyncio.gather(*coro.values(), return_exceptions=return_exceptions)
    return dict(zip(coro.keys(), res_list))

Since asyncio.gather preserves the order of the coroutines in the result list, this should behave identical to multi (as long as return_exceptions is False). At least it seems to work so far...

@bdarnell
Copy link
Member

bdarnell commented Jun 4, 2022

Yes, asyncio.gather is very similar to gen.multi (without the dict functionality). Most of the time they're interchangeable but there are some subtle differences (and I'm not sure I know all of them). The main difference has to do with what happens when Future.cancel is called (Tornado's support for cancellation is poor and it tends to just ignore that method; asyncio is better about actually cancelling things although this can cause surprises for code expecting the Tornado semantics).

@jrheard-seesaw
Copy link
Contributor

i'm experiencing this as well when using pyright. I'm seeing errors like this:

  error: Argument of type "list[CoroutineType[Any, Any, dict[str, Any] | str | None]]" cannot be assigned to parameter "children" of type "List[_Yieldable] | Dict[Any, _Yieldable]" in function "multi"
    Type "list[CoroutineType[Any, Any, dict[str, Any] | str | None]]" is not assignable to type "List[_Yieldable] | Dict[Any, _Yieldable]"
      "list[CoroutineType[Any, Any, dict[str, Any] | str | None]]" is not assignable to "List[_Yieldable]"
        Type parameter "_T@list" is invariant, but "CoroutineType[Any, Any, dict[str, Any] | str | None]" is not the same as "_Yieldable"
        Consider switching from "list" to "Sequence" which is covariant
      "list[CoroutineType[Any, Any, dict[str, Any] | str | None]]" is not assignable to "Dict[Any, _Yieldable]" (reportArgumentType)

I suspect that changing the type annotation of gen.multi's children param from Union[List[_Yieldable], Dict[Any, _Yieldable]], to Union[Sequence[_Yieldable], Mapping[Any, _Yieldable]], would make it more usable - but I haven't had a chance to sit down and prove this myself. Just wanted to add this comment in case it's helpful. Thx!

jrheard-seesaw added a commit to jrheard-seesaw/tornado that referenced this issue Apr 2, 2025
bdarnell added a commit that referenced this issue Apr 28, 2025
#3142: Improve usability of multi() and multi_future() type annotations
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants