Ability to spool multipart form to temporary files

[vytas7]

Although Falcon is proud of (almost) never performing I/O behind the user's back, in some use cases it might desirable to iterate a multipart form more than once.

We could add a .spool() method to the MultipartForm class that returns a spooled object which could be iterated multiple times. Larger files would need to be saved in temporary files. Or we could use SpooledTemporaryFiles for all parts.

Ideally, the spooled object should be usable as a context manager, and it should dispose all the temporary files when exiting.

Also, the form could hold a reference to the spooled form, and delegate iteration to it upon repeated attempts. TBD whether this is a good idea.

What to do with the async side of things? Should we schedule SpooledTemporaryFile methods in an executor, or should we optionally support aiofiles?

[vytas7]

See also 0b01001001/spectree#435 for a use case in a downstream project.

[CaselIT]

how about we to this the stream level? That way the multipar form could just re-create itself starting with the stream at the beginning

@work I have a custom request that caches the stream content that's just something like this

class Request(falcon.Request):
    __slots__ = ()

    def peek_data(self) -> bytes:
        """Returns the data in the request body without consuming the bounded input stream"""
        bs = self.bounded_stream
        if isinstance(bs.stream, BytesIO):
            return bs.stream.getvalue()
        data = bs.read()
        bs.stream = BytesIO(data)
        bs._bytes_remaining = bs.stream_len
        return data

but this could be either a request option or a method in the request so that one could opt into this on a pre-request basis (note that the method/option could just wrap the stream in a reply stream so that it can reply the stream without loading everything in memory at the beginning)

[vytas7]

That's a good idea per se @CaselIT , but I think we already have this one captured in #1746.

Moreover, I don't think it would be as effective here, because that would force the user of, e.g., Spectree to enable stream caching for all requests, which may not be desirable in the general case. Also, it is less efficient to parse the form twice or more, the successive times from disk.

[CaselIT]

the caching could on a per-request basis, not for all request.

it is less efficient to parse the form twice or more, the successive times from disk.

I'm not following how this proposal would not parse the form multiple times, you want to cache the pickle of the parsed form? that doesn't sound great

[vytas7]

I've added the needs-decision label, it is not clear we necessarily want to implement it this way.

But if we do, it would probably be something along the lines of https://github.com/0b01001001/spectree/pull/435/files. Basically, we would pipe the content of individual parts into temporary files, like other frameworks do.