Does Airbyte support long-polling (streaming API) paradigm?

[John Bai (pai911)]

For example, Twitter provides a streaming API like this https://developer.twitter.com/en/docs/twitter-api/v1/tweets/filter-realtime/guides/connecting

The HTTP connection will be kept alive as long as can be, and the client can incrementally download realtime data.

I'm thinking this will make a job running potentially very long, and records will be emitted as they are sent to us.

Just need to make sure if this is something supported by Airbyte? Is there gonna be any problem if I do implement a connector to connect to an API endpoint that'll keep the connection indefinitely?

Thank you!

[M. Marx (marcosmarxm)]

I don't think so, Airbyte has a default of 72h (3 days) timeout for jobs. The paradigm source connectors use today is batch.