feat: connect to remote browser services

[l2ysho]

This is still WIP, but I will left comment on places which are worth to look at.

[l2ysho]

Browserless is good example of minimal implementation

temp-examples/examples/browserless-playwright.ts

const endpointUrl = `wss://production-sfo.browserless.io?token=${token}`

class BrowserlessProvider extends RemoteBrowserProvider {
    async connect() {
        return { url: endpointUrl };
    }
}

[l2ysho]

On the other hand browserbase need full implementation

temp-examples/examples/browserbase-playwright.ts

class BrowserbaseWsProvider extends RemoteBrowserProvider<{ id: string }> {
   type = 'websocket' as const;
   maxOpenBrowsers = 2;

    async connect() {
        const response = await fetch('https://api.browserbase.com/v1/sessions', {
            method: 'POST',
            headers: { 'x-bb-api-key': apiKey, 'Content-Type': 'application/json' },
            body: JSON.stringify({ projectId }),
        });

        if (!response.ok) {
            throw new Error(`Failed to create Browserbase session: ${response.status} ${response.statusText}`);
        }

        const session = await response.json();

        const url = `wss://connect.browserbase.com?apiKey=${apiKey}&sessionId=${session.id}`;
        return { url, context: { id: session.id } };
    }

    async release({ id }: { id: string }) {
        await fetch(`https://api.browserbase.com/v1/sessions/${id}`, {
            method: 'POST',
            headers: { 'x-bb-api-key': apiKey, 'Content-Type': 'application/json' },
            body: JSON.stringify({ status: 'REQUEST_RELEASE' }),
        })
    }
}

[barjin]

Should we have a separate package (@crawlee/remote or similar) with these implementations for major remote browser providers?

So our users can do

import { BrowserbaseBackend } from '@crawlee/remote';
import { PlaywrightCrawler } from '@crawlee/playwright';

new PlaywrightCrawler({
    launchContext: { 
        remoteBrowser: new BrowserbaseBackend({ url: '', token: '', ...})
    }
})

Imo this could be really useful. Definitely not necessary to add in this PR, though, it looks pretty self-contained.

[l2ysho]

Somewhere in process we dropped this idea, but I am open to create an issue. In general (at least for services I worked with) setup is similar service to service. Maybe also this class BrowserbaseWsProvider extends RemoteBrowserProvider is too much boilerplate for what is really does.

[l2ysho]

I am aware that this is not the best name for it

[l2ysho]

traditional WS does not return context (as CDP does) so no way to pass this context to another browser instance

[l2ysho]

locally this is handled by launchPersistentContext but here we have to grab context from remote browser

[janbuchar]

I guess it shouldn't be a problem, but can you wire up https://gologin.com/docs/api-reference/sdks/nodejs-sdk into crawlee with this?

[barjin]

Thank you @l2ysho! (and sorry for putting the review away for this long 😅 )

The implementation seems mostly solid to me. During testing, I came across a few rough edges I'd like to discuss ⬇️

[barjin]

Perhaps we should enforce mutual exclusivity between launchOptions and connectOptions on the type level?

As a user, I don't know if PlaywrightCrawler will launch a local dark-theme browser, or will connect to the remote one (or a combination of these)?

[barjin]

Btw, it seems that different launchOptions are ignored when using connectOptions and connectOverCDPOptions (and some are ignored in both).

Maybe I'm being too rough, but I'd just enforce having at most one of launchOptions | connectOptions | connectOverCDPOptions for now. We can always relax this if the need ever arises.

[l2ysho]

true, I am afraid there will be more funny stuff after my last 6 rewrites of this :] Lets throw when launchOptions is used with remote and do mutually exclusive connectOptions | connectOverCDPOptions | remoteBrowser (also throw)

[barjin]

Options for connecting to a remote browser via WebSocket.

Both the Playwright's internal "client-server" protocol (no public name that I know of) and CDP are based on WebSocket (both are higher-level protocols).

Perhaps we can make this into type?: 'cdp' | 'playwright'?

[l2ysho]

I guess I will drop that websocket completely, it is specific to playwright (and obsolete ?). cc @janbuchar

[barjin]

Au contraire - according to the docs, the Playwright-specific protocol offers more detailed control over the browser. Especially since this is inside playwright-launcher.ts, we should imo keep it.

[barjin]

nvm, I see the commit message now

Callers who genuinely need Playwright's connect() can still use connectOptions directly.

sounds reasonable 👍

[barjin]

Should we have a separate package (@crawlee/remote or similar) with these implementations for major remote browser providers?

So our users can do

import { BrowserbaseBackend } from '@crawlee/remote';
import { PlaywrightCrawler } from '@crawlee/playwright';

new PlaywrightCrawler({
    launchContext: { 
        remoteBrowser: new BrowserbaseBackend({ url: '', token: '', ...})
    }
})

Imo this could be really useful. Definitely not necessary to add in this PR, though, it looks pretty self-contained.

[barjin]

Is there something we can express with RemoteBrowserConfig, that we cannot do with the other interfaces?

The RemoteBrowserConfig interface is IIUC halfway between the connectOptions object (simple) and the RemoteBrowserProvider class (for advanced use cases)... and I'm not convinced we need all three.

edit: perhaps we can keep using the ...Config internally, but export only ...Provider?

[barjin]

Some providers (e.g., Browserless) authenticate using a token in the query params (docs), so we'd leak the secret anyway.

It feels like this should be the logger's responsibility, and we shouldn't deal with this here. How about, e.g., logging just the hostname in the error messages?

[barjin]

So... which one is right? 😄

crawlee/packages/playwright-crawler/src/internals/playwright-launcher.ts

Line 97 in 779105a

* Takes precedence over `connectOverCDPOptions` / `connectOptions` if both are set.

It seems that the comment there is wrong, and this error message is right, see example:

import { PlaywrightCrawler } from './packages/playwright-crawler/src/index.js';
import { chromium, firefox } from 'playwright';

const firefoxServer = await firefox.launchServer();

const crawler = new PlaywrightCrawler({
    launchContext: {
        remoteBrowser: {
            connect: async () => {
                const chromiumServer = await chromium.launchServer();
                return { url: chromiumServer.wsEndpoint() };
            },
            type: 'websocket',
            release: async () => {}
        },
        connectOptions: {
            wsEndpoint: firefoxServer.wsEndpoint()
        }
    },
    requestHandler: async ({ page, log }) => {
        log.info(await page.evaluate(() => navigator.userAgent));
    },
});

await crawler.run(['https://crawlee.dev/js']);

This will print Firefox's user agent, i.e., remoteBrowser is dropped, connectOptions prevails

One more reason to enforce the mutual exclusivity on the type level 😄

[barjin]

Please let's also add a docs article about how to set up the remote browser-enabled crawler (short page with examples, main limitations, etc. would imo suffice for now).

[l2ysho]

done, also temp-examples dropped