(EAI-639) setup WebDataSource (#611)

* scraping web pages for content

* processing sitemap for full directory urls, processing some additional urls

* modify drone file for testing puppeteer in cron env

* tests for WebDataSource and helper functions

---------

Co-authored-by: mongodben <[email protected]>

Author: yakubova92

Diff for: .drone.yml

@@ -18,6 +18,7 @@ steps:
     image: node:18
     commands:
       - npm ci
+      - npx playwright install chromium --with-deps
       - npm run build
       - npm run lint
       - npm run test

Diff for: ingest-service.dockerfile

@@ -1,5 +1,8 @@
 # Build stage
-FROM node:18-alpine
+FROM node:18
+
+# Install Playwright with dependencies
+RUN npx playwright install chromium --with-deps
 
 WORKDIR /bin
 COPY . ./
@@ -8,7 +11,7 @@ RUN npm run bootstrap -- --scope='{mongodb-rag-core,mongodb-rag-ingest,ingest-mo
 RUN npm run build -- --scope='{mongodb-rag-core,mongodb-rag-ingest,ingest-mongodb-public}'
 
 # Add git for GitDataSource
-RUN apk add --no-cache git
+RUN apt-get update && apt-get install -y git
 
 ENV NODE_ENV=production