delete old code

Author: mjarkk

README.md

@@ -1,261 +1,125 @@
-# For Deno client readme [click here](./DENO_README.md)
-
-# For the v2 scraper client readme [click here](./V2_README.md)
-
 # RT-CV scraper client
 
-A client that can be spawned by a scraper to ease communication with [RT-CV](https://github.com/script-development/RT-CV)
-
-## Communication flow
-
-```
-custom scraper <> rtcv_scraper_client (this project) <> RT-CV
-```
-
-1. The custom scraper spawns rtcv_scraper_client as child process
-2. The custom scraper sends it's credentials to the child process via stdin
-3. rtcv_scraper_client handles the authentication and reports if it went successfull
-4. The custom scraper starts scraping and sends every scraped result to it's child process (rtcv_scraper_client)
-5. rtcv_scraper_client sends the scraped data to rt-cv and reports if it was successfull
-6. ...
-
-## Methods
-
-### Authentication
-
-There are 3 authentication methods
-- `set_credentials` if you have one RT-CV server
-- `set_multiple_credentials` if you have multiple RT-CV servers
-- `set_mock` if you want to debug a scraper, this mocks the RT-CV server
+A helper program that aims to ease the communication between a scraper and [RT-CV](https://github.com/script-development/RT-CV)
 
-#### `set_credentials`
+## How does this work?
 
-Set credentials and location of a RT-CV server
+This scraper client handles authentication and communication with RT-CV and beside that also has a cache for the already fetched reference numbers.
 
-Example input
+This scraper works like this:
 
-```json
-{"type":"set_credentials","content":{"server_location":"http://localhost:4000","api_key_id":"111111111111111111111111","api_key":"ddd"}}
-```
-
-Ok Response
-
-```json
-{"type":"ok"}
-```
-
-#### `set_multiple_credentials`
+1. You run `rtcv_scraper_client` in your terminal
+2. The scraper client reads `env.json` and authenticates with RT-CV
+3. The scraper client spawns the program you have defined as args for the `rtcv_scraper_client`, for example with an npm scraper it would be something like `rtcv_scraper_client npm run start`
+4. Your scraper can now easially talk with `RT-CV` via `rtcv_scraper_client` using http requests where the http server address is defiend by a shell variable set by the scraper client `$SCRAPER_ADDRESS`
 
-Set credentials and location of multiple RT-CV servers
+## Why this client?
 
-Note that we need to define which server the primary should be, the primary server is used to fetch secrets and recently scraped reference numbers
+Every scraper needs to communicate with RT-CV and the amound of code that require is quite a lot.
 
-Example input
+If we have the same code for communicating with RT-CV we only have a single point of failure and updating / adding features is easy.
 
-```json
-{"type":"set_credentials","content":[{"primary":true,"server_location":"http://localhost:4000","api_key_id":"111111111111111111111111","api_key":"ddd"},{"server_location":"http://localhost:4000","api_key_id":"111111111111111111111111","api_key":"ddd"}]}
-```
+## Example
 
-Ok Response
+A Deno example
 
-```json
-{"type":"ok"}
+```ts
+// denoexample.ts
+const req = await fetch(Deno.env.get('SCRAPER_ADDRESS') + '/users')
+const users = await req.json()
+console.log(users)
 ```
 
-#### `set_mock`
-
-Enable mock mode
-
-Example input:
-
-```json
-{"type":"set_mock","content":{}}
-```
-
-You can also provide mock secrets for the `*_secret` methods.
-
-This object need to follow the following type convention `key (string) -> value (any)`
-
-```json
-{"type":"set_mock","content":{"secrets":{"users": [{"username":"foo","password":"bar"}],"user":{"username":"foo","password":"bar"}}}}
-```
-
-Ok Response
-
-```json
-{"type":"ok"}
-```
-
-### `send_cv`
-
-Send a scraped CV to RT-CV and triggers the `set_cached_reference` for the reference number of this cv if there where matches on this CV
-
-Example input
-
-```json
-{"type":"send_cv","content":{"reference_number":"abcd","..":".."}}
-```
-
-Ok Response
-
-*The content represents if a match was made*
-
-```json
-{"type":"ok","content":true}
-```
-
-### `get_secret`
-
-Get a user defined secret from the server
-
-Example input
-
-```json
-{"type":"get_secret","content":{"encryption_key":"my-very-secret-encryption-key", "key":"key-of-value"}}
-```
-
-Ok Response
-
-```jsonc
-{"type":"ok","content":{/*Based on the content stored in the secret value*/}}
-```
-
-### `get_users_secret`
-
-Get a secret from the server where the contents is a strictly defined list of users
-
-Example input
-
-```json
-{"type":"get_users_secret","content":{"encryption_key":"my-very-secret-encryption-key", "key":"users"}}
-```
-
-Ok Response
-
-```json
-{"type":"ok","content":[{"username":"foo","password":"foo"},{"username":"bar","password":"bar"}]}
-```
-
-### `get_user_secret`
-
-Get a secret from the server where the contents is a strictly defined user
-
-Example input
-
-```json
-{"type":"get_user_secret","content":{"encryption_key":"my-very-secret-encryption-key", "key":"user"}}
-```
-
-Ok Response
-
-```json
-{"type":"ok","content":{"username":"foo","password":"bar"}}
+```sh
+# rtcv_scraper_client deno run -A denoexample.ts
+credentials set
+testing connections..
+connected to RTCV
+running scraper..
+Check file:///.../denoexample.ts
+[ { username: "username here", password: "password here" } ]
 ```
 
-### `set_cached_reference`
-
-Save a reference number to the cache with a timeout of 3 days
-This cache is used to avoid sending the same CV twice or scraping data that has already been scraped
+## Setup & Run
 
-*Note that "send_cv" also executes this function automatically*
+### *1.* Install the helper
 
-Example input
-
-```json
-{"type":"set_cached_reference","content":"abcd"}
+```sh
+go install github.com/script-development/rtcv_scraper_client@latest
 ```
 
-Ok Response
+### *2.* Obtain a `env.json`
 
-```json
-{"type":"ok"}
+Create a `env.json` file with the following content **(this file can also be obtained from the RTCV dashboard, tough note that you might need to add login_users yourself)**
+```js
+{
+    "primary_server": {
+        "server_location": "http://localhost:4000",
+        "api_key_id": "aa",
+        "api_key": "bbb"
+    },
+    "alternative_servers": [
+        // If you want to send CVs to multiple servers you can add additional servers here
+    ],
+    "login_users": [
+        {"username": "scraping-site-username", "password": "scraping-site-password"}
+    ]
+}
 ```
 
-### `set_short_cached_reference`
+### *3.* Develop / Deploy a scraper using `rtcv_scraper_client`
 
-Same as previouse one except this one has a time out of 12 hours
-
-### `has_cached_reference`
-
-Is there a cache entry for a specific reference number?
-
-Example input
-
-```json
-{"type":"has_cached_reference","content":"abcd"}
-```
+You can now prefix your scraper's run command with `rtcv_scraper_client` and the scraper client program will run a webserver as long as your scraper runs where via you can communicate with RT-CV.
 
-Ok Response
+If you have for a NodeJS project you can run your program like this:
 
-```json
-{"type":"ok","content":true}
+```sh
+rtcv_scraper_client npm run start
 ```
 
-### `ping`
+## Routes available
 
-Send a ping message to the server and the server will respond with a pong
+Notes:
+- The http method can be anything
+- If errors occur the response will be the error message with a 400 or higher status code
 
-Example input
+### `$SCRAPER_ADDRESS/send_cv`
 
-```json
-{"type":"ping"}
-```
+Sends a cv to rtcv and remembers the reference number
 
-Ok Response
+- Body: In JSON the cv send to RT-CV
+- Resp: **true** / **false** if the cv was sent to RT-CV
 
-```json
-{"type":"pong"}
-```
+### `$SCRAPER_ADDRESS/users`
 
-## How to develop / Debug
+Returns the login users from the `env.json`
 
-```sh
-# After a change update the binary in your path using:
-go install
+- Body: None
+- Resp: **true** / **false** the login users from env.json
 
-# Then in your scraper project
-cd ../some-scraper
-go run .
-```
+### `$SCRAPER_ADDRESS/set_cached_reference`
 
-Debug data send by scraper to this program
-```sh
-# Logs the IO to scraper_client_input.log
-export LOG_SCRAPER_CLIENT_INPUT=true
-
-# Logs the debug messages to scraper_client.log
-export ENABLE_SCRAPER_CLIENT_LOG=true
+Manually add another cached reference with the default ttl (3 days)
 
-# Now run your scraper
-go run .
+Note that this is also done by the send_cv route
 
-# Now all messages written to this program are also written to scraper_client_input.log
-# You can follow the input by tailing the file in a new terminal:
-tail -f scraper_client_input.log
+- Body: The reference number
+- Resp: **true**
 
-# All debug messages are now visible in scraper_client.log
-# You can follow the input by tailing the file in a new terminal:
-tail -f scraper_client_input.log
+### `$SCRAPER_ADDRESS/set_short_cached_reference`
 
-# After collecting the input data you can also use rtcv_scraper_client to replay sending the data
-# This is very handy for debugging
-rtcv_scraper_client -replay scraper_client_input.log
+manually add another cached reference with a short ttl (12 hours)
 
-# There are also additional options for the -replay command:
-rtcv_scraper_client \
-    -replay ./scraper_client_input.log \
-    -replaySkipCommands has_cached_reference,set_cached_reference,ping,get_users_secret
-```
+- Body: The reference number
+- Resp: **true**
 
+### `$SCRAPER_ADDRESS/get_cached_reference`
 
-## How to ship?
+Check if a reference number is in the cache
 
-Currently we don't have pre build binaries so you'll need to compile the binary yourself
+- Body: The reference number
+- Resp: **true** / **false**
 
-```Dockerfile
-FROM golang:alpine AS obtain-rtcv-client
-RUN go install github.com/script-development/rtcv_scraper_client@latest
+## Deno client docs
 
-FROM denoland/deno:alpine AS runtime
-COPY --from=obtain-rtcv-client /go/bin/rtcv_scraper_client /bin/rtcv_scraper_client
-```
+[click here](./DENO_README.md)