Added API documentation

Signed-off-by: Rohan Chitale <[email protected]>

Author: rchitale7

DEVELOPER_GUIDE.md

@@ -41,6 +41,11 @@ Core Dependencies:
 pip install -r remote_vector_index_builder/core/requirements.txt
 ```
 
+API Dependencies:
+```
+pip install -r remote_vector_index_builder/app/requirements.txt
+```
+
 Test Dependencies:
 ```
 pip install -r test_remote_vector_index_builder/requirements.txt
@@ -76,6 +81,21 @@ pytest test_remote_vector_index_builder/
 ```
 
 ## Building Docker Images
+
+There are 3 images vended by this repository: 
+- `faiss-base`
+  - Uses NVIDIA CUDA image as a base image
+  - Adds the dependencies needed to install `faiss`, and builds `faiss` from source
+- `core`
+  - Uses `faiss-base` as a base image
+  - Adds the code for the core index build functionalities: building an index and remote store I/O
+- `api`
+  - Uses `core` as a base image
+  - Adds the code for a `fastAPI` server with `_build` and `_status` APIs
+    - The `_build` API triggers an index build workflow and returns a job id to the caller 
+    - The `_status` API gets the status of the workflow, given a job id
+    - The index build workflow is executed in the background, using the `core` library functions
+
 The Github CIs automatically publish snapshot images to Dockerhub at [opensearchstaging/remote-vector-index-builder](https://hub.docker.com/r/opensearchstaging/remote-vector-index-builder).
 
 The following are the commands to build the images locally:
@@ -85,25 +105,38 @@ The [Faiss repository](https://github.com/facebookresearch/faiss/) is added as a
 ```
 git submodule update --init
 ```
-The Faiss base image can only be created on an NVIDIA GPU powered machine with CUDA Toolkit installed.
+The `faiss-base` image can only be created on an NVIDIA GPU powered machine with CUDA Toolkit installed.
 
 Please see the section [Provisioning an instance for development](#provisioning-an-instance-for-development) to provision an instance for development.
 
-Run the below command to create the Faiss base image:
+Run the below command to create the `faiss-base` image:
 ```
 docker build  -f ./base_image/build_scripts/Dockerfile . -t opensearchstaging/remote-vector-index-builder:faiss-base-latest
 ```
 
 ### Core Image
-The path [`/remote-vector-index-builder/core`](/remote_vector_index_builder/core/) contains the code for core index build functionalities:
-1. Building an Index
-2. Object Store I/O
+The path [`/remote-vector-index-builder/core`](/remote_vector_index_builder/core/) contains the code for the `core` image.
+Run the below command to create the `core` image:
 
-Build an image with the above core functionalities:
 ```
 docker build  -f ./remote_vector_index_builder/core/Dockerfile . -t opensearchstaging/remote-vector-index-builder:core-latest
 ```
 
+The image can be built on any type of machine with `docker` installed
+
+### API Image
+The path [`/remote-vector-index-builder/app`](/remote_vector_index_builder/app/) contains the code for the `api` image. 
+Run the below command to create `api` image. 
+
+```
+docker build  -f ./remote_vector_index_builder/app/Dockerfile . -t opensearchstaging/remote-vector-index-builder:api-latest
+```
+
+The image can be built on any type of machine with `docker` installed
+
+Note that any docker image tag can be used when building locally. It may be more useful to tag the image with a personal dockerhub name, 
+in order to push the image to your remote dockerhub account. 
+
 ## Provisioning an instance for development
 
 A NVIDIA GPU powered machine with CUDA Toolkit installed is required to build a Faiss base image and to run the Docker images to build an index.
@@ -112,6 +145,49 @@ Typically an [EC2 G5](https://aws.amazon.com/ec2/instance-types/g5/) 2xlarge ins
 
 [Setup an EC2 Instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EC2_GetStarted.html)
 
+## Running Docker Images
+
+### Run the API Image as standalone container
+The API image implements the [Remote Vector Service API Contract](/API.md). It provides out-of-the-box APIs
+that directly integrate with the OpenSearch k-NN Remote Vector Service Client. The default image
+configuration allows it to be used to run integration tests for the OpenSearch k-NN Remote Vector component.
+This configuration maintains the job state in an in-memory dictionary with a TTL, and uses a fixed size
+thread pool to execute index build workflows. 
+
+
+The dictionary size and TTL is controllable via the Docker container settings. For example, you can set the TTL
+to `None`, to ensure requests never get deleted. You are also free to implement a separate, custom API image, 
+as long as it conforms to the Remote Vector Service API Contract and provides endpoints for the Remote Vector Service Client. 
+This custom API image can still use the `core` image libraries to execute the index build workflow.
+
+Follow the steps below to use run the API image locally. Note that s3 is currently the only supported Remote Store repository
+1. [Provision an instance for development](#provisioning-an-instance-for-development)
+2. Create a s3 bucket, upload vector and doc id binaries to the bucket
+3. Ensure the instance has AWS credentials to connect with the s3 bucket
+    - Use any option from 3-11: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials
+4. Clone the repository to the instance : `git clone https://github.com/opensearch-project/remote-vector-index-builder.git`
+5. Change into the top-level directory: `cd remote-vector-index-builder`
+6. Build the docker image: `docker build  -f ./remote_vector_index_builder/app/Dockerfile . -t opensearchstaging/remote-vector-index-builder:api-latest`
+    - Note that any image tag can be used, not just `opensearchstaging/remote-vector-index-builder:api-latest`
+7. Run the docker image: `docker run --gpus all -p 80:1025 opensearchstaging/remote-vector-index-builder:api-latest`
+8. In a separate terminal, issue a build request:
+    ```
+    curl -XPOST "http://0.0.0.0:80/_build" \
+    -H 'Content-Type: application/json' \
+    -d '
+        {
+            "repository_type": "s3",
+            "container_name": "<your_s3_bucket>>",
+            "vector_path": "<vector_path_in_s3_bucket>",
+            "doc_id_path": "<doc_id_path_in_s3_bucket>",
+            "dimension": "<vector dimension>",
+            "doc_count": "<number of vectors>"
+        }
+    '
+    ```
+   This will return a job id, if the build request was successfully submitted.
+9. Check the status of your build request: `curl -XGET "http://0.0.0.0:80/_status/<job_id>"`
+
 ## Memory Profiling
 
 Simple memory profiling can be done during development to get memory usage statistics during the Index Build process.
@@ -157,3 +233,4 @@ def my_func():
     del b
     return a
 ```
+