Skip to content

Commit b037808

Browse files
mayoormayoor
authored
Merge branch 'main' into aqua_apiserver
2 parents 5bc7330 + 747aead commit b037808

File tree

3 files changed

+144
-26
lines changed

3 files changed

+144
-26
lines changed

‎docs/source/user_guide/large_language_model/aqua_client.rst

+131
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,131 @@
1+
AI Quick Actions HTTP Client
2+
****************************
3+
4+
.. versionadded:: 2.13.0
5+
6+
The AI Quick Actions client is a centralized, reusable component for interacting with the OCI Model Deployment service.
7+
8+
**Implementation Highlights:**
9+
10+
- Offers both synchronous (Client) and asynchronous (AsyncClient)
11+
- Integrates with OCI Authentication patterns
12+
13+
Authentication
14+
==============
15+
16+
The AI Quick Actions client supports the same authentication methods as other OCI services, including API Key, session token, instance principal, and resource principal. For additional details, please refer to the `authentication guide <https://accelerated-data-science.readthedocs.io/en/latest/user_guide/cli/authentication.html>`_. Ensure you have the necessary `access policies <https://docs.oracle.com/en-us/iaas/data-science/using/model-dep-policies-auth.htm>`_ to connect to the OCI Data Science Model Deployment endpoint.
17+
18+
Usage
19+
=====
20+
21+
Sync Usage
22+
----------
23+
24+
**Text Completion**
25+
26+
.. code-block:: python3
27+
28+
from ads.aqua import Client
29+
ads.set_auth(auth="security_token", profile="<replace-with-your-profile>")
30+
31+
client = Client(endpoint="https://<MD_OCID>/predict")
32+
response = client.generate(
33+
prompt="Tell me a joke",
34+
payload={"model": "odsc-llm"},
35+
stream=False,
36+
)
37+
print(response)
38+
39+
**Chat Completion**
40+
41+
.. code-block:: python3
42+
43+
from ads.aqua import Client
44+
ads.set_auth(auth="security_token", profile="<replace-with-your-profile>")
45+
46+
client = Client(endpoint="https://<MD_OCID>/predict")
47+
response = client.chat(
48+
messages=[{"role": "user", "content": "Tell me a joke."}],
49+
payload={"model": "odsc-llm"},
50+
stream=False,
51+
)
52+
print(response)
53+
54+
**Streaming**
55+
56+
.. code-block:: python3
57+
58+
from ads.aqua import Client
59+
ads.set_auth(auth="security_token", profile="<replace-with-your-profile>")
60+
61+
client = Client(endpoint="https://<MD_OCID>/predict")
62+
response = client.chat(
63+
messages=[{"role": "user", "content": "Tell me a joke."}],
64+
payload={"model": "odsc-llm"},
65+
stream=True,
66+
)
67+
68+
for chunk in response:
69+
print(chunk)
70+
71+
**Embedding**
72+
73+
.. code-block:: python3
74+
75+
from ads.aqua import Client
76+
ads.set_auth(auth="security_token", profile="<replace-with-your-profile>")
77+
78+
client = Client(endpoint="https://<MD_OCID>/predict")
79+
response = client.embeddings(
80+
input=["one", "two"]
81+
)
82+
print(response)
83+
84+
85+
Async Usage
86+
-----------
87+
88+
The following examples demonstrate how to perform the same operations using the asynchronous client with Python's async/await syntax.
89+
90+
**Text Completion**
91+
92+
.. code-block:: python3
93+
94+
from ads.aqua import AsyncClient
95+
ads.set_auth(auth="security_token", profile="<replace-with-your-profile>")
96+
97+
client = AsyncClient(endpoint="https://<MD_OCID>/predict")
98+
response = await client.generate(
99+
prompt="Tell me a joke",
100+
payload={"model": "odsc-llm"},
101+
stream=False,
102+
)
103+
print(response)
104+
105+
**Streaming**
106+
107+
.. code-block:: python3
108+
109+
from ads.aqua import AsyncClient
110+
ads.set_auth(auth="security_token", profile="<replace-with-your-profile>")
111+
112+
client = AsyncClient(endpoint="https://<MD_OCID>/predict")
113+
async for chunk in await client.generate(
114+
prompt="Tell me a joke",
115+
payload={"model": "odsc-llm"},
116+
stream=True,
117+
):
118+
print(chunk)
119+
120+
**Embedding**
121+
122+
.. code-block:: python3
123+
124+
from ads.aqua import AsyncClient
125+
ads.set_auth(auth="security_token", profile="<replace-with-your-profile>")
126+
127+
client = AsyncClient(endpoint="https://<MD_OCID>/predict")
128+
response = await client.embeddings(
129+
input=["one", "two"]
130+
)
131+
print(response)

‎docs/source/user_guide/large_language_model/index.rst

+5-25
Original file line numberDiff line numberDiff line change
@@ -4,39 +4,19 @@
44
Large Language Model
55
####################
66

7-
Oracle Cloud Infrastructure (OCI) provides fully managed infrastructure to work with Large Language Model (LLM).
7+
Oracle Cloud Infrastructure (OCI) `Data Science <https://www.oracle.com/artificial-intelligence/data-science>`_ is a fully managed, serverless platform that empowers data science teams to build, train, and manage machine learning models on Oracle Cloud Infrastructure.
88

9-
Train and Deploy LLM
10-
********************
11-
You can train LLM at scale with multi-node and multi-GPU using `Data Science Jobs (Jobs) <https://docs.oracle.com/en-us/iaas/data-science/using/jobs-about.htm>`_, and deploy it with `Data Science Model Deployment (Model Deployments) <https://docs.oracle.com/en-us/iaas/data-science/using/model-dep-about.htm>`_. The following blog posts show examples training and deploying Llama2 models:
9+
The platform features `AI Quick Actions <https://docs.oracle.com/en-us/iaas/data-science/using/ai-quick-actions.htm>`_, which enable you to deploy, evaluate, and fine-tune foundation models directly within OCI Data Science. Designed for users eager to quickly harness AI capabilities, these actions provide a streamlined, code-free, and efficient environment for working with foundation models. You can access AI Quick Actions directly from the Data Science Notebook.
1210

13-
* `Multi-GPU multinode fine-tuning Llama2 on OCI Data Science <https://blogs.oracle.com/ai-and-datascience/post/multi-gpu-multi-node-finetuning-llama2-oci>`_
14-
* `Deploy Llama 2 in OCI Data Science <https://blogs.oracle.com/ai-and-datascience/post/llama2-oci-data-science-cloud-platform>`_
15-
* `Quantize and deploy Llama 2 70B on cost-effective NVIDIA A10 Tensor Core GPUs in OCI Data Science <https://blogs.oracle.com/ai-and-datascience/post/quantize-deploy-llama2-70b-costeffective-a10s-oci>`_
16-
17-
18-
Integration with LangChain
19-
**************************
20-
ADS is designed to work with LangChain, enabling developers to incorporate various LangChain components and models deployed on OCI seamlessly into their applications. Additionally, ADS can package LangChain applications and deploy it as a REST API endpoint using OCI Data Science Model Deployment.
21-
22-
* `Bridging cloud and conversational AI: LangChain and OCI Data Science platform <https://blogs.oracle.com/ai-and-datascience/post/cloud-conversational-ai-langchain-oci-data-science>`_
23-
* `Deploy LangChain applications as OCI model deployments <https://blogs.oracle.com/ai-and-datascience/post/deploy-langchain-application-as-model-deployment>`_
24-
25-
26-
.. admonition:: Installation
27-
:class: note
28-
29-
Install ADS and other dependencies for LLM integrations.
30-
31-
.. code-block:: bash
32-
33-
$ python3 -m pip install "oracle-ads[llm]"
11+
Detailed documentation on deploying LLM models in OCI Data Science using AI Quick Actions is available `here <https://github.com/oracle-samples/oci-data-science-ai-samples/blob/main/ai-quick-actions/model-deployment-tips.md>`_ and `here <https://docs.oracle.com/en-us/iaas/data-science/using/ai-quick-actions-model-deploy.htm>`_.
3412

13+
This section provides comprehensive information on integrating OCI with **LangChain, Autogen, LlamaIndex**, and other third-party **LLM frameworks**.
3514

3615

3716
.. toctree::
3817
:maxdepth: 2
3918

19+
aqua_client
4020
training_llm
4121
langchain_models
4222
autogen_integration

‎docs/source/user_guide/large_language_model/training_llm.rst

+8-1
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,14 @@
11
Training Large Language Model
22
*****************************
33

4+
.. admonition:: Note
5+
:class: note
6+
7+
The example provided below is obsolete. Instead, use the `AI Quick Actions <https://docs.oracle.com/en-us/iaas/data-science/using/ai-quick-actions.htm>`_, which can be used to deploy, evaluate, and fine-tune foundation models in OCI Data Science.
8+
9+
10+
11+
412
.. versionadded:: 2.8.8
513

614
Oracle Cloud Infrastructure (OCI) `Data Science Jobs (Jobs) <https://docs.oracle.com/en-us/iaas/data-science/using/jobs-about.htm>`_
@@ -55,4 +63,3 @@ The same training script also support Parameter-Efficient Fine-Tuning (PEFT). Yo
5563
torchrun llama_finetuning.py --enable_fsdp --use_peft --peft_method lora \
5664
--pure_bf16 --batch_size_training 1 \
5765
--model_name meta-llama/Llama-2-7b-hf --output_dir /home/datascience/outputs
58-

0 commit comments

Comments
 (0)