Merge pull request #26 from Azure-Samples/pmishra/few-shot-classifier

bug fixes and move to few-shot classifier

Author: nawanas

End_to_end_Solutions/AOAISearchDemo/README.md

@@ -9,6 +9,10 @@ The repo includes sample data so it's ready to try end to end. In this sample ap
 
 The experience allows users to ask questions about the Surface Devices specifications, troubleshooting help, warranty as well as sales, availability and trend related questions.
 
+There are two pre-recorded voiceovers that shows how enterprises can use this architecture for their different users/audiences. The demo uses two different personas:
+> 1. Emma is marketing lead [demo](./docs/Emma%20Miller_with%20voice.mp4)
+> 2. Dave is regional sales manager [demo](./docs/Dave%20Huang_with%20voice.mp4)
+
 ![RAG Architecture](docs/appcomponents.png)
 
 ## Features
@@ -22,6 +26,7 @@ The experience allows users to ask questions about the Surface Devices specifica
 * Handling failures gracefully and ability to retry failed queries against other data sources
 * Handling token limitations
 * Using fine-tuned model for classification in the orchestrator
+* > *Due to unavailability of fine-tuned models in certain regions, we have updated the code to use gpt-4 based few-shot classifer. Added a new section on how to test this classifier in [promptFlow](./docs/prompt_flow.md)*
 * Using instrumentation for debugging and also for driving certain usage reports from the logs
 
 ![Chat screen](docs/chatscreen.png)
@@ -42,8 +47,11 @@ The experience allows users to ask questions about the Surface Devices specifica
   * **Important**: Ensure you can run `python --version` from console. On Ubuntu, you might need to run `sudo apt install python-is-python3` to link `python` to `python3`.
 * [Node.js](https://nodejs.org/en/download/)
 * [Git](https://git-scm.com/downloads)
-* [Powershell 7+ (pwsh)](https://github.com/powershell/powershell) - For Windows users only.
+* [PowerShell 7+ (pwsh)](https://github.com/powershell/powershell)
   * **Important**: Ensure you can run `pwsh.exe` from a PowerShell command. If this fails, you likely need to upgrade PowerShell.
+* [The AzureAD PowerShell module version 2.0.2.180 or above](https://learn.microsoft.com/en-us/powershell/module/azuread/?view=azureadps-2.0)
+* [ODBC Driver for SQL Server v18](https://learn.microsoft.com/en-us/sql/connect/odbc/download-odbc-driver-for-sql-server)
+
 
 >NOTE: Your Azure Account must have `Microsoft.Authorization/roleAssignments/write` permissions, such as [User Access Administrator](https://learn.microsoft.com/azure/role-based-access-control/built-in-roles#user-access-administrator) or [Owner](https://learn.microsoft.com/azure/role-based-access-control/built-in-roles#owner).  
 
@@ -66,6 +74,7 @@ Due to high demand, Azure OpenAI resources can be difficult to spin up on the fl
    - `AZURE_OPENAI_CLASSIFIER_MODEL {Name of Azure OpenAI model to be used to do dialog classification}`.
    - `AZURE_OPENAI_CLASSIFIER_DEPLOYMENT {Name of existing Azure OpenAI model deployment to be used for dialog classification}`.
     * Ensure the model you specify for `AZURE_OPENAI_DEPLOYMENT` and `AZURE_OPENAI_MODEL` is a Chat GPT model, since the demo utilizes the ChatCompletions API when requesting completions from this model.
+    * Ensure the model you specify for `AZURE_OPENAI_CLASSIFIER_DEPLOYMENT` and `AZURE_OPENAI_CLASSIFIER_MODEL` is compatible with the Completions API, since the demo utilizes the Completions API when requesting completions from this model.
     * You can also use existing Search and Storage Accounts.  See `./infra/main.parameters.json` for list of environment variables to pass to `azd env set` to configure those existing resources.
 2. Go to `app/backend/bot_config.yaml`. This file contains the model configuration definitions for the Azure OpenAI models that will be used. It defines request parameters like temperature, max_tokens, etc., as well as the the deployment name (`engine`) and model name (`model_name`) of the deployed models to use from your Azure OpenAI resource. These are broken down by task, so the request parameters and model for doing question classification on a user utterance can differ from those used to turn natural language into SQL for example. You will want the deployment name (`engine`) for the `approach_classifier` to match the one set for `AZURE_OPENAI_CLASSIFIER_DEPLOYMENT`. For the rest, you wil want the deployment name (`engine`) and model name (`model_name`) to match `AZURE_OPENAI_DEPLOYMENT` and `AZURE_OPENAI_MODEL` respectively. For the models which specify a `total_max_tokens`, you will want to set this value to the maximum number of tokens your deployed GPT model allows for a completions request. This will allow the backend service to know when prompts need to be trimmed to avoid a token limit error.
     * Note that the config for `approach_classifier` doesn't contain a system prompt, this is because the demo expects this model to be a fine-tuned GPT model rather than one trained using few-shot training. You will need to provide a fine-tuned model trained on some sample data for the dialog classification to work well. For more information on how to do this, checkout the [fine-tuning section](README.md#fine-tuning).
@@ -203,6 +212,10 @@ You can find helpful resources on how to fine-tune a model on the Azure OpenAI w
 
 ***Answer***: Yes, as part of the development of the application, we included some basic logging to capture what is happening around a user conversation. Application Insights was used as the logging backend. The [log reports](docs/log_reports.md) document has some sample KQL queries and reports based on these logs
 
+***Question***: Are there suggestions on how to develop and test prompts
+
+***Answer***: Yes, we have included documentation on how you could leverage Prompt Flow for developing and testing prompts. An example of developing and performing bulk test on few-shot classifier is included in the [prompt flow](docs/prompt_flow.md) document.
+
 ### Troubleshooting
 
 If you see this error while running `azd deploy`: `read /tmp/azd1992237260/backend_env/lib64: is a directory`, then delete the `./app/backend/backend_env folder` and re-run the `azd deploy` command.  This issue is being tracked here: <https://github.com/Azure/azure-dev/issues/1237>

End_to_end_Solutions/AOAISearchDemo/app/backend/.env.template

@@ -17,6 +17,7 @@ AZURE-OPENAI-CLASSIFIER-API-KEY=""
 # Azure Cognitive Search
 AZURE-SEARCH-SERVICE=""
 AZURE-SEARCH-INDEX=""
+AZURE-SEARCH-KEY=""
 KB-FIELDS-CONTENT="content"
 KB-FIELDS-CATEGORY="category"
 KB-FIELDS-SOURCEPAGE="sourcepage"

End_to_end_Solutions/AOAISearchDemo/app/backend/app.py

@@ -1,23 +1,26 @@
 import datetime
 import json
 import mimetypes
-import openai
 import time
+
+import openai
 import yaml
 from azure.core.credentials import AzureKeyCredential
 from azure.identity import DefaultAzureCredential
 from azure.search.documents import SearchClient
 from azure.storage.blob import BlobServiceClient
 from backend.approaches.approach_classifier import ApproachClassifier
-from backend.approaches.chatunstructured import ChatUnstructuredApproach
 from backend.approaches.chatstructured import ChatStructuredApproach
-from common.contracts.chat_session import ChatSession, ParticipantType, DialogClassification
+from backend.approaches.chatunstructured import ChatUnstructuredApproach
 from backend.config import DefaultConfig
 from backend.contracts.chat_response import Answer, ApproachType, ChatResponse
-from backend.contracts.error import OutOfScopeException, UnauthorizedDBAccessException
+from backend.contracts.error import (OutOfScopeException,
+                                     UnauthorizedDBAccessException)
 from backend.data_client.data_client import DataClient
 from backend.utilities.access_management import AccessManager
-from flask import Flask, request, jsonify
+from common.contracts.chat_session import (ChatSession, DialogClassification,
+                                           ParticipantType)
+from flask import Flask, jsonify, request
 
 # Use the current user identity to authenticate with Azure OpenAI, Cognitive Search and Blob Storage (no secrets needed, 
 # just use 'az login' locally, and managed identity when deployed on Azure). If you need to use keys, use separate AzureKeyCredential instances with the 
@@ -44,9 +47,9 @@
 logger = DefaultConfig.logger
 
 chat_approaches = {
-    ApproachType.unstructured.value: ChatUnstructuredApproach(search_client, DefaultConfig.KB_FIELDS_SOURCEPAGE,
+    ApproachType.unstructured.name: ChatUnstructuredApproach(search_client, DefaultConfig.KB_FIELDS_SOURCEPAGE,
                                         DefaultConfig.KB_FIELDS_CONTENT, logger, search_threshold_percentage = DefaultConfig.SEARCH_THRESHOLD_PERCENTAGE),
-    ApproachType.structured.value: ChatStructuredApproach(DefaultConfig.SQL_CONNECTION_STRING, logger)                        
+    ApproachType.structured.name: ChatStructuredApproach(DefaultConfig.SQL_CONNECTION_STRING, logger)                        
 }
 
 
@@ -129,11 +132,11 @@ def chat():
         if classification_override:
             approach_type = ApproachType(classification_override)
         else:
-            openai.api_base = f"https://{DefaultConfig.AZURE_OPENAI_CLASSIFIER_SERVICE}.openai.azure.com"
-            openai.api_key = DefaultConfig.AZURE_OPENAI_CLASSIFIER_API_KEY
+            openai.api_base = f"https://{DefaultConfig.AZURE_OPENAI_GPT4_SERVICE}.openai.azure.com"
+            openai.api_key = DefaultConfig.AZURE_OPENAI_GPT4_API_KEY
             approach_type = approach_classifier.run(history, bot_config)
 
-        logger.info(f"question_type: {approach_type.value}", extra=properties)
+        logger.info(f"question_type: {approach_type.name}", extra=properties)
 
         if approach_type == ApproachType.chit_chat:
             chit_chat_canned_response =  "I'm sorry, but the question you've asked is outside my area of expertise. I'd be happy to help with any questions related to Microsoft Surface PCs and Laptops. Please feel free to ask about those, and I'll do my best to assist you!"
@@ -162,7 +165,7 @@ def chat():
         simplified_history = [{"participant_type": dialog.participant_type.value, "utterance": dialog.utterance} for dialog in filtered_chat_session.conversation]
         simplified_history.append({"participant_type": ParticipantType.user.value, "utterance": user_message})
 
-        impl = chat_approaches.get(approach_type.value)
+        impl = chat_approaches.get(approach_type.name)
 
         if not impl:
             return jsonify({"error": "unknown approach"}), 400

End_to_end_Solutions/AOAISearchDemo/app/backend/approaches/approach.py

@@ -1,6 +1,8 @@
-from backend.contracts.chat_response import ChatResponse
 from typing import List
 
+from backend.contracts.chat_response import ChatResponse
+
+
 class Approach:
     def run(self, history: List[dict], overrides: dict) -> ChatResponse:
         raise NotImplementedError

End_to_end_Solutions/AOAISearchDemo/app/backend/approaches/approach_classifier.py

@@ -1,22 +1,59 @@
+from textwrap import dedent
+from typing import List
+
 import openai
 from backend.approaches.approach import Approach
-from backend.config import DefaultConfig
 from backend.contracts.chat_response import ApproachType
+from common.contracts.chat_session import DialogClassification
 from common.logging.log_helper import CustomLogger
-from typing import List
+
 
 class ApproachClassifier(Approach):
     def __init__(self, logger: CustomLogger):
         self.logger = logger
-    
+
     def run(self, history: List[str], bot_config) -> ApproachType:
-        response = openai.Completion.create(
-            prompt=history[-1]['utterance'] + ' ->',
-            **bot_config["approach_classifier"]["openai_settings"]
+
+        message_list = [
+            {
+                "role": "system",
+                "content": dedent(bot_config["approach_classifier"]["system_prompt"])
+            }
+        ]
+
+        if bot_config["approach_classifier"]["history"]["include"]:
+            for message in history[-((bot_config["approach_classifier"]["history"]["length"]*2) + 1):]:
+                if message["participant_type"] == "user":
+                    message_list.append(
+                        {"role": "user", "content": message["utterance"]})
+                else:
+                    classification = ''
+                    if message['question_type'] == DialogClassification.structured_query.name:
+                        classification = ApproachType.structured.value
+                    elif message['question_type'] == DialogClassification.unstructured_query.name:
+                        classification = ApproachType.unstructured.value
+                    elif message['question_type'] == DialogClassification.chit_chat.name:
+                        classification = ApproachType.chit_chat.value
+                    else:
+                        classification = ApproachType.unstructured.value
+                    message_list.append(
+                        {"role": "assistant", "content": classification})
+        else:
+            message_list.append(
+                {"role": "user", "content": history[-1]["utterance"]})
+        try:
+            response = openai.ChatCompletion.create(
+                messages=message_list,
+                **bot_config["approach_classifier"]["openai_settings"]
             )
+        except openai.error.InvalidRequestError as e:
+            self.logger.error(
+                f"OpenAI API Error: {e.message}", exc_info=True)
+            raise e
 
-        q :str = response['choices'][0]['text'].strip()
-        self.log_aoai_response_details(f'Classification Prompt:{history[-1]["utterance"]}', f'Response: {q}', response)
+        q: str = response['choices'][0]['message']['content']
+        self.log_aoai_response_details(
+            f'Classification Prompt:{history[-1]["utterance"]}', f'Response: {q}', response)
 
         if q == "1":
             return ApproachType.structured
@@ -28,14 +65,15 @@ def run(self, history: List[str], bot_config) -> ApproachType:
             # Continuation: Return last question type from history if it exists
             if len(history) > 1:
                 last_question_type = history[-2]['question_type']
-                if last_question_type == "structured_query":
+                if last_question_type == DialogClassification.structured_query.value:
                     return ApproachType.structured
-                elif last_question_type == "unstructured_query":
+                elif last_question_type == DialogClassification.unstructured_query.value:
                     return ApproachType.unstructured
-                elif last_question_type == "chit_chat":
+                elif last_question_type == DialogClassification.chit_chat.value:
                     return ApproachType.chit_chat
                 else:
-                    raise Exception(f"Unknown question type: {last_question_type}")
+                    raise Exception(
+                        f"Unknown question type: {last_question_type}")
             else:
                 return ApproachType.unstructured
         else:
@@ -49,4 +87,5 @@ def log_aoai_response_details(self, prompt, result, aoai_response):
             "aoai_response[MS]": aoai_response.response_ms
         }
         addl_properties = self.logger.get_updated_properties(addl_dimensions)
-        self.logger.info(f"prompt: {prompt}, response: {result}", extra=addl_properties)
+        self.logger.info(
+            f"prompt: {prompt}, response: {result}", extra=addl_properties)