diff --git a/app/docs/01-functional-view/01-information-analysis-and-processing/06-information-extraction-from-documents.mdx b/app/docs/01-functional-view/01-information-analysis-and-processing/06-information-extraction-from-documents.mdx index 882f0fbf..d2a469f3 100644 --- a/app/docs/01-functional-view/01-information-analysis-and-processing/06-information-extraction-from-documents.mdx +++ b/app/docs/01-functional-view/01-information-analysis-and-processing/06-information-extraction-from-documents.mdx @@ -1,8 +1,6 @@ --- title: "Information Extraction from Documents" description: "" -hide_title: true -hide_table_of_contents: true --- import useBaseUrl from "@docusaurus/useBaseUrl"; @@ -32,9 +30,10 @@ Example: Customer name, address, total amount are extracted from an invoice e.g.
- ## Implementation + ## Technical Documentation For very simple PDF extractions, where the data is very well structured, and small, then simple parsing and Large Language Model prompting may be sufficient. For tasks that require contextual understanding, question answering, and handling large documents Vector RAG is a powerful and recommended approach for extracting and utilizing data from PDFs with Large Language Models. +
### Related Best Practices
diff --git a/app/docs/02-technical-view/02-narrow-ai/03-classification.mdx b/app/docs/02-technical-view/02-narrow-ai/03-classification.mdx index fa5b868f..0e966bd2 100644 --- a/app/docs/02-technical-view/02-narrow-ai/03-classification.mdx +++ b/app/docs/02-technical-view/02-narrow-ai/03-classification.mdx @@ -24,15 +24,6 @@ import PageContributors from "@site/src/components/PageContributors";

Classification

-::::danger[IMPORTANT - How to make this page visible] - -- To make this page visible on the sidebar, you need to remove the sidebar_class_name: hidden from the top of the file. -- To enable the links to this page (for example in the index.mdx file), you need to remove the disabled=true from the IconLinkButton component on that specific page. - -Don't forget to remove this message. - -:::: -
@@ -79,73 +70,299 @@ Don't forget to remove this message.
- - - Download Source Code - - - ## 1 Overview ### Description -... +Classification is a fundamental machine learning technique aimed at organizing input data into distinct classes. During the classification process, the model undergoes training using the 'train subset' function, followed by evaluation using the 'test subset' function. A notable distinction between classification and regression tasks lies in their output variables. While classification deals with discrete target variables, regression tasks involve continuous output variables. ### Expected Outcome -... +Essentially, the expected outcome is to recognize patterns in the training data, and use these patterns to classify new data into one of the pre-defined classes. + +#### Scenarios + +Classification can be utilized in various scenarios, such as spam detection, image recognition, fraud detection, medical diagnosis, among others. For example, the figure below illustrates how a classification model determines whether an incoming message falls under the category of SPAM or Inbox (non-SPAM). + +![](@site/static/narrow-ai/classification/images/classification-model.png) + +The core principles of classification involve supervised learning, where models are trained using labeled data to recognize patterns and make predictions on new, unseen data. ### Benefits -... +- **Automated and Accurate Predictions**: allows for the automation of tasks that would otherwise require manual analysis. +- **Improved Decision-Making**: by understanding the patterns and relationships within data, classification models can help businesses, and organizations make better decisions. For example, in marketing, classification can help identify customer segments, personalize marketing campaigns, and improve customer experience. +- **Enhanced Efficiency and Productivity**: can streamline processes and reduce the need for manual intervention, saving time and resources; the increased efficiency can translate into higher productivity and cost savings. For instance, in healthcare, classification models can help diagnose diseases faster and more accurately, leading to improved patient outcomes. +- **Improved Customer Experience**: classification can help businesses personalize interactions with customers by understanding their preferences and needs; this can lead to increased customer satisfaction and loyalty. +- **Scalability and Efficiency**: many classification algorithms can handle large datasets efficiently, making them suitable for applications involving large datasets; they can also be implemented in real-time for applications where speed is critical. +- **Enhanced Security**: can identify unusual patterns or anomalies that may indicate potential threats, such as fraud or cyberattacks; this allows for proactive security measures and reduced risk. +- **Learning Relationships**: it allows for learning relationships between input features and predefined classes. + For example, a model might learn that the presence of certain visual features, e.g., given color or shape, is strongly correlated with the class label ‘whale’ in an image classification task. +- **Provides Prediction Intervals**: it can generate prediction intervals to provide a range of possible outcomes with a certain level of confidence. -### Key Concepts +## 2 Pre-requisites -... +### Supported SAP HANA versions and editions +SAP HANA’s Machine Learning capabilities, includes both the Predictive Analysis Library (PAL) and the Automated Predictive Library (APL), are supported on various SAP HANA versions and editions. Key supported platforms include: + +- **SAP HANA Cloud**: Fully supported and recommended platform due to easier management of components like PAL and APL. More information on [HANA Cloud and Setup process](/docs/technology/sap-hana-cloud#setup). + - [**Prerequisites:**](https://support.sap.com/en/alm/sap-cloud-alm/operations/expert-portal/setup-managed-services/setup-hana-cloud.html?anchorId=section_2065216453_c#section_copy_copy) + - You need to have access to SAP BTP Cockpit ([Setup Guide](/docs/technology/sap-business-technology-platform#setup-guide)). + - Your user in BTP Cockpit needs to be a member with the space role "Space Developer" for the space in which you want to create the SAP HANA Cloud database instance was created +- **SAP HANA express edition**: Used for development and testing. [Installation instructions for SAP HANA](https://developers.sap.com/group.hxe-install-clients.html) and [Installation instructions for Client API](https://help.sap.com/doc/1d0ebfe5e8dd44d09606814d83308d4b/2.0.06/en-US/Installation.html). +- **SAP Datasphere (formerly Data Warehouse Cloud)**: Supports PAL and API functions, provided underlying [SAP HANA Cloud Script Server](https://help.sap.com/docs/SAP_DATASPHERE/9f804b8efa8043539289f42f372c4862/287194276a7d4d778ec98fdde5f61335.html) is enabled for the tenant -## 2 Pre-requisites +Note, PAL and APL are also available from ([SAP HANA Platform 2.0 SPS 04](https://help.sap.com/doc/9c87e50e69c744f785f41ec5568b47d8/2.0.04/en-US/SAP_HANA_Machine_Learning_Overview_Guide_en.pdf)) as well as SAP HANA express edition + +### Required SAP HANA Components +**SAP HANA Cloud**: PAL and APL are pre-installed within the SAP HANA Cloud environment. During configuration of the database instance, the necessary services need to be enabled and user permissions to be granted + +**Predictive Analysis Library (PAL)**: it provides a wide range of common and specialized native in-database functions for predictive analysis and machine learning in scenarios like classification, regression, time series forecasting, outlier detection, text processing and analysis as well as text embedding + +**Automated Predictive Library (APL)**: it provides native in-database functions for creating predictive models in a simplified manner due to the automation of many of the steps. + +### Required User Authorizations and Roles +Executing PAL and APL functions requires specific database privileges. These are granted via predefined roles: + +- For [PAL](https://help.sap.com/docs/hana-cloud-database/sap-hana-cloud-sap-hana-database-predictive-analysis-library/security-security-253f2b5?locale=en-US): You must be assigned one of the following roles to execute PAL procedures. The roles for the PAL library are automatically created when the Application Function Library (AFL) is enabled. The role names are: + - AFL__SYS_AFL_AFLPAL_EXECUTE + - AFL__SYS_AFL_AFLPAL_EXECUTE_WITH_GRANT_OPTION +- For [APL](https://help.sap.com/docs/apl/419fd47c26b345239fdbb5e476a6bc54/654f4d920c5846b6b56903c7b06b0785.html): + - *SAP HANA Cloud*: The database role sap.pa.apl.base.roles::APL_EXECUTE is required. This role grants privileges to execute APL stored procedures, access related schemas, and use the APL cache. It should be granted by a database administrator to the APL user. +- **General Privileges**: Beyond specific PAL/APL roles, users will need standard SQL privileges to access the data tables used as input for the algorithms (e.g., SELECT on relevant schemas/tables) and potentially privileges to create temporary tables or views depending on the workflow (e.g., CREATE TEMPORARY TABLE). + +### Summary + +| Prerequisite Category | Item | Notes | +|--------------------------------|-------------------------------------------------------------|----------------------------------------------------------------------------------------------------| +| **SAP HANA Cloud Environment** | SAP HANA Cloud, SAP HANA Cloud trial and free-tier | See trial / free-tier limitations. | +| **SAP HANA Cloud Configuration** | Enabled services | Script Server and NLP services | +| **User Authorizations** | PAL Execution Role Granted | User needs AFL__SYS_AFL_AFLPAL_EXECUTE (Cloud/On-Prem) role. | +| | APL Execution Role Granted | User needs sap.pa.apl.base.roles::APL_EXECUTE (Cloud/On-Prem Procedure) or direct AFL roles (On-Prem Direct). | +| | Data Access Privileges (SELECT) | User needs SELECT access on input data tables/views. | +| | Object Creation Privileges (Optional) | May need CREATE TEMPORARY TABLE etc., depending on workflow. | + +### High-Level Architecture + +![](@site/static/narrow-ai/classification/images/arch.png) + +#### Calling PAL and APL Functions +- PAL functions can be called through the SQL interface or using the Python or R Machine Learning clients (hana-ml). +- APL functions can be called through the SQL interface or using the Python Machine Learning client (hana-ml). +- The Natural Language Processing functions for Text Analysis, Named Entity Recognition or Part of Speech tagging can be called via PAL functions. In addition, the Text Embedding function can be called via PAL functions as well as the SQL function [VECTOR_EMBEDDING()](https://help.sap.com/docs/hana-cloud-database/sap-hana-cloud-sap-hana-database-vector-engine-guide/creating-text-embeddings-in-sap-hana-cloud). +- Both PAL and APL functions can be called out of the box in the HANA Cloud Database Core Container, or as part of the HANA Cloud Elastic Compute Nodes. +- SAP BTP Applications can embed use of PAL or APL functions as part of standard SQLScript design-time artifacts, e.g. multi-target or full-stack CAP applications. Furthermore, usage of PAL and APL functions may also be utilized via applications leveraging Python runtimes and scripts using the Python Machine Learning client (hana-ml). -... ## 3 Key Choices and Guidelines -... +Classification is a fundamental technique in machine learning used to categorize data points into predefined labels based on their features. It plays a crucial role in various real-world applications, such as predicting whether an employee will churn or remain in a company. The core principles of classification involve supervised learning, where models are trained using labeled data to recognize patterns and make predictions on new, unseen data. -## 4 Implementation +There are different approaches to classification, each employing distinct methodologies. Some of the most commonly used algorithms include decision trees, gradient boosting, and neural networks, each suited to different types of data and classification challenges. Classification itself can be categorized into three main types. -... +- **Binary** classification involves distinguishing between two possible outcomes, such as "yes" or "no" and "spam" or "not spam." +- **Multi-class** classification extends binary classification by categorizing data into multiple distinct groups, such as classifying different product categories. +- **Multi-label** classification allows a single data point to belong to multiple categories, such as tagging a document with multiple relevant topics. Or multi-target classification, where a single model shall be used to predict multiple target labels using a single model. -
- - +Handling of text data and moreover text embeddings as features has emerged as crucial capability for many classification scenarios, as only by the use of text embedding models the semantic understanding of input text could be captured by classification algorithms. The following [link](https://github.com/SAP-samples/hana-ml-samples/blob/main/Python-API/pal/notebooks/24QRC04_2.23-unlocking_text_data.ipynb) shows an example. - - Download Source Code - +### Algorithm Selection +With the great variety of algorithms, the classification scenario type as well as data patterns like number of rows (observations), number of features / type of features and cardinality of features influence the selection of classification models and their use cases. -... +| Dataset Size | Algorithm | Description | +|---------------|------------|--------------| +| **Small to Medium size data sets** | Logistic Regression | Used in simple binary classification problems, such as predicting "yes" or "no" labels. | +| | k – Nearest Neighbors (k – NN) | | +| | Decision Trees | Used in cases for easy model interpretability and model explainability. | - +| Dataset Size | Algorithm | Description | +|---------------|------------|--------------| +| **Medium to Large data sets** | Random Forest | Used for complex classification problems where accuracy is highly relevant. | +| | Support Vector Machines (SVM) | Used for classification scenarios with clear margin of separation, e.g., image classification. | +| | XGBoost | Used in high performance classification problems, e.g., fraud detection. | +| | Neural Networks | Also known as Deep Learning, it is used for complex and high dimensional data scenarios, e.g., speech recognition. | +| | LightGBM | Used for large scale and high performance applications, e.g., web search ranking. | +| | Multinomial Naive Bayes | Used for discrete data, specially for text classification tasks where word counts are used as features for sentiment analysis. | - - - Download Source Code - +| Dataset Size | Algorithm | Description | +|---------------|------------|--------------| +| **Very Large data sets** | Deep Learning, i.e., Convolutional Neural Networks (CNN) architecture for images, Recurrent Neural Networks (RNN) for sequence data. | Used for very large and complex datasets, e.g., natural language processing. | +| | Distributed Machine Learning Frameworks such as Spark MLLib, TensorFlow. | Used for scalable machine learning scenarios across distributed computing environments, e.g., large scale recommendation systems. | -... +### Algorithm experimentation and selection +AutoML frameworks like [SAP HANA Cloud PAL’s AutoML](https://help.sap.com/docs/hana-cloud-database/sap-hana-cloud-sap-hana-database-predictive-analysis-library/automl-automl) implementation greatly help users in probing multiple algorithms and parameter variations for a given scenario for selecting the best possible algorithm and model. In the following [link](https://github.com/SAP-samples/hana-ml-samples/blob/main/Python-API/usecase-examples/sapcommunity-automl-examples/Example%20-%20Using%20PAL%20AutoML%20for%20Classification.ipynb) you can view an example on how to apply it. - +### Data Preparation and Model Evaluation +The success of any machine learning model is fundamentally dependent on two core processes: rigorous data preparation and comprehensive model evaluation. These steps ensure the data is clean and relevant and that the model's performance is accurately measured against the right metrics. +For a detailed guide on the methodologies, key metrics, and best practices for preparing your data and evaluating model performance across different ML tasks, please refer to our unified [ML Data Preparation and Model Evaluation Best Practices](/docs/technical-view/narrow-ai/data-preparation-and-model-evaluation) page. + +### Leverage In-Database ML functions from SAP HANA whenever possible +- **Prioritize HANA ML:** If your data resides in SAP HANA (Cloud/on-premise) or SAP Datasphere, using hana-ml is generally the most efficient approach due to minimized data movement and optimized performance. - +### Anomaly and Outlier Detection +Identifying anomalies or outliers is a critical aspect of building robust machine learning models. These unusual data points can represent data quality issues, unique real-world events, or errors in data collection that might otherwise skew model training and evaluation. +For a comprehensive guide on the methodologies and best practices for discovering anomalies in your data, please refer to our unified [Anomaly Detection Best Practices](/docs/technical-view/narrow-ai/anomaly-detection) page. - - Download Source Code - +## 4 Implementation + +When developing regression models with SAP tools and libraries, the choice of your application's runtime dictates the relevant information. Different approaches can be used to embed the classification scenario artifacts. + +### Programming Model Selection Guidelines +When developing linear regression scenarios with SAP HANA’s Predictive Analysis Library (PAL), consider the following guidelines: -... +- **Recommended Approach (Data Science Workflows):** Utilize the Python hana-ml library for a streamlined, intuitive experience aligned with standard data science practices, including convenient data manipulation and integration with machine learning workflows. This approach yields best productivity during ML model experimentation until the final model of choice and parameterization has been determined. +- **Alternative Approaches:** + - SQLScript database procedures and table functions are the typical HANA design-time artifacts for embedding and invoking PAL and APL procedures into BTP CAP or XSA full-stack applications. Artifact generations from the hana-ml machine learning client in Python is supported. + +The final choice should align with your team’s expertise and project requirements. + + +
+ + + +### Recommendation +Use the **hana-ml** library for interacting with SAP HANA PAL and APL algorithms via the Python runtime environment. + +### SDKs +- [**hana-ml**](https://help.sap.com/doc/1d0ebfe5e8dd44d09606814d83308d4b/latest/en-US/hana_ml.html) (Recommended for SAP HANA ML interaction) + +### Tutorials and Learning Journeys +- **SAP Community Blogs:** + - [Build your Machine Learning Scenario for your SAP HANA Cloud application from Python](https://www.youtube.com/watch?v=CX38-95uBtc) + - [Exploring ML Explainability in SAP HANA PAL – Classification and Regression](https://community.sap.com/t5/technology-blogs-by-sap/exploring-ml-explainability-in-sap-hana-pal-classification-and-regression/ba-p/13681514) +- **SAP Developer Tutorials:** + - [Predictive AI with SAP AI Core](https://developers.sap.com/tutorial-navigator.html?search=classification) (General AI Core usage) +- **SAP Education:** + - [Developing AI Models with the Python Machine Learning Client for SAP HANA](https://learning.sap.com/learning-journeys/developing-ai-models-with-the-python-machine-learning-clients-for-sap-hana) + - [Developing Classification Models with the Python Machine Learning Client for SAP HANA](https://learning.sap.com/learning-journeys/developing-classification-models-with-the-python-machine-learning-client-for-sap-hana) + +### Reference Code +**Recommended** +- [SAP BTP AI Best Practices - Sample Code](https://github.com/SAP-samples/sap-btp-ai-best-practices/tree/main/best-practices/narrow-ai/classification) + +**Relevant Code** +*HybridGradientBoostingTree (HGBT)* + +```python +from hana_ml.algorithms.pal.unified_classification import UnifiedClassification +​ +# --- Training the HybridGradientBoostingTree (HGBT) Model --- +hgbc = UnifiedClassification(func='HybridGradientBoostingTree', + n_estimators = 101, split_threshold=0.1, + learning_rate=0.1, max_depth=6, + split_method='histogram', max_bin_num=256, feature_grouping=True, + tolerant_iter_num=5, + resampling_method='cv', fold_num=5, ref_metric=['auc'], + evaluation_metric = 'error_rate') +​ +print("\nTraining the HGBT model...") +hgbc.fit(data=df_train.drop('EMPLOYEE_ID'), + label='FLIGHT_RISK', + partition_method='stratified', stratified_column='FLIGHT_RISK', training_percent=0.8, + ntiles=20, + build_report=True) +​ +​ +# --- Model Evaluation --- +print("\nComputing the AUC-ROC (Area under the ROC curve) for the trained HGBT model...") +hgbc.get_performance_metrics()['AUC'] +print("Model Evaluation completed.") +``` + + + + +SAP HANA's Predictive Analysis Library (PAL) algorithms can be directly invoked through SQLScript. This approach is beneficial for: + +- **Performance Optimization:** Minimizes data movement by executing directly within the database. +- **Integration:** Allows embedding ML logic within database procedures, views, or calculation views for seamless integration with existing SAP HANA applications. +- **Governance:** Provides a standardized approach within existing database governance frameworks. + +When implementing with SQLScript, you'll typically: +1. Create parameter tables with algorithm-specific settings. +2. Prepare input data in the required format. +3. Call the PAL procedure. +4. Process the resulting output tables. + +### SDKs +- [SAP HANA Predictive Analysis Library (PAL)](https://help.sap.com/docs/SAP_HANA_PLATFORM/2cfbc5cf2bc14f028cfbe2a2bba60a50/c9eeed704f3f4ec39441434db8a874ad.html?version=2.0.05&locale=en-US&q=PCA) + +### Tutorials and Learning Journeys +- The following links point to the relevant SAP HANA PAL documentation for the SQLScript procedures: +- **Hybrid Gradient Boosting Tree Classifier:** [Hybrid Gradient Boosting Tree](https://help.sap.com/docs/SAP_HANA_PLATFORM/319d36de4fd64ac3afbf91b1fb3ce8de/hybrid-gradient-boosting-tree-ca5106c.html) + +### Reference Code +*Hybrid Gradient Boosting Tree* + +```sql +set schema DM_PAL; +​ +drop table PAL_HGBT_DATA_TAB; +create column table PAL_HGBT_DATA_TAB ( + "ATT1" double, + "ATT2" double, + "ATT3" double, + "ATT4" double, + "LABEL" nvarchar(50) +); +insert into PAL_HGBT_DATA_TAB values (1.0, 10.0, 100, 1.0, 'A'); +insert into PAL_HGBT_DATA_TAB values (1.1, 10.1, 100, 1.0, 'A'); +insert into PAL_HGBT_DATA_TAB values (1.2, 10.2, 100, 1.0, 'A'); +insert into PAL_HGBT_DATA_TAB values (1.3, 10.4, 100, 1.0, 'A'); +insert into PAL_HGBT_DATA_TAB values (1.2, 10.3, 100, 1.0, 'A'); +insert into PAL_HGBT_DATA_TAB values (4.0, 40.0, 400, 4.0, 'B'); +insert into PAL_HGBT_DATA_TAB values (4.1, 40.1, 400, 4.0, 'B'); +insert into PAL_HGBT_DATA_TAB values (4.2, 40.2, 400, 4.0, 'B'); +insert into PAL_HGBT_DATA_TAB values (4.3, 40.4, 400, 4.0, 'B'); +insert into PAL_HGBT_DATA_TAB values (4.2, 40.3, 400, 4.0, 'A'); +insert into PAL_HGBT_DATA_TAB values (9.0, 90.0, 900, 2.0, 'A'); +insert into PAL_HGBT_DATA_TAB values (9.1, 90.1, 900, 1.0, 'B'); +insert into PAL_HGBT_DATA_TAB values (9.2, 90.2, 900, 2.0, 'B'); +insert into PAL_HGBT_DATA_TAB values (9.3, 90.4, 900, 1.0, 'B'); +insert into PAL_HGBT_DATA_TAB values (9.2, 90.3, 900, 1.0, 'B'); +​ +drop table PAL_PARAMETER_TAB; +create column table PAL_PARAMETER_TAB ( + "PARAM_NAME" nvarchar(100), + "INT_VALUE" integer, + "DOUBLE_VALUE" double, + "STRING_VALUE" nvarchar(100) +); +insert into PAL_PARAMETER_TAB values ('RESAMPLING_METHOD', null, null, 'cv'); +insert into PAL_PARAMETER_TAB values ('FOLD_NUM', 5, null, null); +insert into PAL_PARAMETER_TAB values ('PARAM_SEARCH_STRATEGY', null, null, 'grid'); +insert into PAL_PARAMETER_TAB values ('EVALUATION_METRIC', null, null, 'ERROR_RATE'); +insert into PAL_PARAMETER_TAB values ('REF_METRIC', null, null, 'AUC'); +insert into PAL_PARAMETER_TAB values ('SEED', 1, null, null); +​ +insert into PAL_PARAMETER_TAB values ('MAX_DEPTH_VALUES', null, null, '{3, 5, 6}'); +insert into PAL_PARAMETER_TAB values ('ITER_NUM_RANGE', null, null, '[4, 2, 10]'); +insert into PAL_PARAMETER_TAB values ('ETA_RANGE', null, null, '[0.1, 0.2, 1.0]'); +insert into PAL_PARAMETER_TAB values ('GAMMA_RANGE', null, null, '[0.1, 0.2, 1.0]'); +​ +drop table PAL_HGBT_MODEL_TAB; -- for predict followed +create column table PAL_HGBT_MODEL_TAB ( + "ROW_INDEX" integer, + "TREE_INDEX" integer, + "MODEL_CONTENT" nclob +); +​ +do begin + LT_DATA_TAB = select * from PAL_HGBT_DATA_TAB; + LT_PARAM_TAB = select * from PAL_PARAMETER_TAB; + call _SYS_AFL.PAL_HGBT(:LT_DATA_TAB, :LT_PARAM_TAB, LT_MODEL_TAB, LT_IMP_TAB, LT_CONFUSE_TAB, LT_STAT_TAB, LT_CV_TAB); + select * from :LT_MODEL_TAB; + select * from :LT_IMP_TAB; + select * from :LT_CONFUSE_TAB; + select * from :LT_STAT_TAB; + select * from :LT_CV_TAB; + insert into PAL_HGBT_MODEL_TAB select * from :LT_MODEL_TAB; +end; +``` @@ -159,7 +376,8 @@ Don't forget to remove this message. ## Related Best Practices
    - + +
@@ -170,20 +388,10 @@ Don't forget to remove this message. ## Related AI Capabilities -

- Content Creation -

- -
    - -
- -

- Information Analysis & Processing -

-
    - + + +
@@ -194,7 +402,11 @@ Don't forget to remove this message. ## Contributors - +
diff --git a/app/docs/02-technical-view/02-narrow-ai/04-clustering.mdx b/app/docs/02-technical-view/02-narrow-ai/04-clustering.mdx index a2a8b5b6..24410807 100644 --- a/app/docs/02-technical-view/02-narrow-ai/04-clustering.mdx +++ b/app/docs/02-technical-view/02-narrow-ai/04-clustering.mdx @@ -24,15 +24,6 @@ import PageContributors from "@site/src/components/PageContributors";

Clustering

-::::danger[IMPORTANT - How to make this page visible] - -- To make this page visible on the sidebar, you need to remove the sidebar_class_name: hidden from the top of the file. -- To enable the links to this page (for example in the index.mdx file), you need to remove the disabled=true from the IconLinkButton component on that specific page. - -Don't forget to remove this message. - -:::: -
@@ -79,73 +70,455 @@ Don't forget to remove this message.
- - - Download Source Code - - - ## 1 Overview ### Description -... +Clustering aims to partition a dataset into subsets (clusters), such that data points within the same cluster exhibit high intra-cluster similarity, while points in different clusters exhibit low inter-cluster similarity. ### Expected Outcome -... +In essence, the clustering model outputs a partition of dataset into groups that share similar characteristics ### Benefits -... +- **Unsupervised Pattern Discovery**: Clustering identifies hidden structures or natural groupings in unlabeled data without prior knowledge or supervision. +- **Data Simplification and Summarization**: It reduces the complexity of large datasets by grouping similar data points, making it easier to analyze and interpret. +- **Anomaly Detection**: Clustering aids in anomaly detection by identifying normal groupings within a dataset, making it easier to spot data points that deviate significantly from the norm. These outliers often indicate unusual or rare events such as fraud, system faults, or other anomalies that require attention. -### Key Concepts +## 2 Pre-requisites -... +### Supported SAP HANA versions and editions + +SAP HANA’s Machine Learning capabilities, includes both the Predictive Analysis Library (PAL) and the Automated Predictive Library (APL), are supported on various SAP HANA versions and editions. Key supported platforms include: + +- **SAP HANA Cloud**: Fully supported and recommended platform due to easier management of components like PAL and APL. More information on [HANA Cloud and Setup process](/docs/technology/sap-hana-cloud#setup). + - [Prerequisites](https://support.sap.com/en/alm/sap-cloud-alm/operations/expert-portal/setup-managed-services/setup-hana-cloud.html?anchorId=section_2065216453_c#section_copy_copy): + - You need to have access to SAP BTP Cockpit ([Setup Guide](/docs/technology/sap-business-technology-platform#setup-guide)). + - Your user in BTP Cockpit needs to be a member with the space role "Space Developer" for the space in which you want to create the SAP HANA Cloud database instance was created +- **SAP HANA express edition**: Used for development and testing. [Installation instructions for SAP HANA](https://developers.sap.com/group.hxe-install-clients.html) and [Installation instructions for Client API](https://help.sap.com/doc/1d0ebfe5e8dd44d09606814d83308d4b/2.0.06/en-US/Installation.html). +- **SAP Datasphere (formerly Data Warehouse Cloud)**: Supports PAL and API functions, provided underlying [SAP HANA Cloud Script Server](https://help.sap.com/docs/SAP_DATASPHERE/9f804b8efa8043539289f42f372c4862/287194276a7d4d778ec98fdde5f61335.html) is enabled for the tenant + +Note, PAL and APL are also available from [SAP HANA Platform 2.0 SPS 04](https://help.sap.com/doc/9c87e50e69c744f785f41ec5568b47d8/2.0.04/en-US/SAP_HANA_Machine_Learning_Overview_Guide_en.pdf) as well as SAP HANA express edition -## 2 Pre-requisites -... +### Required SAP HANA Components + +- **SAP HANA Cloud**: PAL and APL are pre-installed within the SAP HANA Cloud environment. During configuration of the database instance, the necessary services need to be enabled and user permissions to be granted. +- **Predictive Analysis Library (PAL)**: provides a wide range of common and specialized native in-database functions for predictive analysis and machine learning in scenarios like classification, regression, time series forecasting, outlier detection, text processing and analysis as well as text embedding. +- **Automated Predictive Library (APL)**: provides native in-database functions for creating predictive models in a simplified manner due to the automation of many of the steps. While APL supports creating clustering models, it is designed to be an automated framework and does not expose full control over the machine learning model internal working. + +### Required User Authorizations and Roles + +Executing PAL and APL functions requires specific database privileges. These are granted via predefined roles: + +- **For [PAL](https://help.sap.com/docs/hana-cloud-database/sap-hana-cloud-sap-hana-database-predictive-analysis-library/security-security-253f2b5?locale=en-US)**: You must be assigned one of the following roles to execute PAL procedures. The roles for the PAL library are automatically created when the Application Function Library (AFL) is enabled. The role names are: + - `AFL__SYS_AFL_AFLPAL_EXECUTE` + - `AFL__SYS_AFL_AFLPAL_EXECUTE_WITH_GRANT_OPTION` +- **For [APL](https://help.sap.com/docs/apl/419fd47c26b345239fdbb5e476a6bc54/654f4d920c5846b6b56903c7b06b0785.html)**: + - *SAP HANA Cloud*: The database role `sap.pa.apl.base.roles::APL_EXECUTE` is required. This role grants privileges to execute APL stored procedures, access related schemas, and use the APL cache. It should be granted by a database administrator to the APL user. +- **General Privileges**: Beyond specific PAL/APL roles, users will need standard SQL privileges to access the data tables used as input for the algorithms (e.g., SELECT on relevant schemas/tables) and potentially privileges to create temporary tables or views depending on the workflow (e.g., CREATE TEMPORARY TABLE). + +### Summary + +| **Prerequisite Category** | **Item** | **Notes** | +|----------------------------|----------|-----------| +| **SAP HANA Cloud Environment** | SAP HANA Cloud, SAP HANA Cloud trial and free-tier | See trial / free-tier limitations. | +| **SAP HANA Cloud Configuration** | Enabled services | Script Server and NLP services | +| **User Authorizations** | PAL Execution Role Granted | User needs `AFL__SYS_AFL_AFLPAL_EXECUTE` (Cloud/On-Prem) role. | +| | APL Execution Role Granted | User needs `sap.pa.apl.base.roles::APL_EXECUTE` (Cloud/On-Prem Procedure) or direct AFL roles (On-Prem Direct). | +| | Data Access Privileges (SELECT) | User needs `SELECT` access on input data tables/views. | +| | Object Creation Privileges (Optional) | May need `CREATE TEMPORARY TABLE` etc., depending on workflow. | + +### High-level Reference Architecture +![Clustering - High-level Reference Architecture](@site/static/narrow-ai/clustering/images/arch.png) + +#### Calling PAL and APL Functions + +- PAL functions can be called through the SQL interface or using the Python or R Machine Learning clients (hana-ml). +- APL functions can be called through the SQL interface or using the Python Machine Learning client (hana-ml). +- The Natural Language Processing functions for Text Analysis, Named Entity Recognition or Part of Speech tagging can be called via PAL functions. In addition, the Text Embedding function can be called via PAL functions as well as the SQL function [VECTOR_EMBEDDING()](https://help.sap.com/docs/hana-cloud-database/sap-hana-cloud-sap-hana-database-vector-engine-guide/creating-text-embeddings-in-sap-hana-cloud). +- Both PAL and APL functions can be called out of the box in the HANA Cloud Database Core Container, or as part of the HANA Cloud Elastic Compute Nodes. +- SAP BTP Applications can embed use of PAL or APL functions as part of standard SQLScript design-time artifacts, e.g. multi-target or full-stack CAP applications. Furthermore, usage of PAL and APL functions may also be utilized via applications leveraging Python runtimes and scripts using the Python Machine Learning client (hana-ml) ## 3 Key Choices and Guidelines -... +Implementing clustering effectively involves several key decisions and adherence to best practices: -## 4 Implementation +### Leverage In-Database ML functions from SAP HANA whenever possible -... +- **Prioritize HANA ML**: If your data resides in SAP HANA or SAP Datasphere, using `hana-ml` is generally the most efficient approach due to minimized data movement and optimized performance. -
- - +### Algorithm Selection - - Download Source Code - +SAP HANA Predictive Analytics Library, offers a wide range of Clustering algorithms. Choosing the right Clustering model in SAP HANA PAL depends on several key factors related to both the characteristics of your data and the business problem you're trying to address. Some of the commonly used algorithms are as follows: -... +- **K-Means** algorithm partitions n observations or records into k clusters in which each observation belongs to the cluster with the nearest center. Clustering works to group records together according to an algorithm or mathematical formula that attempts to find centroids, or centers, around which similar records gravitate. Given an initial set of k means m1, ..., mk, the algorithm proceeds by alternating between two steps: + - Assignment step: assigns each observation to the cluster with the closest mean. + - Update step: calculates the new means to be the center of the observations in the cluster. + The algorithm repeats until the assignments no longer change. - +- **DBSCAN** (Density-Based Spatial Clustering of Applications with Noise) is a density-based data clustering algorithm. It finds a number of clusters starting from the estimated density distribution of corresponding nodes. DBSCAN requires two parameters: scan radius (eps) and the minimum number of points required to form a cluster (minPts). The algorithm starts with an arbitrary starting point that has not been visited. This point's eps-neighborhood is retrieved, and if the number of points it contains is equal to or greater than minPts, a cluster is started. Otherwise, the point is labeled as noise. These two parameters are very important and are usually determined by user. + PAL provides a method to automatically determine these two parameters. You can choose to specify the parameters by yourself or let the system determine them for you. - +- **Agglomerate Hierarchical Clustering** is a widely used clustering method which can find natural groups within a set of data. The idea is to group the data into a hierarchy or a binary tree of the subgroups. A hierarchical clustering can be either agglomerate or divisive, depending on the method of hierarchical decomposition. + The implementation in PAL follows the agglomerate approach, which merges the clusters with a bottom-up strategy. Initially, each data point is considered as an own cluster. The algorithm iteratively merges two clusters based on the dissimilarity measure in a greedy manner and forms a larger cluster. Therefore, the input data must be numeric and a measure of dissimilarity between sets of data is required, which is achieved by using the following two parameters: + - An appropriate metric (a measure of distance between pairs of groups) + - A linkage criterion which specifies the distances between groups. - - Download Source Code - + An advantage of hierarchical clustering is that it does not require the number of clusters to be specified as the input. And the hierarchical structure can also be used for data summarization and visualization. +- **Spectral clustering** is an algorithm evolved from graph theory, and has been widely used in clustering. Its main idea is to treat all data as points in space, which can be connected by edges. The edge weight between two points farther away is low, while the edge weight between two points closer is high. Cutting the graph composed of all data points to make the edge weight sum between different subgraphs after cutting as low as possible, while make the edge weight sum within the subgraph as high as possible to achieve the purpose of clustering. + It performs a low-dimension embedding of the affinity matrix between samples, followed by k-means clustering of the components of the eigenvectors in the low dimensional space. -... +### Data Preparation and Model Evaluation +The success of any machine learning model is fundamentally dependent on two core processes: rigorous data preparation and comprehensive model evaluation. These steps ensure the data is clean and relevant and that the model's performance is accurately measured against the right metrics. +For a detailed guide on the methodologies, key metrics, and best practices for preparing your data and evaluating model performance across different ML tasks, please refer to our unified [Data Preparation and Model Evaluation](https://sap.sharepoint.com/sites/210313/SitePages/Data%20prep%20and%20model%20evaluation.aspx) page. - +## 4 Implementation + +When developing clustering models with SAP tools and libraries, the choice of your application's runtime dictates the relevant information. Different approaches can be used to embed the time series scenario artifacts. + + +### Programming Model Selection Guidelines +When developing clustering scenarios with SAP HANA’s Predictive Analysis Library (PAL), consider the following guidelines: + +- **Recommended Approach (Data Science Workflows)**: Utilize the Python hana-ml library for a streamlined, intuitive experience aligned with standard data science practices, including convenient data manipulation and integration with machine learning workflows. This approach yields best productivity during ML model experimentation until the final model of choice and parameterization has been determined. +- **Alternative Approaches**: + - **SQLScript database procedures and table functions** are the typical **HANA design-time artifacts** for embedding and invoking PAL and APL procedures into BTP CAP or XSA full-stack applications. Artifact generations from the hana-ml machine learning client in Python is supported. + +The final choice should align with your team’s expertise and project requirements. - +
+ + - - Download Source Code - +### Recommendation + +Use the hana_ml library for interacting with SAP HANA PAL and APL algorithms directly from Python. + +### SDKs + +- [hana_ml](https://help.sap.com/doc/1d0ebfe5e8dd44d09606814d83308d4b/latest/en-US/hana_ml.html) + +### Tutorials and Learning Journeys + +SAP Community Blogs: + +- [Developing AI Models with the Python Machine Learning Client for SAP HANA](https://sap.plateau.com/learning/user/deeplink.do?linkId=ITEM_DETAILS&componentID=LSC_127518&componentTypeID=COURSE&revisionDate=1725890244000&nativelogin=y#/8A265CC6841B3310190018FA82E8C7D9) +- [Outlier Detection by Clustering using Python Machine Learning Client for SAP HANA](https://community.sap.com/t5/technology-blog-posts-by-sap/outlier-detection-by-clustering-using-python-machine-learning-client-for/ba-p/13469349) +- [Model Storage with Python Machine Learning Client for SAP HANA](https://community.sap.com/t5/technology-blog-posts-by-sap/model-storage-with-python-machine-learning-client-for-sap-hana/ba-p/13483099) + +SAP Developer Tutorials: + +- [Predictive AI with SAP AI Core](https://developers.sap.com/group.ai-core-get-started-basics.html) (General AI Core usage) + +### Reference Code + +**Recommended** + +- [SAP BTP AI Best Practices - Sample Code](https://github.com/SAP-samples/sap-btp-ai-best-practices/tree/main/best-practices/narrow-ai/clustering) + +**Relevant Code** + +SAP HANA APL unsupervised clustering algorithm + + +```python +from hana_ml.algorithms.apl.clustering import AutoUnsupervisedClustering +from hana_ml.dataframe import ConnectionContext, DataFrame +Connecting to SAP HANA +​ +CONN = ConnectionContext('HDB_HOST', HDB_PORT, 'HDB_USER', 'HDB_PASS') +​ +# Creates Hana DataFrame +hana_df = DataFrame(CONN, 'select * from APL_SAMPLES.CENSUS') +Creating and fitting the model +​ +model = AutoUnsupervisedClustering(CONN, nb_clusters=5) +model.fit(data=hana_df, key='id') +Debriefing +​ +model.get_metrics() +OrderedDict([('SimplifiedSilhouette', 0.3448029020802121), ('RSS', 4675.706587754118),... +model.get_metrics_by_cluster() +{'Frequency': {1: 0.23053242076908276, + 2: 0.27434649954646656, + 3: 0.09628652318517908, + 4: 0.29919463456199663, + 5: 0.09963992193727494}, + 'IntraInertia': {1: 0.6734978174937322, + 2: 0.7202839995396123, + 3: 0.5516800856975772, + 4: 0.6969632183111357, + 5: 0.5809322138167139}, + 'RSS': {1: 5648.626195319932, + 2: 7189.15459940487, + 3: 1932.5353401986129, + 4: 7586.444631316713, + 5: 2105.879275085588}, + 'SimplifiedSilhouette': {1: 0.1383827622819234, + 2: 0.14716862328457128, + 3: 0.18753797605134545, + 4: 0.13679980173383793, + 5: 0.15481377834381388}, + 'KL': {1: OrderedDict([('relationship', 0.4951910610641741), + ('marital-status', 0.2776259711735807), + ('hours-per-week', 0.20990189265572687), + ('education-num', 0.1996353893520096), + ('education', 0.19963538935200956), + ... +​ +# Predicting which cluster a data point belongs to +​ +applyout_df = model.predict(hana_df) +applyout_df.collect() # returns the output as a pandas DataFrame + id CLOSEST_CLUSTER_1 DISTANCE_TO_CLOSEST_CENTROID_1 +0 30 3 0.640378 +1 63 4 0.611050 +2 66 3 0.640378 +3 110 4 0.611050 +4 335 1 0.851054 +​ + +# Determining the 2 closest clusters + +model.set_params(extra_applyout_settings={'mode':'closest_distances', 'nb_distances': 2}) +applyout_df = model.predict(hana_df) +applyout_df.collect() # returns the output as a pandas DataFrame + id CLOSEST_CLUSTER_1 ... CLOSEST_CLUSTER_2 DISTANCE_TO_CLOSEST_CENTROID_2 +0 30 3 ... 4 0.730330 +1 63 4 ... 1 0.851054 +2 66 3 ... 4 0.730330 +3 110 4 ... 1 0.851054 +4 335 1 ... 4 0.906003 +​ + +# Retrieving the distances to all clusters +​ +model.set_params(extra_applyout_settings={'mode': 'all_distances'}) +applyout_df = model.predict(hana_df) +applyout_df.collect() # returns the output as a pandas DataFrame + id DISTANCE_TO_CENTROID_1 ... DISTANCE_TO_CENTROID_5 +0 30 3 ... 1.160697 +1 63 4 ... 1.160697 +2 66 3 ... 1.160697 +​ + +# Saving the model in the schema named 'MODEL_STORAGE'. Please model_storage class for further features of model storage. +​ +model_storage = ModelStorage(connection_context=CONN, schema='MODEL_STORAGE') +model.name = 'My model name' +model_storage.save_model(model=model) +Reloading the model for further use +​ +model2 = AutoUnsupervisedClustering(conn_context=CONN) +model2.load_model(schema_name='MySchema', table_name='MyTable') +applyout2 = model2.predict(hana_df) +applyout2.head(3).collect() + id CLOSEST_CLUSTER_1 DISTANCE_TO_CLOSEST_CENTROID_1 +0 30 3 0.640378 +1 63 4 0.611050 +2 66 3 0.640378 +​ + +# Exporting the SQL apply code +​ +model = AutoUnsupervisedClustering(CONN, nb_clusters=5, + calculate_sql_expressions='enabled') +model.fit(data=hana_df, key='id') +sql = model.export_apply_code(code_type='HANA', + key='id', + schema_name='APL_SAMPLES', + table_name='CENSUS') +``` + + -... + + +### Recommendation + +SAP HANA's Predictive Analysis Library (PAL) algorithms can be directly invoked through SQLScript. This approach is beneficial for: + +- **Performance Optimization:** Minimizes data movement by executing directly within the database. +- **Integration:** Allows embedding ML logic within database procedures, views, or calculation views for seamless integration with existing SAP HANA applications. +- **Governance:** Provides a standardized approach within existing database governance frameworks. + +When implementing with SQLScript, you'll typically: + +1. Create parameter tables with algorithm-specific settings. +2. Prepare input data in the required format. +3. Call the PAL procedure. +4. Process the resulting output tables. + +### SDKs + +- [SAP HANA Predictive Analysis Library (PAL)](https://help.sap.com/docs/SAP_HANA_PLATFORM/2cfbc5cf2bc14f028cfbe2a2bba60a50/c9eeed704f3f4ec39441434db8a874ad.html?version=2.0.05&locale=en-US&q=PCA) + +### Tutorials and Learning Journeys + +SAP Community Blogs: + +- [SAP HANA PAL Quick Start](https://community.sap.com/t5/technology-blog-posts-by-sap/sap-hana-pal-quick-start/ba-p/13084367) + +The following links point to the relevant SAP HANA PAL documentation for the SQLScript procedures: + + +### Reference Code + +KMeans + +```sql +SET SCHEMA DM_PAL; +​ +DROP TABLE PAL_KMEANS_DATA_TBL; +CREATE COLUMN TABLE PAL_KMEANS_DATA_TBL( + "ID" INTEGER, + "V000" DOUBLE, + "V001" VARCHAR(2), + "V002" DOUBLE +); +INSERT INTO PAL_KMEANS_DATA_TBL VALUES (0, 0.5, 'A', 0.5); +INSERT INTO PAL_KMEANS_DATA_TBL VALUES (1, 1.5, 'A', 0.5); +INSERT INTO PAL_KMEANS_DATA_TBL VALUES (2, 1.5, 'A', 1.5); +INSERT INTO PAL_KMEANS_DATA_TBL VALUES (3, 0.5, 'A', 1.5); +INSERT INTO PAL_KMEANS_DATA_TBL VALUES (4, 1.1, 'B', 1.2); +INSERT INTO PAL_KMEANS_DATA_TBL VALUES (5, 0.5, 'B', 15.5); +INSERT INTO PAL_KMEANS_DATA_TBL VALUES (6, 1.5, 'B', 15.5); +INSERT INTO PAL_KMEANS_DATA_TBL VALUES (7, 1.5, 'B', 16.5); +INSERT INTO PAL_KMEANS_DATA_TBL VALUES (8, 0.5, 'B', 16.5); +INSERT INTO PAL_KMEANS_DATA_TBL VALUES (9, 1.2, 'C', 16.1); +INSERT INTO PAL_KMEANS_DATA_TBL VALUES (10, 15.5, 'C', 15.5); +INSERT INTO PAL_KMEANS_DATA_TBL VALUES (11, 16.5, 'C', 15.5); +INSERT INTO PAL_KMEANS_DATA_TBL VALUES (12, 16.5, 'C', 16.5); +INSERT INTO PAL_KMEANS_DATA_TBL VALUES (13, 15.5, 'C', 16.5); +INSERT INTO PAL_KMEANS_DATA_TBL VALUES (14, 15.6, 'D', 16.2); +INSERT INTO PAL_KMEANS_DATA_TBL VALUES (15, 15.5, 'D', 0.5); +INSERT INTO PAL_KMEANS_DATA_TBL VALUES (16, 16.5, 'D', 0.5); +INSERT INTO PAL_KMEANS_DATA_TBL VALUES (17, 16.5, 'D', 1.5); +INSERT INTO PAL_KMEANS_DATA_TBL VALUES (18, 15.5, 'D', 1.5); +INSERT INTO PAL_KMEANS_DATA_TBL VALUES (19, 15.7, 'A', 1.6); +​ +DROP TABLE #PAL_PARAMETER_TBL; +CREATE LOCAL TEMPORARY COLUMN TABLE #PAL_PARAMETER_TBL( + "PARAM_NAME" NVARCHAR(256), + "INT_VALUE" INTEGER, + "DOUBLE_VALUE" DOUBLE, + "STRING_VALUE" NVARCHAR(1000) +); +INSERT INTO #PAL_PARAMETER_TBL VALUES ('THREAD_RATIO', NULL, 0.2, NULL); +INSERT INTO #PAL_PARAMETER_TBL VALUES ('GROUP_NUMBER', 4, NULL, NULL); +INSERT INTO #PAL_PARAMETER_TBL VALUES ('INIT_TYPE', 1, NULL, NULL); +INSERT INTO #PAL_PARAMETER_TBL VALUES ('DISTANCE_LEVEL',2, NULL, NULL); +INSERT INTO #PAL_PARAMETER_TBL VALUES ('MAX_ITERATION', 100, NULL, NULL); +INSERT INTO #PAL_PARAMETER_TBL VALUES ('EXIT_THRESHOLD', NULL, 1.0E-6, NULL); +INSERT INTO #PAL_PARAMETER_TBL VALUES ('CATEGORY_WEIGHTS', NULL, 0.5, NULL); +​ +CALL _SYS_AFL.PAL_KMEANS(PAL_KMEANS_DATA_TBL, "#PAL_PARAMETER_TBL", ?, ?, ?, ?, ?); +​ +``` + +DBSCAN (Density-Based Spatial Clustering of Applications with Noise) + +```sql +SET SCHEMA DM_PAL; +​ +DROP TABLE PAL_DBSCAN_DATA_TBL; +CREATE COLUMN TABLE PAL_DBSCAN_DATA_TBL ( + "ID" INTEGER, + "V1" DOUBLE, + "V2" DOUBLE, + "V3" VARCHAR(10) +); +INSERT INTO PAL_DBSCAN_DATA_TBL VALUES(1, 0.10, 0.10, 'B'); +INSERT INTO PAL_DBSCAN_DATA_TBL VALUES(2, 0.11, 0.10, 'A'); +INSERT INTO PAL_DBSCAN_DATA_TBL VALUES(3, 0.10, 0.11, 'C'); +INSERT INTO PAL_DBSCAN_DATA_TBL VALUES(4, 0.11, 0.11, 'B'); +INSERT INTO PAL_DBSCAN_DATA_TBL VALUES(5, 0.12, 0.11, 'A'); +INSERT INTO PAL_DBSCAN_DATA_TBL VALUES(6, 0.11, 0.12, 'E'); +INSERT INTO PAL_DBSCAN_DATA_TBL VALUES(7, 0.12, 0.12, 'A'); +INSERT INTO PAL_DBSCAN_DATA_TBL VALUES(8, 0.12, 0.13, 'C'); +INSERT INTO PAL_DBSCAN_DATA_TBL VALUES(9, 0.13, 0.12, 'D'); +INSERT INTO PAL_DBSCAN_DATA_TBL VALUES(10, 0.13, 0.13, 'D'); +INSERT INTO PAL_DBSCAN_DATA_TBL VALUES(11, 0.13, 0.14, 'A'); +INSERT INTO PAL_DBSCAN_DATA_TBL VALUES(12, 0.14, 0.13, 'C'); +INSERT INTO PAL_DBSCAN_DATA_TBL VALUES(13, 10.10, 10.10, 'A'); +INSERT INTO PAL_DBSCAN_DATA_TBL VALUES(14, 10.11, 10.10, 'F'); +INSERT INTO PAL_DBSCAN_DATA_TBL VALUES(15, 10.10, 10.11, 'E'); +INSERT INTO PAL_DBSCAN_DATA_TBL VALUES(16, 10.11, 10.11, 'E'); +INSERT INTO PAL_DBSCAN_DATA_TBL VALUES(17, 10.11, 10.12, 'A'); +INSERT INTO PAL_DBSCAN_DATA_TBL VALUES(18, 10.12, 10.11, 'B'); +INSERT INTO PAL_DBSCAN_DATA_TBL VALUES(19, 10.12, 10.12, 'B'); +INSERT INTO PAL_DBSCAN_DATA_TBL VALUES(20, 10.12, 10.13, 'D'); +INSERT INTO PAL_DBSCAN_DATA_TBL VALUES(21, 10.13, 10.12, 'F'); +INSERT INTO PAL_DBSCAN_DATA_TBL VALUES(22, 10.13, 10.13, 'A'); +INSERT INTO PAL_DBSCAN_DATA_TBL VALUES(23, 10.13, 10.14, 'A'); +INSERT INTO PAL_DBSCAN_DATA_TBL VALUES(24, 10.14, 10.13, 'D'); +INSERT INTO PAL_DBSCAN_DATA_TBL VALUES(25, 4.10, 4.10, 'A'); +INSERT INTO PAL_DBSCAN_DATA_TBL VALUES(26, 7.11, 7.10, 'C'); +INSERT INTO PAL_DBSCAN_DATA_TBL VALUES(27, -3.10, -3.11, 'C'); +INSERT INTO PAL_DBSCAN_DATA_TBL VALUES(28, 16.11, 16.11, 'A'); +INSERT INTO PAL_DBSCAN_DATA_TBL VALUES(29, 20.11, 20.12, 'C'); +INSERT INTO PAL_DBSCAN_DATA_TBL VALUES(30, 15.12, 15.11, 'A'); +​ +DROP TABLE #PAL_PARAMETER_TBL; +CREATE LOCAL TEMPORARY COLUMN TABLE #PAL_PARAMETER_TBL ( + "PARAM_NAME" VARCHAR(100), + "INT_VALUE" INTEGER, + "DOUBLE_VALUE" DOUBLE, + "STRIN_VALUE" VARCHAR(100) +); +​ +INSERT INTO #PAL_PARAMETER_TBL VALUES ('THREAD_RATIO', NULL, 0.2, NULL); +INSERT INTO #PAL_PARAMETER_TBL VALUES ('AUTO_PARAM', NULL, NULL, 'true'); +INSERT INTO #PAL_PARAMETER_TBL VALUES ('DISTANCE_METHOD', 1, NULL, NULL); +​ +CALL _SYS_AFL.PAL_DBSCAN (PAL_DBSCAN_DATA_TBL, "#PAL_PARAMETER_TBL", ?, ?, ?, ?); +``` + +Agglomerate Hierarchical Clustering + +```sql +SET SCHEMA DM_PAL; +​ +DROP TABLE PAL_HIERARCHICAL_CLUSTERING_DATA_TBL; +CREATE COLUMN TABLE PAL_HIERARCHICAL_CLUSTERING_DATA_TBL ("POINT" NVARCHAR(100), "X1" DOUBLE, "X2" DOUBLE , "X3" INTEGER); +INSERT INTO PAL_HIERARCHICAL_CLUSTERING_DATA_TBL VALUES ('0', 0.5, 0.5, 1); +INSERT INTO PAL_HIERARCHICAL_CLUSTERING_DATA_TBL VALUES ('1', 1.5, 0.5, 2); +INSERT INTO PAL_HIERARCHICAL_CLUSTERING_DATA_TBL VALUES ('2', 1.5, 1.5, 2); +INSERT INTO PAL_HIERARCHICAL_CLUSTERING_DATA_TBL VALUES ('3', 0.5, 1.5, 2); +INSERT INTO PAL_HIERARCHICAL_CLUSTERING_DATA_TBL VALUES ('4', 1.1, 1.2, 2); +INSERT INTO PAL_HIERARCHICAL_CLUSTERING_DATA_TBL VALUES ('5', 0.5, 15.5, 2); +INSERT INTO PAL_HIERARCHICAL_CLUSTERING_DATA_TBL VALUES ('6', 1.5, 15.5, 3); +INSERT INTO PAL_HIERARCHICAL_CLUSTERING_DATA_TBL VALUES ('7', 1.5, 16.5, 3); +INSERT INTO PAL_HIERARCHICAL_CLUSTERING_DATA_TBL VALUES ('8', 0.5, 16.5, 3); +INSERT INTO PAL_HIERARCHICAL_CLUSTERING_DATA_TBL VALUES ('9', 1.2, 16.1, 3); +INSERT INTO PAL_HIERARCHICAL_CLUSTERING_DATA_TBL VALUES ('10', 15.5, 15.5, 3); +INSERT INTO PAL_HIERARCHICAL_CLUSTERING_DATA_TBL VALUES ('11', 16.5, 15.5, 4); +INSERT INTO PAL_HIERARCHICAL_CLUSTERING_DATA_TBL VALUES ('12', 16.5, 16.5, 4); +INSERT INTO PAL_HIERARCHICAL_CLUSTERING_DATA_TBL VALUES ('13', 15.5, 16.5, 4); +INSERT INTO PAL_HIERARCHICAL_CLUSTERING_DATA_TBL VALUES ('14', 15.6, 16.2, 4); +INSERT INTO PAL_HIERARCHICAL_CLUSTERING_DATA_TBL VALUES ('15', 15.5, 0.5, 4); +INSERT INTO PAL_HIERARCHICAL_CLUSTERING_DATA_TBL VALUES ('16', 16.5, 0.5, 1); +INSERT INTO PAL_HIERARCHICAL_CLUSTERING_DATA_TBL VALUES ('17', 16.5, 1.5, 1); +INSERT INTO PAL_HIERARCHICAL_CLUSTERING_DATA_TBL VALUES ('18', 15.5, 1.5, 1); +INSERT INTO PAL_HIERARCHICAL_CLUSTERING_DATA_TBL VALUES ('19', 15.7, 1.6, 1); +​ +DROP TABLE #PAL_PARAMETER_TBL; +CREATE LOCAL TEMPORARY COLUMN TABLE #PAL_PARAMETER_TBL ("PARAM_NAME" VARCHAR(256), "INT_VALUE" INTEGER, "DOUBLE_VALUE" DOUBLE, "STRING_VALUE" VARCHAR(1000)); +INSERT INTO #PAL_PARAMETER_TBL VALUES ('THREAD_RATIO', NULL, 0.5, NULL); +INSERT INTO #PAL_PARAMETER_TBL VALUES ('CLUSTER_NUM', 4, NULL, NULL); +INSERT INTO #PAL_PARAMETER_TBL VALUES ('CLUSTER_METHOD', 4, NULL, NULL); +INSERT INTO #PAL_PARAMETER_TBL VALUES ('DISTANCE_FUNC', 10, NULL, NULL); +INSERT INTO #PAL_PARAMETER_TBL VALUES ('DISTANCE_DIMENSION', NULL, 3, NULL); +INSERT INTO #PAL_PARAMETER_TBL VALUES ('NORMALIZE_TYPE', 0, NULL, NULL); +INSERT INTO #PAL_PARAMETER_TBL VALUES ('CATEGORICAL_VARIABLE', NULL, NULL, 'X3'); +INSERT INTO #PAL_PARAMETER_TBL VALUES ('CATEGORY_WEIGHTS', NULL, 0.1, NULL); +​ +CALL _SYS_AFL.PAL_HIERARCHICAL_CLUSTERING (PAL_HIERARCHICAL_CLUSTERING_DATA_TBL, #PAL_PARAMETER_TBL, ?, ?); +``` @@ -159,7 +532,8 @@ Don't forget to remove this message. ## Related Best Practices
    - + +
@@ -170,20 +544,8 @@ Don't forget to remove this message. ## Related AI Capabilities -

- Content Creation -

- -
    - -
- -

- Information Analysis & Processing -

-
    - +
@@ -194,7 +556,11 @@ Don't forget to remove this message. ## Contributors - +
diff --git a/app/docs/02-technical-view/02-narrow-ai/08-training-deployment-custom-ai-models.mdx b/app/docs/02-technical-view/02-narrow-ai/08-training-deployment-custom-ai-models.mdx index 6e0defe3..cb6cb2b0 100644 --- a/app/docs/02-technical-view/02-narrow-ai/08-training-deployment-custom-ai-models.mdx +++ b/app/docs/02-technical-view/02-narrow-ai/08-training-deployment-custom-ai-models.mdx @@ -24,15 +24,6 @@ import PageContributors from "@site/src/components/PageContributors";

Training and Deployment of Custom AI Models

-::::danger[IMPORTANT - How to make this page visible] - -- To make this page visible on the sidebar, you need to remove the sidebar_class_name: hidden from the top of the file. -- To enable the links to this page (for example in the index.mdx file), you need to remove the disabled=true from the IconLinkButton component on that specific page. - -Don't forget to remove this message. - -:::: -
@@ -55,20 +46,20 @@ Don't forget to remove this message.
  • - + Teaser [TODO]
  • - + Webinar [TODO]
  • - + Webinar (PDF Presentation) [TODO] @@ -79,131 +70,130 @@ Don't forget to remove this message.
- - - Download Source Code - - - ## 1 Overview ### Description -... +The training and deployment of custom AI models within SAP BTP is designed to augment the company's existing capabilities by introducing **tailored AI solutions**. This section explores best practices for training and deploying custom models, focusing on integrating them seamlessly into existing business processes. + +Building custom AI models involves an initial assessment of existing solutions in the company’s SAP Business AI portfolio. This initial step ensures that the deployment leverages and complements the current technology landscape, rather than duplicating capabilities or functions. + +It is recommended that you go through [SAP Business AI](https://dam.sap.com/mac/download/ap/ghJGAiZ.htm?rc=10) to understand the strategy and AI offerings to support your business scenario. Before you go for a custom predictive model development, it is recommended to review the [available solutions](https://discovery-center.cloud.sap/index.html#/servicessearch/AI?regions=all&category=all) in the [SAP Business AI portfolio](https://discovery-center.cloud.sap/ai-catalog/) to find out-of-the-box offerings that suit your business needs. If you haven't found a suitable solution, then it's time to consider building a custom AI model on the [SAP BTP platform](/docs/technology/sap-business-technology-platform). ### Expected Outcome -... +The primary goal of this practice is to effectively integrate custom AI models (**tailored AI solutions**) into an existing business structure, thereby maximizing the utility and impact of AI technologies. It aims to optimize operations, improve decision-making, and deliver scalable, [sustainable AI](https://www.sap.com/india/assetdetail/2025/02/688181fe-f37e-0010-bca6-c68f7e60039b.html) improvements that align well with business goals. ### Benefits -... +- **Improved Operational Efficiency**: Custom AI models, in particular predictive models, help automate routine decision-making processes, reduce manual effort, and enable real-time insights. This is beneficial across different business domains, including logistics, procurement, finance, etc. +- **Data-Driven Decision Making**: Using AI models, you can uncover hidden patterns in historical data and empower business users to make informed decisions, improving accuracy and reducing risks in business processes. +- **Seamless Integration**: SAP BTP facilitates the integration of AI models with both SAP and non-SAP applications, Data from [SAP HANA](https://www.sap.com/products/data-cloud/hana.html), CDS views, [SAP DataSphere](https://www.sap.com/products/data-cloud/datasphere.html), and OData services, ensuring consistent and actionable insights directly within business workflows. +- **Scalability and Reusability**: Custom AI models developed on SAP BTP can be scaled easily and can be used across multiple processes and business units, which further enhances return on investment. +- **Personalization at Scale**: Using AI models, you can enable hyper-personalized experiences in sales, marketing, and customer service by anticipating customer needs and preferences. ### Key Concepts -... - -## 2 Pre-requisites - -... - -## 3 Key Choices and Guidelines - -... - -## 4 Implementation +- **Model Lifecycle Management**: It involves stages like data acquisition, preprocessing, feature engineering, model training, validation, deployment, and monitoring, which can be easily orchestrated using [SAP AI Core](/docs/technology/sap-ai-core) and [AI Launchpad](/docs/technology/sap-ai-launchpad). +- **Explainability and Trust**: Explainability is an important aspect in building trust in AI capabilities. Responsible model development should offer interpretable insights (via SHAP, LIME, etc.) to ensure business stakeholders can trust the predictions. +- **Real-Time and Batch Inference**: Real-time inference refers to immediate prediction based on the input, while Batch inference takes a batch of input and performs the inference together. [SAP BTP](https://www.sap.com/india/products/technology-platform.html) supports both inference patterns- real-time predictions via APIs or batch scoring using pipelines integrated into business processes. -... +### Use cases of Custom AI Models -
- - +Custom AI models help organizations forecast future outcomes based on historical data - turning transactional records into actionable foresight. Within SAP environments, custom AI models are used to improve planning, reduce risk, personalize interactions, and automate decisions across multiple lines of business. - - Download Source Code - +Here are a few examples of use cases for predictive modeling: +- **Payment Default Prediction**: Build a model to identify customers or invoices at risk of late payment, helping finance teams to avoid delayed payments. e.g., a custom model on [SAP AI Core](/docs/technology/sap-ai-core) predicts payment delays using open item data from S/4HANA Finance, customer risk profiles, and past payment history. +- **Sales Forecasting and Deal Conversion**: Use historical opportunity data to predict which deals are likely to close and when, improving sales planning accuracy. e.g., a model trained in [SAP HANA PAL](https://help.sap.com/docs/SAP_HANA_PLATFORM/319d36de4fd64ac3afbf91b1fb3ce8de/sap-hana-predictive-analysis-library-pal-c9eeed7.html?locale=en-US) forecasts conversion probabilities using opportunity status, engagement metrics, and salesperson history from SAP Sales Cloud. +- **Custom Demand Forecasting by Region or Channel**: Create granular demand forecasts tailored by product, location, or channel, e.g., a Python-based model on [BTP AI Core](/docs/technology/sap-ai-core) integrates POS data, SAP orders, and external weather trends to forecast demand for retail products by city. +- **Supplier Lead Time Risk Prediction**: Predict which POs may face delays based on historical delivery patterns, supplier behavior, and external signals, e.g., a [HANA PAL](https://help.sap.com/docs/SAP_HANA_PLATFORM/319d36de4fd64ac3afbf91b1fb3ce8de/sap-hana-predictive-analysis-library-pal-c9eeed7.html?locale=en-US) model analyzes PO data from Ariba and delivery KPIs to flag at-risk suppliers before issuing orders. +- **Predictive Maintenance for Custom Assets**: Train a model using IoT data to predict failures of industry-specific or non-standard machines. e.g., a predictive maintenance model deployed via [AI Core](/docs/technology/sap-ai-core) uses telemetry from manufacturing equipment to anticipate failures, integrated with SAP EAM. +- **Employee Exit Risk Modeling (Custom Factors)**: Build a model tailored to your workforce dynamics to detect attrition risks based on internal metrics, not just generic benchmarks. e.g., a custom [HANA PAL logistic regression model](https://help.sap.com/docs/SAP_HANA_PLATFORM/319d36de4fd64ac3afbf91b1fb3ce8de/logistic-regression-with-elastic-net-regularization-46effe5.html?locale=en-US) uses leave patterns, performance reviews, and engagement survey scores from SuccessFactors. +- **Custom Product Return Prediction**: Develop product-specific models to predict return likelihood based on order combinations, customer type, and past behavior. e.g., a model on [BTP AI Core](/docs/technology/sap-ai-core) predicts return rates using sales orders, complaints, and material master data from S/4HANA. +- **Churn Risk for Subscription-Based Services**: Forecast customer churn using usage data, ticket volume, and satisfaction scores. e.g. a custom classification model deployed on [AI Core](/docs/technology/sap-ai-core) uses support ticket logs, CSAT scores, and subscription usage data from SAP CRM or CX Suite. +- **Warehouse Inventory Risk Modeling**: Identify items likely to become out of stock using historical movement data, product lifecycle, and seasonal trends. e.g., A [decision tree model](https://help.sap.com/docs/SAP_HANA_PLATFORM/319d36de4fd64ac3afbf91b1fb3ce8de/decision-trees-c8ab80c.html?locale=en-US) flags slow-moving items and suggests liquidation or markdown actions, integrated with SAP EWM. -... +## 2 Pre-requisites - +### Commercial - +- [SAP AI Core](/docs/technology/sap-ai-core) +- [SAP AI Launchpad](/docs/technology/sap-ai-launchpad) +- [SAP HANA Cloud](https://discovery-center.cloud.sap/serviceCatalog/sap-hana-cloud?region=all&tab=service_plan) on SAP BTP (Optional) - - Download Source Code - +You can find pricing details for the above services in the [SAP Discovery Center](https://discovery-center.cloud.sap/serviceCatalog?commercialModel=btpea®ions=all). -... +### Technical - +1. Setup SAP Business Technology Platform (SAP BTP) subaccount ([Setup Guide](/docs/technology/sap-business-technology-platform)) +2. Create an instance of SAP AI Core ([Setup Guide](/docs/technology/sap-ai-core#setup)) +3. Subscribe to SAP AI Launchpad ([Setup Guide](/docs/technology/sap-ai-launchpad#setup)) +4. Create an instance of SAP HANA Cloud ([Setup Guide](/docs/technology/sap-hana-cloud#setup)) - +### High-level Reference Architecture - - Download Source Code - +![High-level Reference Architecture](@site/static/narrow-ai/training-deployment-custom-ai-models/images/arch.png) -... +## 3 Key Choices and Guidelines - +Custom model development and operationalizing is a multi-step process. These key implementation choices and guidelines at each step will help you achieve the best outcome for your custom AI scenario. +- **Problem Definition & Business Alignment** +Clearly define the ML problem and ensure it aligns with business goals to drive measurable value. Engage stakeholders early for domain-specific insights and success criteria. +- **Data Strategy** +Collect high-quality, representative data while ensuring ethical sourcing and regulatory compliance. Preprocess and engineer features to maximize model effectiveness and fairness. AI/ML use cases require a lot of data, thus, processing of data and pushing the Custom AI model development close to the data source helps to reduce data movement and latency. It is recommended to consider HANA-based [data pre-processing](https://help.sap.com/docs/SAP_HANA_PLATFORM/319d36de4fd64ac3afbf91b1fb3ce8de/preprocessing-algorithms-d521673.html?locale=en-US) and [ML algorithms](https://help.sap.com/docs/SAP_HANA_PLATFORM/319d36de4fd64ac3afbf91b1fb3ce8de/pal-procedures-f652a81.html?locale=en-US) if your data is already in [SAP HANA](https://discovery-center.cloud.sap/serviceCatalog/sap-hana-cloud?region=all&tab=). +- **Model Development** +Select appropriate algorithms based on problem complexity, interpretability, and available data. You can start with simpler algorithms or smaller models and then increase the complexity as needed, supporting your business case. If you are using deep learning for your use case, consider pretrained models for faster development cycles. +- **Model Evaluation & Validation** +Validate models using robust metrics, cross-validation, and edge-case testing. These evaluations and validations will help you build robust and reliable models. +- **Choosing the right resource plan** +Training and deploying a custom model would require compute resources, and SAP BTP provides a variety of [resource plan](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/choose-resource-plan-train) that caters to a variety of model development/deployment. Choosing the right resource plan helps to speed up your training cycle, reduce latency on inference, and be cost-effective. You may use the guidelines below to choose the best resource plan for your use case. + - Small training resource plans that are suitable to train your classic regressors, trees, XGB, and simple neural nets. + - Small inference resource plans to serve your classic regressors, trees, XGB, and simple neural nets. + - Larger inference resource plans to serve LLMs, vision models, and deeper/larger neural nets. + Note: [Large training resource plans](https://me.sap.com/notes/3485650) allow for distributed training/fine-tuning of larger models like LLMs, vision models, and deeper/larger neural nets, but they may be available only for SAP internal purposes. +- **Deployment & MLOps** +Use MLOps tools for reproducible deployment, scalability, and automated monitoring. Ensure [MLOps pipeline](https://help.sap.com/docs/ai-launchpad/sap-ai-launchpad/ml-operations?locale=en-US) or [CI/CD](https://discovery-center.cloud.sap/serviceCatalog/continuous-integration--delivery?region=all) pipelines are in place for efficient and secure model releases. You can also consider containerized (Docker-based) deployments to scale your AI applications. +- **Monitoring & Retraining** +Continuously [monitor model performance](https://help.sap.com/docs/ai-launchpad/sap-ai-launchpad/producer-view-metrics-for-model?locale=en-US), data drift, and user feedback. Set up retraining [workflows](https://help.sap.com/docs/ai-launchpad/sap-ai-launchpad/workflow-executables?locale=en-US) to maintain accuracy and relevance over time. +- **Security & Privacy** +Safeguard data and models through encryption, access control, and protection against adversarial attacks. Ensure privacy using anonymization, federated learning, or confidential computing where applicable. +- **AI Ethics & Responsible AI** +Embed fairness, transparency, and accountability throughout the model lifecycle. It is advised to use explainability tools and documentation (like model cards) to support ethical governance. See [Responsible AI](https://www.sap.com/india/products/artificial-intelligence/ai-ethics.html). - -
-
+## 4 Implementation -
+Based on the machine learning problem you are addressing, you can experiment with several models on the dataset that you are working with. Once the model is built and you are happy with the performance of that model, you can choose any of the following ways to deploy your custom model to support your business applications. -## Related Best Practices +AI Core supports container-based deployment. In order to deploy your model, you can perform the following steps: -
    - -
+1. [Landscape Setup (setting up AI Core /AI Launchpad)](https://developers.sap.com/tutorials/ai-core-setup.html): This is a one-time activity. You will be able to reuse the same setup for further deployments. +2. [Setup for Git Repositories](https://developers.sap.com/tutorials/ai-core-helloworld.html): This sets up your git repositories to have easier development management as well as pipeline management for model deployment in AI Core. +3. [Set up your environment for container-ready development](https://developers.sap.com/tutorials/ai-core-code.html): SAP AI Core allows Docker/container-based deployment. Before you deploy your model, you need to create a Docker/container for the same. +4. Build your custom AI model in a container-ready environment: You can build your custom AI model as per your requirement using classical machine learning models, neural networks, or lightweight LLMs as well. Below are some of the references for your model development. + a. [Build a House Price Predictor Model](https://sap.sharepoint.com/sites/210313/SitePages/Technical%20-%20Narrow%20AI.aspx): Based on your use case, you can leverage [best practice content](https://github.com/amanichopra/sap-aicore-train/tree/main) to build your Custom AI models. + b. Yoga Pose Image Classification: Use [training](https://github.com/amanichopra/model-serving-aicore) and [inference](https://developers.sap.com/tutorials/ai-core-tensorflow-byod.html) pipeline to create a classifier for yoga pose images. + c. [Utilizing GPUs in SAP AI Core](https://developers.sap.com/tutorials/ai-core-data.html): If your business application requires image processing, audio/video processing, or large computation, GPU provide a significant boost in processing time (both training and inference). You can leverage AI core instances with GPU support to leverage the same. + d. [Ingest data into your AI model with SAP AI Core](https://developers.sap.com/tutorials/ai-core-data.html) +5. [Generate Metrics and Compare Models in SAP AI Core](https://developers.sap.com/tutorials/ai-core-metrics.html): You can run all your experiments in an AI Core container and generate several metrics to compare the models to find the best fit for your business problem. +6. [Make Predictions for your AI model with SAP AI Core](https://developers.sap.com/tutorials/ai-core-deploy.html): Once you have finalized your model, you can create an inference pipeline to make it available for integration with your business applications. -
+The above steps help you build and deploy your AI model on the AI core. AI Launchpad provides you a unified access to AI Core and helps you manage your configuration, deployments end-to-end. -
+If you wish to perform the same activities using the command line in VS Code, you can leverage the [SAP AI Core Toolkit for VS Code](https://help.sap.com/docs/sap-ai-core/sap-ai-core-toolkit/sap-ai-core-toolkit?locale=en-US) and perform the same steps. [SAP AI Core using VS Code Toolkit](https://developers.sap.com/tutorials/ai-core-vs-code-toolkit.html) explains the step-by-step process to achieve the same.
-## Related AI Capabilities - -

- Content Creation -

- -
    - -
- -

- Information Analysis & Processing -

+## Related Best Practices
    - + + + + +
@@ -214,7 +204,10 @@ Don't forget to remove this message. ## Contributors - +
diff --git a/app/docs/02-technical-view/02-narrow-ai/index.mdx b/app/docs/02-technical-view/02-narrow-ai/index.mdx index ad94e542..7ca2719d 100644 --- a/app/docs/02-technical-view/02-narrow-ai/index.mdx +++ b/app/docs/02-technical-view/02-narrow-ai/index.mdx @@ -29,9 +29,9 @@ import PageViewTracker from "@site/src/components/tracking/PageViewTracker";
@@ -40,7 +40,7 @@ import PageViewTracker from "@site/src/components/tracking/PageViewTracker"; diff --git a/app/docs/02-technical-view/03-ai-services/04-custom-joule-skills.mdx b/app/docs/02-technical-view/03-ai-services/04-custom-joule-skills.mdx index 29954268..8936e668 100644 --- a/app/docs/02-technical-view/03-ai-services/04-custom-joule-skills.mdx +++ b/app/docs/02-technical-view/03-ai-services/04-custom-joule-skills.mdx @@ -1,8 +1,7 @@ --- title: "Custom Joule Skills" -description: "" +description: "Joule Skills is the approach to tailor Joule's unique capabilities to every specific business need." hide_title: true -sidebar_class_name: hidden --- import TabItem from "@theme/TabItem"; @@ -24,15 +23,6 @@ import PageContributors from "@site/src/components/PageContributors";

Custom Joule Skills

-::::danger[IMPORTANT - How to make this page visible] - -- To make this page visible on the sidebar, you need to remove the sidebar_class_name: hidden from the top of the file. -- To enable the links to this page (for example in the index.mdx file), you need to remove the disabled=true from the IconLinkButton component on that specific page. - -Don't forget to remove this message. - -:::: -
@@ -53,7 +43,7 @@ Don't forget to remove this message.
-
    + {/*
    • @@ -74,84 +64,428 @@ Don't forget to remove this message.
      • -
    +
*/}
- - - Download Source Code - - ## 1 Overview ### Description -... +Joule Skills is the approach to tailor Joule's unique capabilities to every specific business need. + +The supported way to extend Joule’s built-in capabilities is via Joule Studio (in SAP Build). It lets you define, build, deploy, monitor and improve your own specific workflows for Joule to follow. It also enables seamless and easy integration with APIs, both SAP and external, using the destinations in your BTP Subaccount and Build environment. + +Joule Skills are a vital tool to improving and finetuning Joule to accelerate your business and solve problems with no-code, easy to deploy solution. ### Expected Outcome -... +When using Joule Studio to develop custom skills you can expect: + +- Easy integration with SAP Systems and third-party APIs to extend base functionality +- Traceable version changes and transparency in deployment +- Workflow-level execution transparency +- Easy deployment +- Integration with SAP Build ### Benefits -... +- **No-code extension of functionality** - Joule Studio does not require any code to create a skill and give Joule more actions to perform +- **Integrated in SAP Build** - works in a familiar environment and allows for extension with RPA automations and workflows +- **Secure and compliant** - Joule runs within SAP Build/BTP environments, but skills may call external APIs via BTP Destinations; data flow should follow enterprise security and governance policies. +- **Integration made simple** - Joule Skills provides easy integration with SAP and third party APIs ### Key Concepts -... +Some key concepts are part of Build Process Automation and can be found ([here](https://help.sap.com/docs/build-process-automation/sap-build-process-automation-dev/using-sap-build-process-automation?state=DRAFT&version=Dev&locale=en-US)). + +Here are the Joule Studio specific concepts: +- **Joule Skill** - a predefined workflow that gets executed by Joule based on the description given to it. +- **Joule Skill Project** - project type that can contain multiple individual skills +- **Input Parameters** - required parameters for the defined skill in order to execute properly +- **Output Parameters** - parameters that can be passed to another part of a workflow as a result after the skill is complete ## 2 Pre-requisites -... +### Commercial +- Joule Subscription +- SAP Build with build-default service plan +- License for an active developer user in SAP Build +- BTP Subaccount with active subscriptions to SAP Build Process Automation and SAP Cloud Identity Services + +### Technical +1. Set up a SAP Business Technology Platform (SAP BTP) subaccount ([Setup Guide](/docs/technology/sap-business-technology-platform#setup-guide)) +2. Set up SAP Build ([Setup Guide](https://help.sap.com/docs/build-service/build-service-guide/initial-setup-of-sap-build?version=Cloud&locale=en-US)) +3. Set up subscription to Cloud Identity Services ([Setup Guide](https://help.sap.com/docs/cloud-identity-services/cloud-identity-services/initial-setup?locale=en-US&version=Cloud&q=Initial)) +4. Set up Joule Studio ([Setup Guide](https://help.sap.com/docs/Joule_Studio/45f9d2b8914b4f0ba731570ff9a85313/04b323352fa645238211ce017f634d34.html?locale=en-US#set-up-joule-studio-using-the-booster)) + +### High-level reference architecture + +![High-level reference architecture for Custom Joule Skills](@site/static/ai-services/custom-joule-skills/images/arch.png) ## 3 Key Choices and Guidelines -... +### Joule Skills Development -## 4 Implementation +- Be very clear and descriptive when writing the Joule Skill Description as it is used to determine if this skill is needed for the task at hand. +- Each skill should be a self-contained function that fulfils one specific task in order to minimize confusion, improve performance and supportability. +- It is not advisable to give Skills the ability to delete or modify critical data without consulting a human first. +- When creating multiple skills in a project, make sure to follow consistency in naming and ensure they do not overlap in function and description. Also bundle them based on common or complimentary functionality +- The deployment of Joule skills project should be done in the custom environment that can be shared with authorized users or user groups. +- When the 'Allow Joule to generate a response' option is disabled, ensure that a suitable custom message (via Send Message) is configured for every possible output of the action component +- Carefully designate mandatory input and output fields in the Joule skill based on the use case. Joule will not execute the action component until all required fields are captured from the user's prompt. If only some mandatory fields are provided, Joule will continue prompting the user to supply the remaining ones before proceeding +- Apply conditions to action outputs such as verifying that a list contains at least one object before triggering downstream steps. This helps ensure logical flow and prevents unnecessary errors +- For actions using the 'GET' method in Joule skills, it's recommended to apply precise filters to narrow the search scope and avoid API timeout issues +- Joule skill inputs are always treated as text (string data type). To ensure compatibility with downstream action components, convert these inputs to the appropriate data types using functions like StringToNumber(skill-input). +- Use meaningful parameter descriptions instead of abbreviated or encoded labels. This helps Joule better interpret the context during skill execution. For example, use ‘location’ instead of ‘loc’. +- If Joule fails to respond after a prompt, use the debug mode in Joule Studio to test skill interactions. Additionally, check the ‘Request Logs’ view to confirm whether the correct skill was triggered. -... +### Joule Skills Interactions -
- - +Provide clear, specific, and actionable prompts when interacting with Joule. The more precise your input, the better Joule can interpret and execute the intended skill. Here are a few illustrative examples: - - Download Source Code - +
+--- + +*Update the tax code in all the line items of the purchase order ‘123456’ with ‘I1’. Please confirm the update is successful after you complete the task.* -... +--- - +--- - +*Create a sales order using customer PO ‘123456’ for ‘Material1’ with a quantity of 10 PCs.* - - Download Source Code - +--- +
-... +**You may also take an approach with multi-line prompts as:** - +
+--- - +*Identify purchase order ‘123456’.* +*Update the tax code to I1 on all purchase order line items.* +*Generate a summary confirming the update and state any issues encountered.* - - Download Source Code - +--- -... +--- - +*The sales order should reference the customer's purchase order number 123456.* +*Include ‘Material1’ in the sales order line item with a quantity of 10 PCs.* - +---
+### Joule Skills Limitations + +- The deployment of Joule skills to an environment in the build tenant may require up to 5 minutes to 30 minutes to complete. +- The Field Map function cannot directly map a list of objects (output from an action component) to downstream build components like RPA automation artifacts. As a workaround, you can pass the entire list as a generic object (data type ‘Any’). While this won't trigger errors during project save or release, deployment of the Joule skill may result in errors. +- Joule skills can invoke published automations and processes from SAP Build Process Automation. However, the reverse i.e. calling Joule skills from SAP Build Process Automation is not supported. +- Automations and processes triggered from SAP Build Process Automation within Joule skills are asynchronous. Therefore, their outputs cannot be used as part of the ongoing user conversation in Joule. +- If permissions are modified in a shared environment, any deployed projects must be redeployed for the changes to take effect. + +## 4 Implementation + +### Useful example scenarios + +
+ + ### Read Sales Order Details + +

Description

+Create a Joule skill that reads sales order details based on given sales order number and displays details. + +This example demonstrates how Joule can perform read operations from APIs. + +

Tutorial Steps

+1. Appropriate description and ‘Allow Joule to generate a response’ activated in the Joule trigger as shown below: +![Step 1](@site/static/ai-services/custom-joule-skills/images/read-sales-order-details/1.png) +2. Joule input parameter/s and output parameters are configured as per below: +![Step 2-1](@site/static/ai-services/custom-joule-skills/images/read-sales-order-details/2.png) +![Step 2-2](@site/static/ai-services/custom-joule-skills/images/read-sales-order-details/3.png) +3. +Create a destination variable as type ‘destination’ as shown below: +![Step 3](@site/static/ai-services/custom-joule-skills/images/read-sales-order-details/4.png) +4. Bind the input parameters of the action component as shown below: +![Step 4](@site/static/ai-services/custom-joule-skills/images/read-sales-order-details/5.png) +5. Bind the Joule skill outputs (‘End’ step) to the action component output as shown below: +![Step 5](@site/static/ai-services/custom-joule-skills/images/read-sales-order-details/6.png) +6. Release and deploy to the shared environment to test the Joule skill. +*Please note: Choose the appropriate BTP destination while deploying the Joule Studio project.* +7. Launch the Joule Testing of the Joule interface from Control Tower->Environment->Joule tab->Launch + +

Demo

+ + +
+ +
+ + ### List Sales Order Items + +

Description

+Create a Joule skill that lists sales order items based on given sales order number and displaying in a list. + +This example demonstrates how Joule can perform read operations from APIs and display multiple values. + +

Tutorial Steps

+1. Add appropriate description and ‘Allow Joule to generate a response’ activated in the Joule trigger as shown below: +![Step 1](@site/static/ai-services/custom-joule-skills/images/list-sales-order-items/1.png) +2. Create Joule input parameter/s and output parameters are configured as per below: +![Step 2-1](@site/static/ai-services/custom-joule-skills/images/list-sales-order-items/2.png) +![Step 2-2](@site/static/ai-services/custom-joule-skills/images/list-sales-order-items/3.png) +3. Create a destination variable as type ‘destination’ as shown below. A destination variable is required to connect the action to a configured system. The destination configuration can be done after the skill is deployed. +![Step 3](@site/static/ai-services/custom-joule-skills/images/list-sales-order-items/4.png) +4. Bind the input parameters of the action component as shown below: +![Step 4](@site/static/ai-services/custom-joule-skills/images/list-sales-order-items/5.png) +5. Bind the Joule skill outputs (‘End’ step) to the action component output as shown below: +![Step 5](@site/static/ai-services/custom-joule-skills/images/list-sales-order-items/6.png) +6. Release and deploy to the shared environment to test the Joule skill. +*Please note: Choose the appropriate BTP destination while deploying the Joule Studio project.* +7. Launch the Joule Testing of the Joule interface from Control Tower->Environment->Joule tab->Launch + +

Demo

+ +
+ +
+ + ### Create Sales Order + +

Description

+Create a Joule Skill that creates a sales order based on given header data: Sold to, Salse Order Type, Sales Organization, Distribution Channel, Organizations Division and Purchase Order By Customer. + +This example demonstrates how Joule can create records in backend systems using APIs. + +

Tutorial Steps

+1. Add appropriate description and ‘Allow Joule to generate a response’ deactivated in the Joule trigger as shown below: +![Step 1](@site/static/ai-services/custom-joule-skills/images/create-sales-order/1.png) +2. Create Joule input parameter/s and output parameters are configured as per below: +![Step 2-1](@site/static/ai-services/custom-joule-skills/images/create-sales-order/2.png) +![Step 2-2](@site/static/ai-services/custom-joule-skills/images/create-sales-order/3.png) +3. Create a destination variable as type ‘destination’ as shown below: +![Step 3](@site/static/ai-services/custom-joule-skills/images/create-sales-order/4.png) +4. Bind the input parameters of the action component as shown below: +![Step 4](@site/static/ai-services/custom-joule-skills/images/create-sales-order/5.png) +5. Add ‘Send Message’ step to Joule skill flow and bind the action parameters at the ‘Text’ and ‘Title’ parameters, specify the system name and system URL in the ‘Action Buttons’ section as shown below: +![Step 5](@site/static/ai-services/custom-joule-skills/images/create-sales-order/6.png) +6. Bind the Joule skill outputs (‘End’ step) to the action component output as shown below: +![Step 6](@site/static/ai-services/custom-joule-skills/images/create-sales-order/7.png) +7. Release and deploy to the shared environment to test the Joule skill. +*Please note: Choose the appropriate BTP destination while deploying the Joule Studio project.* +8. Launch the Joule Testing of the Joule interface from Control Tower->Environment->Joule tab->Launch + +

Demo

+ +
+ +
+ + ### Update Sales Order + +

Description

+Create a Joule Skill that creates a sales order item and updates a given sales order by number. + +This example demonstrates how Joule can modify records in backend systems using APIs. + +

Tutorial Steps

+1. Add appropriate description and ‘Allow Joule to generate a response’ deactivated in the Joule trigger as shown below: +![Step 1](@site/static/ai-services/custom-joule-skills/images/update-sales-order/1.png) +2. Define Joule input parameter/s and output parameters are configured as per below: +![Step 2-1](@site/static/ai-services/custom-joule-skills/images/update-sales-order/2.png) +![Step 2-2](@site/static/ai-services/custom-joule-skills/images/update-sales-order/3.png) +3. Create a destination variable as type ‘destination’ as shown below: +![Step 3](@site/static/ai-services/custom-joule-skills/images/update-sales-order/4.png) +4. Bind the input parameters of the action component as shown below: +![Step 4](@site/static/ai-services/custom-joule-skills/images/update-sales-order/5.png) +5. Add ‘Send Message’ step to Joule skill flow and bind the action parameters at the ‘Text’ and ‘Title’ parameters, specify the system name and system URL in the ‘Action Buttons’ section as shown below: +![Step 5](@site/static/ai-services/custom-joule-skills/images/update-sales-order/6.png) +6. Bind the Joule skill outputs (‘End’ step) to the action component output as shown below: +![Step 6](@site/static/ai-services/custom-joule-skills/images/update-sales-order/7.png) +7. Release and deploy to the shared environment to test the Joule skill. +*Please note: Choose the appropriate BTP destination while deploying the Joule Studio project.* +8. Launch the Joule Testing of the Joule interface from Control Tower->Environment->Joule tab->Launch + +

Demo

+ +
+ +
+ + ### Invoke an approval workflow from SAP BPA + +

Description

+Create a Joule skill that invokes an approval workflow from SAP Build Process Automation. + +This example demonstrates how Joule can call existing Build Process Automation components. + +

Tutorial Steps

+1. Appropriate description and ‘Allow Joule to generate a response’ deactivated in the Joule trigger as shown below: +![Step 1](@site/static/ai-services/custom-joule-skills/images/invoke-an-approval-workflow-from-sap-bpa/1.png) +2. Joule input parameters are configured as per below: +![Step 2-1](@site/static/ai-services/custom-joule-skills/images/invoke-an-approval-workflow-from-sap-bpa/2.png) +3. Add the published BPA project of ‘sales order approval’ as shown below: +![Step 3](@site/static/ai-services/custom-joule-skills/images/invoke-an-approval-workflow-from-sap-bpa/3.png) +4. Bind the input parameters of the BPA project ‘sales order approval’ as shown below: +![Step 4](@site/static/ai-services/custom-joule-skills/images/invoke-an-approval-workflow-from-sap-bpa/4.png) +*Please Note: For the workflow item approval, you may assign a user (holding the process participant role), or the user group defined at the BTP subaccount level. In this example, initiator and approver has been setup as a same person and hence ‘Email’ from the Joule skill context has been used.* +5. Add ‘Send Message’ step to Joule skill flow and enter the message at the ‘Text’ and ‘Title’ parameters, specify the system name and system URL in the ‘Action Buttons’ section as shown below: +![Step 2-2](@site/static/ai-services/custom-joule-skills/images/invoke-an-approval-workflow-from-sap-bpa/5.png) +6. Release and deploy to the shared environment to test the Joule skill. +*Please note: Choose the appropriate BTP destination while deploying the Joule Studio project.* +7. Launch the Joule Testing of the Joule interface from Control Tower->Environment->Joule tab->Launch + +

Demo

+ +
+ +
+ + ### Download Sales Order Details using RPA Process + +

Description

+Create a Joule skill that invokes an RPA from SAP Build Process Automation to download top sales order details. + +This example demonstrates how Joule can call existing SAP Build Process Automation components. + +

Tutorial Steps

+1. Appropriate description and ‘Allow Joule to generate a response’ deactivated in the Joule trigger as shown below: +![Step 1](@site/static/ai-services/custom-joule-skills/images/download-sales-order-details-using-rpa-process/1.png) +2. Joule input and output parameters are configured as per below: +![Step 2-1](@site/static/ai-services/custom-joule-skills/images/download-sales-order-details-using-rpa-process/2.png) +![Step 2-2](@site/static/ai-services/custom-joule-skills/images/download-sales-order-details-using-rpa-process/3.png) +3. Create a destination variable as type ‘destination’ as shown below: +![Step 3](@site/static/ai-services/custom-joule-skills/images/download-sales-order-details-using-rpa-process/4.png) +4. Bind the input parameters of the action component as shown below: +![Step 4](@site/static/ai-services/custom-joule-skills/images/download-sales-order-details-using-rpa-process/5.png) +5. Use the type conversion method to convert the string to numeric as shown below: +![Step 5](@site/static/ai-services/custom-joule-skills/images/download-sales-order-details-using-rpa-process/6.png) +6. Add a condition step to verify the results should contain at least one item as shown below: +![Step 6-1](@site/static/ai-services/custom-joule-skills/images/download-sales-order-details-using-rpa-process/7.png) +![Step 6-2](@site/static/ai-services/custom-joule-skills/images/download-sales-order-details-using-rpa-process/8.png) +7. Add the published ‘Download suppliers in a spreadsheet’ automation and bind its input parameters as shown below: +![Step 7](@site/static/ai-services/custom-joule-skills/images/download-sales-order-details-using-rpa-process/9.png) +8. Add ‘Send Message’ step to Joule skill flow in the ‘if item found’ branch and specify the message for automation trigger as shown below: +![Step 8](@site/static/ai-services/custom-joule-skills/images/download-sales-order-details-using-rpa-process/10.png) +9. Add ‘Send Message’ step to Joule skill flow in the ‘no item found’ branch and specify the message as shown below: +![Step 9](@site/static/ai-services/custom-joule-skills/images/download-sales-order-details-using-rpa-process/11.png) +10. Bind the Joule skill outputs (‘End’ step) to the action component output as shown below: +![Step 10](@site/static/ai-services/custom-joule-skills/images/download-sales-order-details-using-rpa-process/12.png) +11. Release and deploy to the shared environment to test the Joule skill. +*Please note: Choose the appropriate BTP destination while deploying the Joule Studio project.* + +

Demo

+ +
+ +{/*
+ + ### Translate text through Joule using SAP Translation Hub + +

Description

+Create a Joule skill that uses the SAP Translation Hub API to translate text into another language . + +This example demonstrates how Joule can call existing SAP AI Services. + +

Tutorial Steps

+1. Appropriate description and ‘Allow Joule to generate a response’ activated in the Joule trigger as shown below: +![Step 1](@site/static/ai-services/custom-joule-skills/images/translate-text-through-joule-using-sap-translation-hub/1.png) +2. Joule input and output parameters are configured as per below: +![Step 2-1](@site/static/ai-services/custom-joule-skills/images/translate-text-through-joule-using-sap-translation-hub/placeholder.png) +![Step 2-1](@site/static/ai-services/custom-joule-skills/images/translate-text-through-joule-using-sap-translation-hub/placeholder.png) +3. Create a destination variable as type ‘destination’ as shown below: +![Step 3](@site/static/ai-services/custom-joule-skills/images/translate-text-through-joule-using-sap-translation-hub/placeholder.png) +4. Bind the input parameters of the action component as shown below: +![Step 4](@site/static/ai-services/custom-joule-skills/images/translate-text-through-joule-using-sap-translation-hub/placeholder.png) +5. Bind the Joule skill outputs (‘End’ step) to the action component output as shown below: +![Step 5](@site/static/ai-services/custom-joule-skills/images/translate-text-through-joule-using-sap-translation-hub/placeholder.png) +6. Create another Joule skill. Give an appropriate description and ‘Allow Joule to generate a response’ deactivated in the Joule trigger as shown below: +![Step 6](@site/static/ai-services/custom-joule-skills/images/translate-text-through-joule-using-sap-translation-hub/placeholder.png) +7. Joule input and output parameters are configured as per below: +![Step 7-1](@site/static/ai-services/custom-joule-skills/images/translate-text-through-joule-using-sap-translation-hub/placeholder.png) +![Step 7-2](@site/static/ai-services/custom-joule-skills/images/translate-text-through-joule-using-sap-translation-hub/placeholder.png) +8. Add the first Joule skill to get the BCP language code of the target language as shown below: +![Step 8](@site/static/ai-services/custom-joule-skills/images/translate-text-through-joule-using-sap-translation-hub/placeholder.png) +9. Bind the input parameters of the action component as shown below: +![Step 9](@site/static/ai-services/custom-joule-skills/images/translate-text-through-joule-using-sap-translation-hub/placeholder.png) +10. Create a destination variable as type ‘destination’ as shown below: +![Step 10](@site/static/ai-services/custom-joule-skills/images/translate-text-through-joule-using-sap-translation-hub/placeholder.png) +11. Bind the input parameters of the action component as shown below: +![Step 11](@site/static/ai-services/custom-joule-skills/images/translate-text-through-joule-using-sap-translation-hub/placeholder.png) +12. A dd ‘Send Message’ step to Joule skill flow branch and specify the message for automation trigger as shown below: +![Step 12](@site/static/ai-services/custom-joule-skills/images/translate-text-through-joule-using-sap-translation-hub/placeholder.png) +13. Bind the Joule skill outputs (‘End’ step) to the action component output as shown below: +![Step 13](@site/static/ai-services/custom-joule-skills/images/translate-text-through-joule-using-sap-translation-hub/placeholder.png) +14. Release and deploy to the shared environment to test the Joule skill. +*Please note: Choose the appropriate BTP destination while deploying the Joule Studio project.* +15. Launch the Joule Testing of the Joule interface from Control Tower->Environment->Joule tab->Launch + +

Demo

+ +
*/} + +
+ + ### Summarise text through Joule using SAP AI Core + +

Description

+This example demonstrates how a Joule skill can call existing SAP AI Core APIs. This Joule skill utilizes an LLM model deployed in SAP AI Core to generate a summary of the provided text based on the specified word limit. + +

Tutorial Steps

+1. Appropriate description and ‘Allow Joule to generate a response’ deactivated in the Joule trigger as shown below: +![Step 1](@site/static/ai-services/custom-joule-skills/images/summarise-text-through-joule-using-sap-ai-core/1.png) +2. Joule input and output parameters are configured as per below: +![Step 2-1](@site/static/ai-services/custom-joule-skills/images/summarise-text-through-joule-using-sap-ai-core/2.png) +*Please Note – We are keeping the word length as optional here as ‘30’ word limit has been maintained as default value in the action definition.* +3. Create a destination variable as type ‘destination’ as shown below: +![Step 2-2](@site/static/ai-services/custom-joule-skills/images/summarise-text-through-joule-using-sap-ai-core/3.png) +4. Bind the input parameters of the action component as shown below: +![Step 3](@site/static/ai-services/custom-joule-skills/images/summarise-text-through-joule-using-sap-ai-core/4.png) +5. Add ‘Send Message’ step to Joule skill flow in the ‘no item found’ branch and specify the message as shown below: +![Step 4](@site/static/ai-services/custom-joule-skills/images/summarise-text-through-joule-using-sap-ai-core/5.png) +6. Bind the Joule skill outputs (‘End’ step) to the action component output as shown below: +![Step 5](@site/static/ai-services/custom-joule-skills/images/summarise-text-through-joule-using-sap-ai-core/6.png) +7. Release and deploy to the shared environment to test the Joule skill. +*Please note: Choose the appropriate BTP destination while deploying the Joule Studio project.* +8. Launch the Joule Testing of the Joule interface from Control Tower->Environment->Joule tab->Launch + +

Demo

+ +
+ +### Learn more + +We recommend this tutorial: [Build Your First Joule Skill in Joule Studio | SAP Tutorials](https://developers-qa-blue.wcms-nonprod.c.eu-de-2.cloud.sap/group.joule-studio-first-skill.html) +
@@ -159,7 +493,7 @@ Don't forget to remove this message. ## Related Best Practices
    - +
@@ -171,19 +505,11 @@ Don't forget to remove this message. ## Related AI Capabilities

- Content Creation -

- -
    - -
- -

- Information Analysis & Processing + Conversational Interaction

    - +
@@ -194,7 +520,10 @@ Don't forget to remove this message. ## Contributors - + diff --git a/app/docs/02-technical-view/03-ai-services/index.mdx b/app/docs/02-technical-view/03-ai-services/index.mdx index fc90071f..01bd60ad 100644 --- a/app/docs/02-technical-view/03-ai-services/index.mdx +++ b/app/docs/02-technical-view/03-ai-services/index.mdx @@ -29,6 +29,7 @@ import PageViewTracker from "@site/src/components/tracking/PageViewTracker";

Document Processing

    +
diff --git a/app/src/css/custom.css b/app/src/css/custom.css index fde25c07..fb730da5 100644 --- a/app/src/css/custom.css +++ b/app/src/css/custom.css @@ -45,6 +45,8 @@ /* Table of contents */ --ifm-toc-border-color: var(--sap-horizon-neutral-5); + + --ifm-alert-border-color: var(--sap-horizon-neutral-5); } /* For readability concerns, you should choose a lighter palette in dark mode. */ @@ -1143,3 +1145,33 @@ html button#truste-consent-required:hover { height: 100px; } } + +/* Details */ + +details.alert { + border: 1px solid var(--sap-horizon-neutral-5); + background-color: var(--sap-horizon-neutral-2) !important; + color: var(--ifm-heading-color); + --docusaurus-details-decoration-color: var(--ifm-color-primary); + --ifm-alert-border-color: var(--ifm-color-primary); +} + +[data-theme="dark"] details.alert { + border-color: var(--sap-horizon-neutral-10); + background-color: var(--sap-horizon-neutral-11) !important; +} + +details.alert summary > * { + margin: 0; + font-size: 1rem; + display: inline-block; +} + +details.alert summary a { + color: var(--ifm-color-primary); + text-decoration: none; +} + +details.alert summary a:hover { + text-decoration: underline; +} diff --git a/app/src/data/capabilityIcons.ts b/app/src/data/capabilityIcons.ts index 3b449eb8..e1479baf 100644 --- a/app/src/data/capabilityIcons.ts +++ b/app/src/data/capabilityIcons.ts @@ -114,17 +114,18 @@ const capabilityIcons: Record = { // Narrow AI "/docs/technical-view/narrow-ai/regression": "scatter-chart", "/docs/technical-view/narrow-ai/time-series-forecasting": "area-chart", - "/docs/technical-view/narrow-ai/classification": "lateness", + "/docs/technical-view/narrow-ai/classification": "tag", "/docs/technical-view/narrow-ai/recommendation": "lateness", - "/docs/technical-view/narrow-ai/clustering": "lateness", + "/docs/technical-view/narrow-ai/clustering": "heatmap-chart", "/docs/technical-view/narrow-ai/anomaly-detection": "quality-issue", "/docs/technical-view/narrow-ai/predictive-ai-bdc-btp": "lateness", - "/docs/technical-view/narrow-ai/deployment-custom-predictive-models": "puzzle", "/docs/technical-view/narrow-ai/data-preparation-and-model-evaluation": "timesheet", + "/docs/technical-view/narrow-ai/training-deployment-custom-ai-models": "puzzle", // AI Services "/docs/technical-view/ai-services/document-information-extraction": "ppt-attachment", - "/docs/technical-view/ai-services/sap-document-ai": "attachment-video" + "/docs/technical-view/ai-services/sap-document-ai": "attachment-video", + "/docs/technical-view/ai-services/custom-joule-skills": "da" }; /** diff --git a/app/src/data/contributors.json b/app/src/data/contributors.json index 2c5be69e..e20f099e 100644 --- a/app/src/data/contributors.json +++ b/app/src/data/contributors.json @@ -64,6 +64,10 @@ "rajeshwari-kute": { "name": "Kute, Rajeshwari", "photo": "/img/contributors/rajeshwari-kute.jpg" + }, + "prasun-das": { + "name": "Das, Prasun", + "photo": "/img/contributors/prasun-das.jpg" } } } diff --git a/app/static/ai-services/custom-joule-skills/images/arch.png b/app/static/ai-services/custom-joule-skills/images/arch.png new file mode 100644 index 00000000..7e0e9107 Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/arch.png differ diff --git a/app/static/ai-services/custom-joule-skills/images/create-sales-order/1.png b/app/static/ai-services/custom-joule-skills/images/create-sales-order/1.png new file mode 100644 index 00000000..0a5f465c Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/create-sales-order/1.png differ diff --git a/app/static/ai-services/custom-joule-skills/images/create-sales-order/2.png b/app/static/ai-services/custom-joule-skills/images/create-sales-order/2.png new file mode 100644 index 00000000..01c3bcd1 Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/create-sales-order/2.png differ diff --git a/app/static/ai-services/custom-joule-skills/images/create-sales-order/3.png b/app/static/ai-services/custom-joule-skills/images/create-sales-order/3.png new file mode 100644 index 00000000..3f050f68 Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/create-sales-order/3.png differ diff --git a/app/static/ai-services/custom-joule-skills/images/create-sales-order/4.png b/app/static/ai-services/custom-joule-skills/images/create-sales-order/4.png new file mode 100644 index 00000000..fdf25680 Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/create-sales-order/4.png differ diff --git a/app/static/ai-services/custom-joule-skills/images/create-sales-order/5.png b/app/static/ai-services/custom-joule-skills/images/create-sales-order/5.png new file mode 100644 index 00000000..fb6ef81c Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/create-sales-order/5.png differ diff --git a/app/static/ai-services/custom-joule-skills/images/create-sales-order/6.png b/app/static/ai-services/custom-joule-skills/images/create-sales-order/6.png new file mode 100644 index 00000000..05acecda Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/create-sales-order/6.png differ diff --git a/app/static/ai-services/custom-joule-skills/images/create-sales-order/7.png b/app/static/ai-services/custom-joule-skills/images/create-sales-order/7.png new file mode 100644 index 00000000..f509559e Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/create-sales-order/7.png differ diff --git a/app/static/ai-services/custom-joule-skills/images/download-sales-order-details-using-rpa-process/1.png b/app/static/ai-services/custom-joule-skills/images/download-sales-order-details-using-rpa-process/1.png new file mode 100644 index 00000000..da7597df Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/download-sales-order-details-using-rpa-process/1.png differ diff --git a/app/static/ai-services/custom-joule-skills/images/download-sales-order-details-using-rpa-process/10.png b/app/static/ai-services/custom-joule-skills/images/download-sales-order-details-using-rpa-process/10.png new file mode 100644 index 00000000..1de03991 Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/download-sales-order-details-using-rpa-process/10.png differ diff --git a/app/static/ai-services/custom-joule-skills/images/download-sales-order-details-using-rpa-process/11.png b/app/static/ai-services/custom-joule-skills/images/download-sales-order-details-using-rpa-process/11.png new file mode 100644 index 00000000..a89b028d Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/download-sales-order-details-using-rpa-process/11.png differ diff --git a/app/static/ai-services/custom-joule-skills/images/download-sales-order-details-using-rpa-process/12.png b/app/static/ai-services/custom-joule-skills/images/download-sales-order-details-using-rpa-process/12.png new file mode 100644 index 00000000..783880d4 Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/download-sales-order-details-using-rpa-process/12.png differ diff --git a/app/static/ai-services/custom-joule-skills/images/download-sales-order-details-using-rpa-process/2.png b/app/static/ai-services/custom-joule-skills/images/download-sales-order-details-using-rpa-process/2.png new file mode 100644 index 00000000..d690306c Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/download-sales-order-details-using-rpa-process/2.png differ diff --git a/app/static/ai-services/custom-joule-skills/images/download-sales-order-details-using-rpa-process/3.png b/app/static/ai-services/custom-joule-skills/images/download-sales-order-details-using-rpa-process/3.png new file mode 100644 index 00000000..d2d0b24d Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/download-sales-order-details-using-rpa-process/3.png differ diff --git a/app/static/ai-services/custom-joule-skills/images/download-sales-order-details-using-rpa-process/4.png b/app/static/ai-services/custom-joule-skills/images/download-sales-order-details-using-rpa-process/4.png new file mode 100644 index 00000000..78dc5310 Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/download-sales-order-details-using-rpa-process/4.png differ diff --git a/app/static/ai-services/custom-joule-skills/images/download-sales-order-details-using-rpa-process/5.png b/app/static/ai-services/custom-joule-skills/images/download-sales-order-details-using-rpa-process/5.png new file mode 100644 index 00000000..54474ca2 Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/download-sales-order-details-using-rpa-process/5.png differ diff --git a/app/static/ai-services/custom-joule-skills/images/download-sales-order-details-using-rpa-process/6.png b/app/static/ai-services/custom-joule-skills/images/download-sales-order-details-using-rpa-process/6.png new file mode 100644 index 00000000..a44cd2c9 Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/download-sales-order-details-using-rpa-process/6.png differ diff --git a/app/static/ai-services/custom-joule-skills/images/download-sales-order-details-using-rpa-process/7.png b/app/static/ai-services/custom-joule-skills/images/download-sales-order-details-using-rpa-process/7.png new file mode 100644 index 00000000..d1149655 Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/download-sales-order-details-using-rpa-process/7.png differ diff --git a/app/static/ai-services/custom-joule-skills/images/download-sales-order-details-using-rpa-process/8.png b/app/static/ai-services/custom-joule-skills/images/download-sales-order-details-using-rpa-process/8.png new file mode 100644 index 00000000..e92145bd Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/download-sales-order-details-using-rpa-process/8.png differ diff --git a/app/static/ai-services/custom-joule-skills/images/download-sales-order-details-using-rpa-process/9.png b/app/static/ai-services/custom-joule-skills/images/download-sales-order-details-using-rpa-process/9.png new file mode 100644 index 00000000..47daf194 Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/download-sales-order-details-using-rpa-process/9.png differ diff --git a/app/static/ai-services/custom-joule-skills/images/invoke-an-approval-workflow-from-sap-bpa/1.png b/app/static/ai-services/custom-joule-skills/images/invoke-an-approval-workflow-from-sap-bpa/1.png new file mode 100644 index 00000000..7839d659 Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/invoke-an-approval-workflow-from-sap-bpa/1.png differ diff --git a/app/static/ai-services/custom-joule-skills/images/invoke-an-approval-workflow-from-sap-bpa/2.png b/app/static/ai-services/custom-joule-skills/images/invoke-an-approval-workflow-from-sap-bpa/2.png new file mode 100644 index 00000000..3db717f0 Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/invoke-an-approval-workflow-from-sap-bpa/2.png differ diff --git a/app/static/ai-services/custom-joule-skills/images/invoke-an-approval-workflow-from-sap-bpa/3.png b/app/static/ai-services/custom-joule-skills/images/invoke-an-approval-workflow-from-sap-bpa/3.png new file mode 100644 index 00000000..79c062de Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/invoke-an-approval-workflow-from-sap-bpa/3.png differ diff --git a/app/static/ai-services/custom-joule-skills/images/invoke-an-approval-workflow-from-sap-bpa/4.png b/app/static/ai-services/custom-joule-skills/images/invoke-an-approval-workflow-from-sap-bpa/4.png new file mode 100644 index 00000000..aca9d028 Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/invoke-an-approval-workflow-from-sap-bpa/4.png differ diff --git a/app/static/ai-services/custom-joule-skills/images/invoke-an-approval-workflow-from-sap-bpa/5.png b/app/static/ai-services/custom-joule-skills/images/invoke-an-approval-workflow-from-sap-bpa/5.png new file mode 100644 index 00000000..d0ed0369 Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/invoke-an-approval-workflow-from-sap-bpa/5.png differ diff --git a/app/static/ai-services/custom-joule-skills/images/list-sales-order-items/1.png b/app/static/ai-services/custom-joule-skills/images/list-sales-order-items/1.png new file mode 100644 index 00000000..590d6b74 Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/list-sales-order-items/1.png differ diff --git a/app/static/ai-services/custom-joule-skills/images/list-sales-order-items/2.png b/app/static/ai-services/custom-joule-skills/images/list-sales-order-items/2.png new file mode 100644 index 00000000..fbdc2497 Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/list-sales-order-items/2.png differ diff --git a/app/static/ai-services/custom-joule-skills/images/list-sales-order-items/3.png b/app/static/ai-services/custom-joule-skills/images/list-sales-order-items/3.png new file mode 100644 index 00000000..14d0934f Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/list-sales-order-items/3.png differ diff --git a/app/static/ai-services/custom-joule-skills/images/list-sales-order-items/4.png b/app/static/ai-services/custom-joule-skills/images/list-sales-order-items/4.png new file mode 100644 index 00000000..b6cbbb5a Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/list-sales-order-items/4.png differ diff --git a/app/static/ai-services/custom-joule-skills/images/list-sales-order-items/5.png b/app/static/ai-services/custom-joule-skills/images/list-sales-order-items/5.png new file mode 100644 index 00000000..bc30e2cc Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/list-sales-order-items/5.png differ diff --git a/app/static/ai-services/custom-joule-skills/images/list-sales-order-items/6.png b/app/static/ai-services/custom-joule-skills/images/list-sales-order-items/6.png new file mode 100644 index 00000000..9f131ec8 Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/list-sales-order-items/6.png differ diff --git a/app/static/ai-services/custom-joule-skills/images/read-sales-order-details/1.png b/app/static/ai-services/custom-joule-skills/images/read-sales-order-details/1.png new file mode 100644 index 00000000..32e499c1 Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/read-sales-order-details/1.png differ diff --git a/app/static/ai-services/custom-joule-skills/images/read-sales-order-details/2.png b/app/static/ai-services/custom-joule-skills/images/read-sales-order-details/2.png new file mode 100644 index 00000000..e6724523 Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/read-sales-order-details/2.png differ diff --git a/app/static/ai-services/custom-joule-skills/images/read-sales-order-details/3.png b/app/static/ai-services/custom-joule-skills/images/read-sales-order-details/3.png new file mode 100644 index 00000000..bee0471f Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/read-sales-order-details/3.png differ diff --git a/app/static/ai-services/custom-joule-skills/images/read-sales-order-details/4.png b/app/static/ai-services/custom-joule-skills/images/read-sales-order-details/4.png new file mode 100644 index 00000000..39b56a88 Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/read-sales-order-details/4.png differ diff --git a/app/static/ai-services/custom-joule-skills/images/read-sales-order-details/5.png b/app/static/ai-services/custom-joule-skills/images/read-sales-order-details/5.png new file mode 100644 index 00000000..2479c5b3 Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/read-sales-order-details/5.png differ diff --git a/app/static/ai-services/custom-joule-skills/images/read-sales-order-details/6.png b/app/static/ai-services/custom-joule-skills/images/read-sales-order-details/6.png new file mode 100644 index 00000000..465f8e1d Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/read-sales-order-details/6.png differ diff --git a/app/static/ai-services/custom-joule-skills/images/summarise-text-through-joule-using-sap-ai-core/1.png b/app/static/ai-services/custom-joule-skills/images/summarise-text-through-joule-using-sap-ai-core/1.png new file mode 100644 index 00000000..020bc83e Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/summarise-text-through-joule-using-sap-ai-core/1.png differ diff --git a/app/static/ai-services/custom-joule-skills/images/summarise-text-through-joule-using-sap-ai-core/2.png b/app/static/ai-services/custom-joule-skills/images/summarise-text-through-joule-using-sap-ai-core/2.png new file mode 100644 index 00000000..8c1a539e Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/summarise-text-through-joule-using-sap-ai-core/2.png differ diff --git a/app/static/ai-services/custom-joule-skills/images/summarise-text-through-joule-using-sap-ai-core/3.png b/app/static/ai-services/custom-joule-skills/images/summarise-text-through-joule-using-sap-ai-core/3.png new file mode 100644 index 00000000..25444cf2 Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/summarise-text-through-joule-using-sap-ai-core/3.png differ diff --git a/app/static/ai-services/custom-joule-skills/images/summarise-text-through-joule-using-sap-ai-core/4.png b/app/static/ai-services/custom-joule-skills/images/summarise-text-through-joule-using-sap-ai-core/4.png new file mode 100644 index 00000000..00a00185 Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/summarise-text-through-joule-using-sap-ai-core/4.png differ diff --git a/app/static/ai-services/custom-joule-skills/images/summarise-text-through-joule-using-sap-ai-core/5.png b/app/static/ai-services/custom-joule-skills/images/summarise-text-through-joule-using-sap-ai-core/5.png new file mode 100644 index 00000000..0f40360d Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/summarise-text-through-joule-using-sap-ai-core/5.png differ diff --git a/app/static/ai-services/custom-joule-skills/images/summarise-text-through-joule-using-sap-ai-core/6.png b/app/static/ai-services/custom-joule-skills/images/summarise-text-through-joule-using-sap-ai-core/6.png new file mode 100644 index 00000000..4678dd5c Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/summarise-text-through-joule-using-sap-ai-core/6.png differ diff --git a/app/static/ai-services/custom-joule-skills/images/translate-text-through-joule-using-sap-translation-hub/1.png b/app/static/ai-services/custom-joule-skills/images/translate-text-through-joule-using-sap-translation-hub/1.png new file mode 100644 index 00000000..0cf28bc6 Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/translate-text-through-joule-using-sap-translation-hub/1.png differ diff --git a/app/static/ai-services/custom-joule-skills/images/translate-text-through-joule-using-sap-translation-hub/placeholder.png b/app/static/ai-services/custom-joule-skills/images/translate-text-through-joule-using-sap-translation-hub/placeholder.png new file mode 100644 index 00000000..b2530460 Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/translate-text-through-joule-using-sap-translation-hub/placeholder.png differ diff --git a/app/static/ai-services/custom-joule-skills/images/update-sales-order/1.png b/app/static/ai-services/custom-joule-skills/images/update-sales-order/1.png new file mode 100644 index 00000000..e485e1dd Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/update-sales-order/1.png differ diff --git a/app/static/ai-services/custom-joule-skills/images/update-sales-order/2.png b/app/static/ai-services/custom-joule-skills/images/update-sales-order/2.png new file mode 100644 index 00000000..4f22d213 Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/update-sales-order/2.png differ diff --git a/app/static/ai-services/custom-joule-skills/images/update-sales-order/3.png b/app/static/ai-services/custom-joule-skills/images/update-sales-order/3.png new file mode 100644 index 00000000..94cb6d68 Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/update-sales-order/3.png differ diff --git a/app/static/ai-services/custom-joule-skills/images/update-sales-order/4.png b/app/static/ai-services/custom-joule-skills/images/update-sales-order/4.png new file mode 100644 index 00000000..6d6a48d5 Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/update-sales-order/4.png differ diff --git a/app/static/ai-services/custom-joule-skills/images/update-sales-order/5.png b/app/static/ai-services/custom-joule-skills/images/update-sales-order/5.png new file mode 100644 index 00000000..3a0a07d8 Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/update-sales-order/5.png differ diff --git a/app/static/ai-services/custom-joule-skills/images/update-sales-order/6.png b/app/static/ai-services/custom-joule-skills/images/update-sales-order/6.png new file mode 100644 index 00000000..978fbebf Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/update-sales-order/6.png differ diff --git a/app/static/ai-services/custom-joule-skills/images/update-sales-order/7.png b/app/static/ai-services/custom-joule-skills/images/update-sales-order/7.png new file mode 100644 index 00000000..d15c1193 Binary files /dev/null and b/app/static/ai-services/custom-joule-skills/images/update-sales-order/7.png differ diff --git a/app/static/img/contributors/prasun-das.jpg b/app/static/img/contributors/prasun-das.jpg new file mode 100644 index 00000000..cdfd5b92 Binary files /dev/null and b/app/static/img/contributors/prasun-das.jpg differ diff --git a/app/static/narrow-ai/classification/images/arch.png b/app/static/narrow-ai/classification/images/arch.png new file mode 100644 index 00000000..5c8dc503 Binary files /dev/null and b/app/static/narrow-ai/classification/images/arch.png differ diff --git a/app/static/narrow-ai/classification/images/classification-model.png b/app/static/narrow-ai/classification/images/classification-model.png new file mode 100644 index 00000000..6145d9df Binary files /dev/null and b/app/static/narrow-ai/classification/images/classification-model.png differ diff --git a/app/static/narrow-ai/clustering/images/arch.png b/app/static/narrow-ai/clustering/images/arch.png new file mode 100644 index 00000000..5c8dc503 Binary files /dev/null and b/app/static/narrow-ai/clustering/images/arch.png differ diff --git a/app/static/narrow-ai/training-deployment-custom-ai-models/images/arch.png b/app/static/narrow-ai/training-deployment-custom-ai-models/images/arch.png new file mode 100644 index 00000000..88ac3a81 Binary files /dev/null and b/app/static/narrow-ai/training-deployment-custom-ai-models/images/arch.png differ