Bump version to 5.5.0 [run doc]

Author: maziyarpanahi

CHANGELOG

@@ -1,3 +1,26 @@
+========
+5.5.0
+========
+----------------
+New Features & Enhancements
+----------------
+* Introduced QWEN2Transformer (#14188)
+* Introduced MiniCPM (#14205)
+* Introduced NLLB (#14209)
+* Implemented Nomic embeddings (#14217)
+* Introduced CamemBertForZeroShotClassification annotator (#14354)
+* Implemented Mxbai Embeddings (#14355)
+* Introduced AlbertForZeroShotClassification (#14361)
+* Introduced Phi-3 (#14373)
+* Implemented Starcoder2 for causal language modeling (#14358)
+* Integrated llama.cpp (#14364)
+* Implemented SnowFlake (#14353)
+* Introduced ONNX support to vision annotators (#14356)
+* Introduced ONNX and OpenVINO support to Missing Annotators (#14359)
+* Added OpenVINO install instructions (#14382)
+* Exported notebooks for release candidate (#14393)
+
+
 ========
 5.4.2
 ========
@@ -9,7 +32,6 @@ New Features & Enhancements
 * Added aggressiveMatching parameter to DocumentSimilarityRanker annotator
 
 
-
 ========
 5.4.1
 ========

README.md

@@ -17,19 +17,31 @@
         <img src="https://static.pepy.tech/personalized-badge/spark-nlp?period=total&units=international_system&left_color=grey&right_color=orange&left_text=pip%20downloads" /></a>
 </p>
 
-Spark NLP is a state-of-the-art Natural Language Processing library built on top of Apache Spark. It provides **simple**, **performant** & **accurate** NLP annotations for machine learning pipelines that **scale** easily in a distributed
-environment.
-Spark NLP comes with **36000+** pretrained **pipelines** and **models** in more than **200+** languages.
+Spark NLP is a state-of-the-art Natural Language Processing library built on top of Apache Spark. It provides **simple**, **performant** & **accurate** NLP annotations for machine learning pipelines that **scale** easily in a distributed environment.
+
+Spark NLP comes with **83000+** pretrained **pipelines** and **models** in more than **200+** languages.
 It also offers tasks such as **Tokenization**, **Word Segmentation**, **Part-of-Speech Tagging**, Word and Sentence **Embeddings**, **Named Entity Recognition**, **Dependency Parsing**, **Spell Checking**, **Text Classification**, **Sentiment Analysis**, **Token Classification**, **Machine Translation** (+180 languages), **Summarization**, **Question Answering**, **Table Question Answering**, **Text Generation**, **Image Classification**, **Image to Text (captioning)**, **Automatic Speech Recognition**, **Zero-Shot Learning**, and many more [NLP tasks](#features).
 
-**Spark NLP** is the only open-source NLP library in **production** that offers state-of-the-art transformers such as **BERT**, **CamemBERT**, **ALBERT**, **ELECTRA**, **XLNet**, **DistilBERT**, **RoBERTa**, **DeBERTa**, **XLM-RoBERTa**, **Longformer**, **ELMO**, **Universal Sentence Encoder**, **Llama-2**, **M2M100**, **BART**, **Instructor**, **E5**, **Google T5**, **MarianMT**, **OpenAI GPT2**, **Vision Transformers (ViT)**, **OpenAI Whisper**, and many more not only to **Python** and **R**, but also to **JVM** ecosystem (**Java**, **Scala**, and **Kotlin**) at **scale** by extending **Apache Spark** natively.
+**Spark NLP** is the only open-source NLP library in **production** that offers state-of-the-art transformers such as **BERT**, **CamemBERT**, **ALBERT**, **ELECTRA**, **XLNet**, **DistilBERT**, **RoBERTa**, **DeBERTa**, **XLM-RoBERTa**, **Longformer**, **ELMO**, **Universal Sentence Encoder**, **Llama-2**, **M2M100**, **BART**, **Instructor**, **E5**, **Google T5**, **MarianMT**, **OpenAI GPT2**, **Vision Transformers (ViT)**, **OpenAI Whisper**, **Llama**, **Mistral**, **Phi**, **Qwen2**, and many more not only to **Python** and **R**, but also to **JVM** ecosystem (**Java**, **Scala**, and **Kotlin**) at **scale** by extending **Apache Spark** natively.
+
+## Model Importing Support
+
+Spark NLP provides easy support for importing models from various popular frameworks:
+
+- **TensorFlow**
+- **ONNX**
+- **OpenVINO**
+- **Llama.cpp (GGUF)**
+
+This wide range of support allows you to seamlessly integrate models from different sources into your Spark NLP workflows, enhancing flexibility and compatibility with existing machine learning ecosystems.
 
 ## Project's website
 
 Take a look at our official Spark NLP page: [https://sparknlp.org/](https://sparknlp.org/) for user
 documentation and examples
 
 ## Features
+
 - [Text Preprocessing](https://sparknlp.org/docs/en/features#text-preproccesing)
 - [Parsing and Analysis](https://sparknlp.org/docs/en/features#parsing-and-analysis)
 - [Sentiment and Classification](https://sparknlp.org/docs/en/features#sentiment-and-classification)
@@ -51,7 +63,7 @@ $ java -version
 $ conda create -n sparknlp python=3.7 -y
 $ conda activate sparknlp
 # spark-nlp by default is based on pyspark 3.x
-$ pip install spark-nlp==5.5.0-rc1 pyspark==3.3.1
+$ pip install spark-nlp==5.5.0 pyspark==3.3.1
 ```
 
 In Python console or Jupyter `Python3` kernel:
@@ -108,6 +120,7 @@ community and we had to build most of the dependencies by ourselves to make them
 architectures, however, they may not work in some environments.
 
 ## Pipelines and Models
+
 For a quick example of using pipelines and models take a look at our official [documentation](https://sparknlp.org/docs/en/install#pipelines-and-models)
 
 #### Please check out our Models Hub for the full list of [pre-trained models](https://sparknlp.org/models) with examples, demo, benchmark, and more
@@ -116,10 +129,11 @@ For a quick example of using pipelines and models take a look at our official [d
 
 ### Apache Spark Support
 
-Spark NLP *5.5.0-rc1* has been built on top of Apache Spark 3.4 while fully supports Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, 3.4.x, and 3.5.x
+Spark NLP *5.5.0* has been built on top of Apache Spark 3.4 while fully supports Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, 3.4.x, and 3.5.x
 
 | Spark NLP | Apache Spark 3.5.x | Apache Spark 3.4.x | Apache Spark 3.3.x | Apache Spark 3.2.x | Apache Spark 3.1.x | Apache Spark 3.0.x | Apache Spark 2.4.x | Apache Spark 2.3.x |
 |-----------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|
+| 5.5.x     | YES                | YES                | YES                | YES                | YES                | YES                | NO                 | NO                 |
 | 5.4.x     | YES                | YES                | YES                | YES                | YES                | YES                | NO                 | NO                 |
 | 5.3.x     | YES                | YES                | YES                | YES                | YES                | YES                | NO                 | NO                 |
 | 5.2.x     | YES                | YES                | YES                | YES                | YES                | YES                | NO                 | NO                 |
@@ -132,6 +146,8 @@ Find out more about `Spark NLP` versions from our [release notes](https://github
 
 | Spark NLP | Python 3.6 | Python 3.7 | Python 3.8 | Python 3.9 | Python 3.10| Scala 2.11 | Scala 2.12 |
 |-----------|------------|------------|------------|------------|------------|------------|------------|
+| 5.5.x     | NO         | YES        | YES        | YES        | YES        | NO         | YES        |
+| 5.4.x     | NO         | YES        | YES        | YES        | YES        | NO         | YES        |
 | 5.3.x     | NO         | YES        | YES        | YES        | YES        | NO         | YES        |
 | 5.2.x     | NO         | YES        | YES        | YES        | YES        | NO         | YES        |
 | 5.1.x     | NO         | YES        | YES        | YES        | YES        | NO         | YES        |
@@ -141,38 +157,45 @@ Find out more about 4.x `SparkNLP` versions in our official [documentation](http
 
 ### Databricks Support
 
-Spark NLP 5.5.0-rc1 has been tested and is compatible with the following runtimes:
+Spark NLP 5.5.0 has been tested and is compatible with the following runtimes:
 
 | **CPU**            | **GPU**            |
 |--------------------|--------------------|
-| 14.0 / 14.0 ML     | 14.0 ML & GPU      |
 | 14.1 / 14.1 ML     | 14.1 ML & GPU      |
 | 14.2 / 14.2 ML     | 14.2 ML & GPU      |
 | 14.3 / 14.3 ML     | 14.3 ML & GPU      |
+| 15.0 / 15.0 ML     | 15.0 ML & GPU      |
+| 15.1 / 15.0 ML     | 15.1 ML & GPU      |
+| 15.2 / 15.0 ML     | 15.2 ML & GPU      |
+| 15.3 / 15.0 ML     | 15.3 ML & GPU      |
+| 15.4 / 15.0 ML     | 15.4 ML & GPU      |
 
 We are compatible with older runtimes. For a full list check databricks support in our official [documentation](https://sparknlp.org/docs/en/install#databricks-support)
 
 ### EMR Support
 
-Spark NLP 5.5.0-rc1 has been tested and is compatible with the following EMR releases:
+Spark NLP 5.5.0 has been tested and is compatible with the following EMR releases:
 
 | **EMR Release**    |
 |--------------------|
 | emr-6.13.0         |
 | emr-6.14.0         |
 | emr-6.15.0         |
 | emr-7.0.0          |
+| emr-7.1.0          |
+| emr-7.2.0          |
 
 We are compatible with older EMR releases. For a full list check EMR support in our official [documentation](https://sparknlp.org/docs/en/install#emr-support)
 
 Full list of [Amazon EMR 6.x releases](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-6x.html)
-Full list 5.5.0-rc1mazon EMR 7.x releases](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-7x.html)
+Full list of [Amazon EMR 7.x releases](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-7x.html)
 
 NOTE: The EMR 6.1.0 and 6.1.1 are not supported.
 
 ## Installation
 
 ### Command line (requires internet connection)
+
 To install spark-nlp packages through command line follow [these instructions](https://sparknlp.org/docs/en/install#command-line) from our official documentation
 
 ### Scala
@@ -182,18 +205,19 @@ deployed to Maven central. To add any of our packages as a dependency in your ap
 from our official documentation.
 
 If you are interested, there is a simple SBT project for Spark NLP to guide you on how to use it in your
-projects [Spark NLP SBT S5.5.0-rc1r](https://github.com/maziyarpanahi/spark-nlp-starter)
+projects [Spark NLP SBT S5.5.0r](https://github.com/maziyarpanahi/spark-nlp-starter)
 
 ### Python
 
 Spark NLP supports Python 3.7.x and above depending on your major PySpark version.
 Check all available installations for Python in our official [documentation](https://sparknlp.org/docs/en/install#python)
 
-
 ### Compiled JARs
+
 To compile the jars from source follow [these instructions](https://sparknlp.org/docs/en/compiled#jars) from our official documenation
 
 ## Platform-Specific Instructions
+
 For detailed instructions on how to use Spark NLP on supported platforms, please refer to our official documentation:
 
 | Platform                | Supported Language(s) |
@@ -206,7 +230,6 @@ For detailed instructions on how to use Spark NLP on supported platforms, please
 | [EMR Cluster](https://sparknlp.org/docs/en/install#emr-cluster)           | Scala, Python         |
 | [GCP Dataproc Cluster](https://sparknlp.org/docs/en/install#gcp-dataproc) | Scala, Python         |
 
-
 ### Offline
 
 Spark NLP library and all the pre-trained models/pipelines can be used entirely offline with no access to the Internet.
@@ -227,7 +250,7 @@ In Spark NLP we can define S3 locations to:
 
 Please check [these instructions](https://sparknlp.org/docs/en/install#s3-integration) from our official documentation.
 
-## Document5.5.0-rc1
+## Document5.5.0
 
 ### Examples
 
@@ -260,7 +283,7 @@ the Spark NLP library:
     keywords = {Spark, Natural language processing, Deep learning, Tensorflow, Cluster},
     abstract = {Spark NLP is a Natural Language Processing (NLP) library built on top of Apache Spark ML. It provides simple, performant & accurate NLP annotations for machine learning pipelines that can scale easily in a distributed environment. Spark NLP comes with 1100+ pretrained pipelines and models in more than 192+ languages. It supports nearly all the NLP tasks and modules that can be used seamlessly in a cluster. Downloaded more than 2.7 million times and experiencing 9x growth since January 2020, Spark NLP is used by 54% of healthcare organizations as the world’s most widely used NLP library in the enterprise.}
     }
-}5.5.0-rc1
+}5.5.0
 ```
 
 ## Community support

build.sbt

@@ -6,7 +6,7 @@ name := getPackageName(is_silicon, is_gpu, is_aarch64)
 
 organization := "com.johnsnowlabs.nlp"
 
-version := "5.5.0-rc1"
+version := "5.5.0"
 
 (ThisBuild / scalaVersion) := scalaVer
 

docs/_layouts/landing.html

@@ -201,7 +201,7 @@ <h3 class="grey h3_title">{{ _section.title }}</h3>
                   <div class="highlight-box">
     {% highlight bash %}
     # Using PyPI
-    $ pip install spark-nlp==5.5.0-rc1
+    $ pip install spark-nlp==5.5.0
 
     # Using Anaconda/Conda
     $ conda install -c johnsnowlabs spark-nlp

docs/en/advanced_settings.md

@@ -52,7 +52,7 @@ spark = SparkSession.builder
     .config("spark.kryoserializer.buffer.max", "2000m")
     .config("spark.jsl.settings.pretrained.cache_folder", "sample_data/pretrained")
     .config("spark.jsl.settings.storage.cluster_tmp_dir", "sample_data/storage")
-    .config("spark.jars.packages", "com.johnsnowlabs.nlp:spark-nlp_2.12:5.5.0-rc1")
+    .config("spark.jars.packages", "com.johnsnowlabs.nlp:spark-nlp_2.12:5.5.0")
     .getOrCreate()
 ```
 
@@ -66,7 +66,7 @@ spark-shell \
   --conf spark.kryoserializer.buffer.max=2000M \
   --conf spark.jsl.settings.pretrained.cache_folder="sample_data/pretrained" \
   --conf spark.jsl.settings.storage.cluster_tmp_dir="sample_data/storage" \
-  --packages com.johnsnowlabs.nlp:spark-nlp_2.12:5.5.0-rc1
+  --packages com.johnsnowlabs.nlp:spark-nlp_2.12:5.5.0
 ```
 
 **pyspark:**
@@ -79,7 +79,7 @@ pyspark \
   --conf spark.kryoserializer.buffer.max=2000M \
   --conf spark.jsl.settings.pretrained.cache_folder="sample_data/pretrained" \
   --conf spark.jsl.settings.storage.cluster_tmp_dir="sample_data/storage" \
-  --packages com.johnsnowlabs.nlp:spark-nlp_2.12:5.5.0-rc1
+  --packages com.johnsnowlabs.nlp:spark-nlp_2.12:5.5.0
 ```
 
 **Databricks:**

docs/en/concepts.md

@@ -66,7 +66,7 @@ $ java -version
 $ conda create -n sparknlp python=3.7 -y
 $ conda activate sparknlp
 # spark-nlp by default is based on pyspark 3.x
-$ pip install spark-nlp==5.5.0-rc1 pyspark==3.3.1 jupyter
+$ pip install spark-nlp==5.5.0 pyspark==3.3.1 jupyter
 $ jupyter notebook
 ```
 

docs/en/examples.md

@@ -18,7 +18,7 @@ $ java -version
 # should be Java 8 (Oracle or OpenJDK)
 $ conda create -n sparknlp python=3.7 -y
 $ conda activate sparknlp
-$ pip install spark-nlp==5.5.0-rc1 pyspark==3.3.1
+$ pip install spark-nlp==5.5.0 pyspark==3.3.1
 ```
 
 </div><div class="h3-box" markdown="1">
@@ -40,7 +40,7 @@ This script comes with the two options to define `pyspark` and `spark-nlp` versi
 # -p is for pyspark
 # -s is for spark-nlp
 # by default they are set to the latest
-!bash colab.sh -p 3.2.3 -s 5.5.0-rc1
+!bash colab.sh -p 3.2.3 -s 5.5.0
 ```
 
 [Spark NLP quick start on Google Colab](https://colab.research.google.com/github/JohnSnowLabs/spark-nlp/blob/master/examples/python/quick_start_google_colab.ipynb) is a live demo on Google Colab that performs named entity recognitions and sentiment analysis by using Spark NLP pretrained pipelines.

docs/en/hardware_acceleration.md

@@ -50,7 +50,7 @@ Since the new Transformer models such as BERT for Word and Sentence embeddings a
 | DeBERTa Large     |        +477%(5.8x)        |
 | Longformer Base   |         +52%(1.5x)        |
 
-Spark NLP 5.5.0-rc1 is built with TensorFlow 2.7.1 and the following NVIDIA® software are only required for GPU support:
+Spark NLP 5.5.0 is built with TensorFlow 2.7.1 and the following NVIDIA® software are only required for GPU support:
 
 - NVIDIA® GPU drivers version 450.80.02 or higher
 - CUDA® Toolkit 11.2