deberta_v3_base_token_classifier_ontonotes giving nonsense results

[dkaufman-rc]

Is there an existing issue for this?

• I have searched the existing issues and did not find a match.

Who can help?

The deberta_v3_base_token_classifier_ontonotes classifier seems to be working very poorly at present. When I run the sample code from https://sparknlp.org/2022/05/06/deberta_v3_base_token_classifier_ontonotes_en_3_0.html, it produces nonsense results - it's returning a lot of found items that aren't valid named entities at all, and the classifications it's giving them are also quite bad.

I'm switching to a different annotator model, so I don't need support on this personally. Just wanted you to know that something about this probably needs to be repaired or withdrawn from public use.

What are you working on?

NER

Current Behavior

On the example text William Henry Gates III (born October 28, 1955) is an American business magnate, software developer, investor, and philanthropist. He is best known as the co-founder of Microsoft Corporation. During his career at Microsoft, Gates held the positions of chairman, chief executive officer (CEO), president and chief software architect, while also being the largest individual shareholder until May 2014. He is one of the best-known entrepreneurs and pioneers of the microcomputer revolution of the 1970s and 1980s. Born and raised in Seattle, Washington, Gates co-founded Microsoft with childhood friend Paul Allen in 1975, in Albuquerque, New Mexico; it went on to become the world's largest personal computer software company. Gates led the company as chairman and CEO until stepping down as CEO in January 2000, but he remained chairman and became chief software architect. During the late 1990s, Gates had been criticized for his business tactics, which have been considered anti-competitive. This opinion has been upheld by numerous court rulings. In June 2006, Gates announced that he would be transitioning to a part-time role at Microsoft and full-time work at the Bill & Melinda Gates Foundation, the private charitable foundation that he and his wife, Melinda Gates, established in 2000.[9] He gradually transferred his duties to Ray Ozzie and Craig Mundie. He stepped down as chairman of Microsoft in February 2014 and assumed a new post as technology adviser to support the newly appointed CEO Satya Nadella. that some of your examples use, this annotator is currently returning results like:

|{III, 20, 22, {entity -> NORP, sentence -> 0, chunk -> 0, confidence -> 0.91501796}}              |
|{held, 230, 233, {entity -> CARDINAL, sentence -> 0, chunk -> 1, confidence -> 0.60650784}}       |
|{chief, 262, 266, {entity -> PERSON, sentence -> 0, chunk -> 2, confidence -> 0.70263743}}        |
|{chief, 307, 311, {entity -> PERSON, sentence -> 0, chunk -> 3, confidence -> 0.77321976}}        |
|{microcomputer, 463, 475, {entity -> PERSON, sentence -> 0, chunk -> 4, confidence -> 0.9698}}    |
|{1975, 615, 618, {entity -> DATE, sentence -> 0, chunk -> 5, confidence -> 0.5885551}}            |
|{2000, 806, 809, {entity -> NORP, sentence -> 0, chunk -> 6, confidence -> 0.6080411}}            |
|{chief, 848, 852, {entity -> PERSON, sentence -> 0, chunk -> 7, confidence -> 0.65038043}}        |
|{opinion, 999, 1005, {entity -> GPE, sentence -> 0, chunk -> 8, confidence -> 0.6541373}}         |
|{part-time, 1116, 1124, {entity -> CARDINAL, sentence -> 0, chunk -> 9, confidence -> 0.9248513}} |
|{work, 1158, 1161, {entity -> CARDINAL, sentence -> 0, chunk -> 10, confidence -> 0.91869617}}    |
|{Melinda, 1177, 1183, {entity -> PERSON, sentence -> 0, chunk -> 11, confidence -> 0.6550622}}    |
|{Melinda, 1259, 1265, {entity -> PERSON, sentence -> 0, chunk -> 12, confidence -> 0.78115696}}   |
|{2000.[9], 1289, 1296, {entity -> NORP, sentence -> 0, chunk -> 13, confidence -> 0.7357296}}     |
|{transferred, 1311, 1321, {entity -> PERSON, sentence -> 0, chunk -> 14, confidence -> 0.8192084}}|
|{Ozzie, 1341, 1345, {entity -> CARDINAL, sentence -> 0, chunk -> 15, confidence -> 0.47456023}

Expected Behavior

Something a lot closer to

+-----------------------------------+-----+----+---------+----------+
|chunk                              |begin|end |ner_label|confidence|
+-----------------------------------+-----+----+---------+----------+
|William Henry Gates III            |0    |22  |PERSON   |0.563375  |
|October 28, 1955                   |30   |45  |DATE     |0.696075  |
|American                           |54   |61  |NORP     |0.9063    |
|Microsoft Corporation              |169  |189 |ORG      |0.98434997|
|Microsoft                          |213  |221 |ORG      |0.9997    |
|Gates                              |224  |228 |PERSON   |0.7739    |
|May 2014                           |391  |398 |DATE     |0.9596    |
|one                                |407  |409 |CARDINAL |0.8819    |
|the 1970s and 1980s                |491  |509 |DATE     |0.382175  |
|Born                               |512  |515 |PERSON   |0.5431    |
|Seattle                            |531  |537 |GPE      |0.9992    |
|Washington                         |540  |549 |GPE      |0.9982    |
|Gates                              |552  |556 |PERSON   |0.9113    |
|Microsoft                          |569  |577 |ORG      |0.9772    |
|Paul Allen                         |601  |610 |PERSON   |0.9894    |
|1975                               |615  |618 |DATE     |0.9223    |

Steps To Reproduce

import sparknlp

from sparknlp.pretrained import *
from sparknlp.base import *
from sparknlp.annotator import *

document_assembler = DocumentAssembler()\
.setInputCol("text")\
.setOutputCol("document")

tokenizer = Tokenizer()\
.setInputCols(['document'])\
.setOutputCol('token') 

tokenClassifier = DeBertaForTokenClassification.pretrained("deberta_v3_base_token_classifier_ontonotes", "en")\
.setInputCols(["document", "token"])\
.setOutputCol("ner")\
.setCaseSensitive(True)\
.setMaxSentenceLength(512) 

# since output column is IOB/IOB2 style, NerConverter can extract entities
ner_converter = NerConverter()\
.setInputCols(['document', 'token', 'ner'])\
.setOutputCol('entities') 

pipeline = Pipeline(stages=[
document_assembler,
tokenizer,
tokenClassifier,
ner_converter
])

text_list = ["""William Henry Gates III (born October 28, 1955) is an American business magnate, software developer, investor, and philanthropist. He is best known as the co-founder of Microsoft Corporation. During his career at Microsoft, Gates held the positions of chairman, chief executive officer (CEO), president and chief software architect, while also being the largest individual shareholder until May 2014. He is one of the best-known entrepreneurs and pioneers of the microcomputer revolution of the 1970s and 1980s. Born and raised in Seattle, Washington, Gates co-founded Microsoft with childhood friend Paul Allen in 1975, in Albuquerque, New Mexico; it went on to become the world's largest personal computer software company. Gates led the company as chairman and CEO until stepping down as CEO in January 2000, but he remained chairman and became chief software architect. During the late 1990s, Gates had been criticized for his business tactics, which have been considered anti-competitive. This opinion has been upheld by numerous court rulings. In June 2006, Gates announced that he would be transitioning to a part-time role at Microsoft and full-time work at the Bill & Melinda Gates Foundation, the private charitable foundation that he and his wife, Melinda Gates, established in 2000.[9] He gradually transferred his duties to Ray Ozzie and Craig Mundie. He stepped down as chairman of Microsoft in February 2014 and assumed a new post as technology adviser to support the newly appointed CEO Satya Nadella.""",
             """The Mona Lisa is a 16th century oil painting created by Leonardo. It's held at the Louvre in Paris."""]

from pyspark.sql.types import StringType, IntegerType

df = spark.createDataFrame(text_list, StringType()).toDF("text")
result = pipeline.fit(df).transform(df)

import pyspark.sql.functions as F

result.select(F.explode(F.arrays_zip(result.entities.result, result.entities.begin, result.entities.end, result.entities.metadata)).alias("cols")) \
                       .show(1000,truncate=False)


### Spark NLP version and Apache Spark

sparknlp 5.5.0
spark 3.5.0

Databricks

### Type of Spark Application

_No response_

### Java Version

_No response_

### Java Home Directory

_No response_

### Setup and installation

_No response_

### Operating System and Version

_No response_

### Link to your project (if available)

_No response_

### Additional Information

_No response_

[maziyarpanahi]

Thanks @dkaufman-rc for reporting this issue. I can confirm this is indeed a bug. There are some new changes made inside DeBertaClassification tensorflow backend that might have caused this.

@ahmedlone127 Could you please have a look? You are the last committer to this annotator, I remember you fixed something so I leave it to you to debug this further.