Conversation Classification

[hamedwaezi01]

In this thread you can find the progress related to the "Conversation Classification" problem.
You may find different approaches like classic ML, Feedforward, CNN, RNN, LSTM, GRU etc. here.
I will try to cover other aspects of the project, like preprocessing and previous works, in different issues.

[hamedwaezi01]

For the last few days, I have been getting some suspiciously wrong metric values and have tried to figure out the reason. I thought it was the model configuration itself but later realized the library that was used for the calculation was the problem. I fixed this issue by calculating the metrics, including accuracy, recall, precision, and area under the ROC curve manually.

Additionally, I updated the K-fold cross-validation to Stratified K-fold cross-validation; the former splits the data without keeping the distribution of labels, while the latter considers the label distribution.

You can find some codes that make the datasets balanced according to the specified ratio of positive over negative. I uploaded a balanced dataset with a ratio of 0.4 to the Teams channel of the Osprey Project.

Also, according to the stats that were retrieved earlier, there was a demand to filter undesired records. Right now, one of the datasets passes only conversations with exactly 2 chatters and with more than 3 messages. Any new dataset can easily apply this feature by overriding the BaseDataset.filter_records method.

[hosseinfani]

@hamedwaezi01
as we talked today, please use a well-known library for metric calculation like scikit or pytrec, ... Nobdy rely on our code for calculating metrics, which is very critical point of evaluation and research.

[hamedwaezi01]

as we talked, I ran some mock prediction-target pairs and updated the code. it is now working.

[hosseinfani]

@hamedwaezi01
what does "working" mean :D
Please put some results/figures to show that.

[hamedwaezi01]

I used the torchmetrics lib again but this time, I ran some small mock prediction-target, like extreme cases where everything is wrong or right or a single entry is wrong, and got the expected results.

[hamedwaezi01]

I currently experimented with 3 models (Feedforward, RNN, CNN) on a balanced (predatory/all == 0.4) toy test. I preserved the cross-validation splits and ran the same records on each model. We expect that the models have similar relative performance if we run them on the real dataset.
The resulting metrics for each model on the test are as follows:

Feedforward: AUCROC: 0.7933513 | AUCPR: 0.6991223 | accuracy: 0.4496788 | precision: 0.9438503 | recall: 0.4172577
Simple RNN: AUCROC: 0.5137964 | AUCPR: 0.6405249 | accuracy: 0.6070664 | precision: 0.0213904 | recall: 0.8888889
Simple CNN: AUCROC: 0.5048176 | AUCPR: 0.4587629 | accuracy: 0.4004283 | precision: 1.0000000 | recall: 0.4004283

As we expected, Feedforward outperformed the other two models. I believe it is because the hyperparameters for RNN and CNN should be tuned. Other than RNN, we can experiment with its more complex siblings, like LSTM and GRU.

NOTES:
You can find the debug-level logs of this experiment under this file: logs/persist/05-09-2023-13-26-41.log

[hamedwaezi01]

There is a problem with RNN giving nan values (this issue), and I am not experimenting with the model on the balanced NON-toy dataset to see if it helps or not.
I also played with how the loss value is applied (using BCELossWithLogits, for example, to be numerically more stable); I still got a nan in one run.
I will let you know about the RNN problem. If you have any ideas, please drop a comment here or on the another issue for RNN.

[hosseinfani]

@hamedwaezi01
thanks for the update.

• About the results on toy.balanced.ds, think of drawing the roc or pr curve as you can see the pr/rec trade-off in rnn baseline.
• For hyperparam tuning, you can use bo: https://towardsdatascience.com/mango-a-new-way-to-make-bayesian-optimisation-in-python-a1a09989c6d8
• I had same problem in opentf. I log the solution here: nan for train and loss values when training dnn OpeNTF#4
  Also, look at the code changes. Hope it helps.

[hamedwaezi01]

Thanks,
I have been trying some configurations, and before implementing big changes, I thought of trying different optimizers. Since the RNN does not work well with the toy set, I am running it on a larger balanced dataset; up until now, the model is learning according to recall and precision metrics, but they could be more stable.

About the curves, I am saving them, and I will post them in future updates.

Another thing about the built-in PyTorch RNN is that it's very slow. I plan to implement a basic but faster RNN model soon.

[hosseinfani]

@hamedwaezi01
thanks for the update.
we need to talk.
don't go with rnn implementation please. find why it's slow. does it engage gpu? we don't reinvent the wheel unless it's absolute neccessary 😀

[hamedwaezi01]

Actually, it does use the dedicated GPU almost to the fullest extent. The recurrent-based models do not support sparse matrix, and I think it is affecting the speed.
BTW, my first run of RNN with Adam optimizer is finished. The evaluation metrics for the test set are:
test set -> AUCROC: 0.8691356 | AUCPR: 0.8675280 | accuracy: 0.8809677 | precision: 0.7524753 | recall: 0.9376459
NOTE: Let's not forget that the datasets are balanced with a ratio of 0.4 (predatory/all)
You can find the logs here. The figures for precision-recall and TPR-FPR curves are as follows:

[hamedwaezi01]

I ran the LSTM module yesterday. I used an Adam optimizer and ReduceLROnPlateau scheduler. The log file for this session can be found here, and you can find the parameters for optimizer and scheduler. The test results for this session and the previous RNN session are as follows respectively:
LSTM test set -> AUCROC: 0.9208137 | AUCPR: 0.9174601 | accuracy: 0.8924213 | precision: 0.9135670 | recall: 0.8334961
RNN test set -> AUCROC: 0.8691356 | AUCPR: 0.8675280 | accuracy: 0.8809677 | precision: 0.7524753 | recall: 0.9376459
The difference between the LSTM and RNN is the loss values figure and higher AUCROC value. I am unsure if it is good, but the loss value did not fluctuate as much as that of RNN.

Simultaneously, I read and learned more about the transformers and attention mechanisms while the model ran.

Right now, I am running the GRU. A couple of forums noted that LSTM usually outperforms GRU. So it is more of an experiment.

Following is the loss value figure per epoch for the best fold of this session: