int8 Quantization

[Phaired]

Hey, I want to try quantizing models. If I go for int8, will I be able to run them without issues? I saw that RTEN supports int8 as a type, but is ocrs updated ?

[robertknight]

RTen has int8 support on the main branch, but not yet in the released version. Once the next version is released (it will be v0.16) and Ocrs is updated to use it, you will be able to run int8 models. I expect this to happen fairly soon (next couple of weeks).

[robertknight]

ocrs v0.10.0 has now been released which is compatible with rten v0.16.0, which supports int8 models. A caveat is that only Conv and MatMul operators have support for int8 internally. For other operators (eg. GRU) the weights can be quantized in the model file, but they will be "inflated" to fp32 during inference, which will run at the same speed as before.

[Phaired]

Oh that was fast, ty
I will look into it and keep you updated if I get good results.

[Phaired]

Hey, I tried something with RTen 0.17 and OCRs, and on a fresh install of rten-convert, I tried to quantize the rec model and it worked great.

(working-model-py3.13) remybarranco@MacBook-Pro-de-Remy quant % time ocrs img.png                                                                 
When the tool is run for the first time, it will download the required models automatically and store them in
~/.cache/ocrs
? Additional examples
Extract text from an image and write to content.txt
$ ocrs image.png -o content.txt
ocrs img.png  0.94s user 0.04s system 98% cpu 1.002 total

(working-model-py3.13) remybarranco@MacBook-Pro-de-Remy quant % time ocrs img.png --rec-model rec_int8.rten 
When the tool is run for the first time, it will download the required models automatically and store them in
~/.cache/ocrs
? Additional examples
Extract text from an image and write to content.txt
$ ocrs image.png -o content.txt
ocrs img.png --rec-model rec_int8.rten  0.80s user 0.04s system 99% cpu 0.848 total

But I'm getting a strange issue when I try to add the detection model from your Hugging Face repo. I downloaded and converted it using rten-convert, but I can’t get any output, even without quantization.

(working-model-py3.13) remybarranco@MacBook-Pro-de-Remy quant % time ocrs img.png --rec-model rec_int8.rten --detect-model text-detection-ssfbcj81.rten 
E
ocrs img.png --rec-model rec_int8.rten --detect-model   0.27s user 0.05s system 95% cpu 0.333 total

As you can see, the red area is really strange. I converted the model using rten-convert, both with and without the --v1 flag.
(yes my test image is from the doc)

[robertknight]

From memory I think the text detection model on Hugging Face required a different (lower?) detection threshold to get comparable results to the original model.

It is on my TODO list to retrain the model so that it works with the same parameters as the original.

Regarding quantization of the recognition model, the convolution weights can be quantized but the GRU operator used to produce the character sequence doesn't yet supported quantized weights. That's also on my TODO list.

[Phaired]

ok I changed the threshold from 0.2 to 0.5 and it worked, ty I'm gonna try quantization on the detection model.
The quantization on the recognition model get me from ~1000ms to 850ms on the same image with almost the same result.