Issues using SeamlessM4T .ggml and .so files for WhatsApp transcription (need guidance)

[dorlanpabon]

Hi everyone,

I came across some files on a forum that are apparently used for transcribing audio in apps like WhatsApp. I’m not very experienced with these tools, but I thought it might be worth sharing here in case someone with more expertise can confirm whether they can be used with whisper.cpp, or knows how to integrate them.

Here are the files I found, originally from Facebook Research’s Seamless Communication project:

• seamless_nano_ggml_9.ggml
• libggml_core_ops_enhanced.so
• libggml_core_ops_universal.so

Reference:
🔗 https://github.com/facebookresearch/seamless_communication/tree/main/ggml

Direct links to the files:

• seamless_nano_ggml_9.ggml: https://drive.google.com/file/d/1-ALpKoSw_w9Ux8mLYvspGX1TErqnWW1Y/view?usp=sharing
• libggml_core_ops_universal.so: https://drive.google.com/file/d/1-KAPH9avR8pPgHJLu4kj16T8_RWFZh3a/view?usp=sharing
• libggml_core_ops_enhanced.so: https://drive.google.com/file/d/1-IFabPJVoLbQANva5scMQJfBqoG2CG2d/view?usp=sharing

There’s also a metadata entry for the model:
[{
"name": "seamless_nano_ggml",
"version": 9,
"hash": "6fd65e3c680d470d71d0b0382089e522debbc36fb37294d8c1c85cafa4a6e0cf",
"uncompressedFileSizeInBytes": 141598746,
"modelExtension": "ggml",
"compressionType": "BROTLI",
"feature": "TRANSCRIPTION"
}]

Questions:

• Can these .ggml and .so files be used with whisper.cpp?
• If so, how would they be integrated?
• Or is there another recommended tool or method for using these models?

Thanks in advance for any help or insight!

[dorlanpabon]

Hi @ggerganov,
I found some .ggml and .so files from Meta’s Seamless project (apparently used for WhatsApp transcription). Do you think they could work similarly to whisper.cpp with GGML? Would appreciate any insight. Thanks!

[ggerganov]

Most like they cannot be used with whisper.cpp. I don't know how these models are used specifically.

[dorlanpabon]

Most like they cannot be used with whisper.cpp. I don't know how these models are used specifically.

Thank you very much, very kind. I'll keep trying to see if I can transcribe a file with this model. I think it's interesting. Based on what I understand, you have to generate a uniti.cpp, but I don't know if it should be generated specifically for the nano model, since the meta repo project only has large and medium. If you have any recommendations on where to start, I'd really appreciate it again.

[dorlanpabon]

Sorry, again, I was looking at this thread. Could I possibly try running this nano model as well as the one you ran in the middle? Please, I haven't been able to do it. ggml-org/ggml#471 (comment)

[dorlanpabon]

https://git.uibk.ac.at/informatik/iis/iis-projects/TransLoco/-/tree/cc6b2e82fdc90ecb9f5b3a3cd05e1a30717c4a2c/server/t2tt-server/t2tt_server/seamless_communication/ggml/examples/python