-
Notifications
You must be signed in to change notification settings - Fork 5
Description
HI!
I want to read previously trained bpe model using yttm python module, with your lib, this is my code snippet:
bpeModel, err := os.Open("data/yttm.model")
reader := bufio.NewReader(bpeModel)
m, err := bpe.ReadModel(reader)
if err != nil {
print(fmt.Errorf("failed to read model %v", err))
}
config := bpe.NewConfig(false, false, false)
fmt.Println(m.EncodeSentence("мама мыла раму", *config))
and I have Broken input: unexpected EOF. This error appears on this line of code
Line 164 in 2feb9cc
logrus.Error("Broken input: ", err) |
As I understood there may be some incompatibles between yttm model dump file extension and you ReadModel
method? Have I train the model with some specific parameters or load it in go with some extra parameters?
P.S. I also need to mention that I encountered an import error while loading the configuration, so I implemented the NewConfig method to resolve it.
Thanks a lot
UPD: as I understood yttm load model as a txt file of char2id and then merges, but your package loads as a binary, so they are incompatible.