Skip to content

can not read yttm trained model #8

@fkurushin

Description

@fkurushin

HI!

I want to read previously trained bpe model using yttm python module, with your lib, this is my code snippet:

bpeModel, err := os.Open("data/yttm.model")
reader := bufio.NewReader(bpeModel)

m, err := bpe.ReadModel(reader)
if err != nil {
print(fmt.Errorf("failed to read model %v", err))
}

config := bpe.NewConfig(false, false, false)
fmt.Println(m.EncodeSentence("мама мыла раму", *config))

and I have Broken input: unexpected EOF. This error appears on this line of code

logrus.Error("Broken input: ", err)

As I understood there may be some incompatibles between yttm model dump file extension and you ReadModel method? Have I train the model with some specific parameters or load it in go with some extra parameters?

P.S. I also need to mention that I encountered an import error while loading the configuration, so I implemented the NewConfig method to resolve it.

Thanks a lot

UPD: as I understood yttm load model as a txt file of char2id and then merges, but your package loads as a binary, so they are incompatible.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions