How to escape empty line?

[fuzihaofzh]

I have a text file with some empty lines inside. If I run build_vocab, it will crash since example have no text for empty lines. How can I deal with this situation? I foud that there is no function in TabularDataset that can handle.
Thanks

[mttk]

Could you please paste the error trace?

[fuzihaofzh]

Hi, @mttk ,
This is the code, input text and trace. I just want to escape the empty, can I add some preprocessing function to it?
Thanks for your help.

code:

TEXT = Field(tokenize = tokenizer, fix_length=70, use_vocab = True)
text_data = data.TabularDataset(path = './untitled.txt', format='tsv', fields = [('text', TEXT)])
TEXT.build_vocab(text_data)

Input:

I love apple.

He also likes it.

We are happy.

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-51-ff76370ce4df> in <module>()
     14 text_data = data.TabularDataset(path = './untitled.txt', format='tsv', fields = [('text', TEXT)])
     15 #text_data.examples = filter(lambda x: hasattr(x, 'text') and len(x.text) > 10, text_data.examples)
---> 16 TEXT.build_vocab(text_data)

/home/me/miniconda2/lib/python2.7/site-packages/torchtext/data/field.pyc in build_vocab(self, *args, **kwargs)
    245                 sources.append(arg)
    246         for data in sources:
--> 247             for x in data:
    248                 if not self.sequential:
    249                     x = [x]

/home/me/miniconda2/lib/python2.7/site-packages/torchtext/data/dataset.pyc in __getattr__(self, attr)
    151         if attr in self.fields:
    152             for x in self.examples:
--> 153                 yield getattr(x, attr)
    154 
    155     @classmethod

AttributeError: 'Example' object has no attribute 'text'

[mttk]

So the hotfix I can offer (assuredly not the best) is to replace the blank lines with some symbol (whitespace works).

I'm thinking of how to fix this but I'm not sure what the default would be. Probably setting a specialized <blank> symbol to empty lines?
What do you expect to get as text in those blank lines?

[fuzihaofzh]

I expect it ignore the empty lines directly. I think adding a <blank> is also needed for some reason. I think it can be left as an option for users. However, leaving such an API will make the function more complicated, can we add some callback functions that will preprocess the input list?

[mttk]

You can currently add a preprocessing hook to the field (argument preprocessing=) which is applied after tokenization (on the tokenized list).

However, that won't work in the case of empty lines as no data is assigned to those fields prior to constructing the vocabulary (this is why build_vocab fails). I'll fix this in the upcoming days, but need to figure out the best way to do it.

You can manually set the blank lines to <blank> and handle them with your custom preprocessing function.

Do we have any update for handling blank lines?