Feature: Vocab Features

[srush]

A couple vocab / field features:

• Would love for Fields to be pickle'able. I understand that other issues, but we really can't be creating vocab on the fly, it's much too slow.

I think something like this would work?

def _getstate(self):
    return dict(self.__dict__, stoi=dict(self.stoi))


def _setstate(self, state):
    self.__dict__.update(state)
    self.stoi = defaultdict(lambda: 0, self.stoi)


torchtext.vocab.Vocab.__getstate__ = _getstate
torchtext.vocab.Vocab.__setstate__ = _setstate

• Would be neat to support two fields sharing a vocab. For instance, if we want to have a separate field for input and output in a language model, but share the weight matrix, or share source and target vocab. We currently do this, but it breaks a lot if things like specials change. Be great to have it in the code.

def merge_vocabs(vocabs, vocab_size=None):
    """
    Merge individual vocabularies (assumed to be generated from disjoint
    documents) into a larger vocabulary.
    Args:
        vocabs: `torchtext.vocab.Vocab` vocabularies to be merged
        vocab_size: `int` the final vocabulary size. `None` for no limit.
    Return:
        `torchtext.vocab.Vocab`
    """
    merged = sum([vocab.freqs for vocab in vocabs], Counter())
    return torchtext.vocab.Vocab(merged,
                                 specials=[UNK_WORD, PAD_WORD,
                                           BOS_WORD, EOS_WORD],
                                 max_size=vocab_size)

[mttk]

Your first example works well for serializing vocabularies, but fields have problems when using lambdas for spitting, and when lambdas are moved to module functions, serialization crashes with SpaCy's tokenizer (open issue in SpaCy: explosion/spaCy#1669).

I have the code for a serialized versions of Fields & Vocabs up and running in the fork where I'm working on streaming datasets, so as soon as I finish up the alpha version of dataset streaming, this should be merged as well.

Shared vocabularies: neat idea, should be added.

[srush]

Cool. Yeah we are not using spacy or lambda and this works for us for now. Not sure how lambdas would ever work.

[jekbradbury]

Another snippet from our code 😃:

# work around torchtext making it hard to share vocabs without sharing other field properties
if args.share_embeddings:
    SRC = copy.deepcopy(SRC)
    SRC.init_token = None
    SRC.eos_token = None
    train_data.fields['src'] = SRC
    dev_data.fields['src'] = SRC

Both of your suggestions look like they can be added without breaking any functionality. Want to create a PR?

[anna-hope]

In our project, we successfully pickle Fields using dill, which can serialise lambdas. The size of the pickled objects tends to be quite large, but otherwise we haven't found other problems. However, an out of the box pickleable version of the Field would be welcome.