Load data from pandas

[jacobcvt12]

It would be nice to load data from pandas versus having to read from a disk file. I normally perform some data manipulation in pandas, and then to use torchtext, have to write the dataframe to disk. Ideally, I could skip the writing to disk.

In this stackoverflow, there's a pretty simple class added to do just this. Would it be possible to merge this contribution into torchtext?

[anna-hope]

I can work on this, unless someone else wants to.

[belerico]

I could get my hands on it if it's still open

[zhangguanheng66]

@belerico please help with a PR for this issue. Thanks.

[anna-hope]

Hi, sorry I disappeared. I have a prototype implementation for this. I can share it.

[anna-hope]

https://gist.github.com/notnami/3c4d636f2b79e206b26acfe349f2657a This is what I have been using. If this looks good, and if @belerico doesn't mind, I can make a PR with this. I can probably clean some things up, and get rid of tqdm

[zhangguanheng66]

@notnami @belerico Thanks for working on it. Please create a PR with proper unit tests and docs.

[anna-hope]

Ok, I'll try to submit a PR by the end of the week.

[belerico]

No problem. Now I'm a little bit busy with my work and studies, but I can make unit test or doc

[belerico]

Someone is already working on it?

[zhangguanheng66]

No, I don't think so.

[dkajtoch]

Is this thread dead or anyone is really working on that feature?

[zhangguanheng66]

@dkajtoch I don't think there are some people working on it now.

[dkajtoch]

I will see what I can do

[cpuhrsch]

If you use this dataset in context of DataLoader each worker gets a copy. Are there any ramifications of doing this for a pandas dataframe?

On the topic of having to write dataframes to disk, doesn't Pandas support an in-memory 0-cost conversion to something like a list of the data entries in their native data types (e.g. python strings or integers)?