Incomplete list of dependencies

[ivyleavedtoadflax]

Hi @Giovanni1085 many thanks for publishing this code, and your very useful paper.

unfortunately I'm having several issues getting it run on my machine. I suspect much of it is caused by not having a complete list of dependencies. Would you consider adding a more comprehensive list in a requirements.txt?

I'll document the issues I'm having in some other issues.

[Giovanni1085]

Hello @ivyleavedtoadflax, thank you for your interest and for testing the code out. Requirements are specified in each sub-folder separately. I agree it would be better to have a single requirements.txt file: would you consider adding it via a pull request?

[ivyleavedtoadflax]

Hi @Giovanni1085 I will happily add a requirements, but I could not get the software to work without one at present. I'd quite like to try to model, so will have a play with it and see if I can get it working with more contemporary versions.

[ivyleavedtoadflax]

Also @Giovanni1085, I see you're at the Turing. Do let me know if you fancy grabbing a coffee at some point when you're in London.

[Giovanni1085]

Unfortunately some of the packages we use do not install very straightforwardly. The versions of the packages we use are listed in each folder of the repository, see:

• crf_baseline
• keras
• tensor flow

You can try the following:

• Create a new python environment, e.g. by conda create --name some_name python=3.5 (and activate).
• pip install git+https://www.github.com/keras-team/keras-contrib.git (this might not be the right version though: please check if it still works).
• pip install sklearn-crfsuite
• pip install -r requirements.txt where requirements.txt contains the following:
  • tensorflow==1.4.0
  • tensorflow-tensorboard==0.1.8
  • numpy==1.13.3
  • scikit-learn==0.19.1
  • keras==2.1.1

Please let me know if this works. Regarding meeting in person, please write me an email.

[ivyleavedtoadflax]

Thanks, I did try this with pip, but I needed to download the tars for scikit-learn==0.19.1, perhaps tf too. I don't normally use conda, but I'll give it a go when I get a chance. This thread might be useful if you end up trying this model in the next couple of weeks @aCampello @lizgzil @nsorros.

[Giovanni1085]

Thanks, I did try this with pip, but I needed to download the tars for scikit-learn==0.19.1, perhaps tf too. I don't normally use conda, but I'll give it a go when I get a chance. This thread might be useful if you end up trying this model in the next couple of weeks @aCampello @lizgzil @nsorros.

@ivyleavedtoadflax were you ultimately successful in running the code or not yet? If not, please let me know if I may help, or I'll close the issue and consider it solved. Thanks.

[ivyleavedtoadflax]

Hi @Giovanni1085, I'm on holiday at the moment, bit will try again next week. Thanks 👍

[ivyleavedtoadflax]

Please let me know if this works. Regarding meeting in person, please write me an email.

Unfortunately, your instructions failed on pip install -r requirements.txt:

  ERROR: Could not find a version that satisfies the requirement tensorflow==1.4.0 (from -r requirements.txt (line 1)) (from versions: 1.13.0rc1, 1.13.0rc2, 1.13.1, 1.13.2, 1.14.0rc0, 1.14.0rc1, 1.14.0, 2.0.0a0, 2.0.0b0, 2.0.0b1)
ERROR: No matching distribution found for tensorflow==1.4.0 (from -r requirements.txt (line 1)

Attempting to install from the archive with pip install https://github.com/tensorflow/tensorflow/archive/v1.4.0.tar.gz also fails with:

    ERROR: Complete output from command python setup.py egg_info:
    ERROR: Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/home/matthew/.pyenv/versions/3.7.0/lib/python3.7/tokenize.py", line 447, in open
        buffer = _builtin_open(filename, 'rb')
    FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pip-req-build-djvw8vxg/setup.py'
    ----------------------------------------
ERROR: Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-req-build-djvw8vxg/

I've experimented with a few things:

# Install old version of python with pyenv (need args or it will fail)

CFLAGS=-I/usr/include/openssl LDFLAGS=-L/usr/lib pyenv install -v 3.5.3
pyenv local 3.5.3
virtualenv --python python3 build/virtualenv
source ./build/virtualenv/bin/activate

# May help solve tensorflow install problems

pip install --upgrade pip setuptools

pip install git+https://www.github.com/keras-team/keras-contrib.git

pip install sklearn-crfsuite
pip install numpy==1.13.1
pip install scikit-learn==0.19.1
pip install keras==2.1.1

pip install https://github.com/tensorflow/tensorflow/archive/v1.4.0.tar.gz

This gives:

     command: /home/matthew/Documents/wellcome/LinkedBooksDeepReferenceParsing/build/virtualenv/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-ou8glumg/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-ou8glumg/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base pip-egg-info
         cwd: /tmp/pip-req-build-ou8glumg/
    Complete output (5 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/home/matthew/Documents/wellcome/LinkedBooksDeepReferenceParsing/build/virtualenv/lib/python3.5/tokenize.py", line 454, in open
        buffer = _builtin_open(filename, 'rb')
    FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pip-req-build-ou8glumg/setup.py'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

I also tried with docker to rule out local issues with the following Dockerfile:

FROM python:3.5.3

RUN pip install --upgrade pip setuptools
RUN pip install git+https://www.github.com/keras-team/keras-contrib.git
RUN pip install sklearn-crfsuite
RUN pip install numpy==1.13.1
RUN pip install scikit-learn==0.19.1
RUN pip install keras==2.1.1
RUN pip install https://github.com/tensorflow/tensorflow/archive/v1.4.0.tar.gz

ENTRYPOINT ["/bin/bash"]

This also fails with a similar error:

     command: /usr/local/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-nfir8t3c/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-nfir8t3c/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base pip-egg-info
         cwd: /tmp/pip-req-build-nfir8t3c/
    Complete output (5 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/usr/local/lib/python3.5/tokenize.py", line 454, in open
        buffer = _builtin_open(filename, 'rb')
    FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pip-req-build-nfir8t3c/setup.py'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

I also tried various other versions of tensorflow from 1.0.0 to 1.14.0. All fail to install.
I will continue to experiment...

[ivyleavedtoadflax]

So I've had some success running keras/main_ThreeTasks.py using the following dependecies in python 3.7.0, but i needed to change the lines described in #4.

absl-py==0.7.1
astor==0.8.0
bleach==1.5.0
cycler==0.10.0
gast==0.2.2
google-pasta==0.1.7
grpcio==1.22.0
h5py==2.9.0
html5lib==0.9999999
joblib==0.13.2
Keras==2.2.4
Keras-Applications==1.0.8
keras-contrib==2.0.8
Keras-Preprocessing==1.1.0
kiwisolver==1.1.0
Markdown==3.1.1
matplotlib==3.1.1
numpy==1.17.0
protobuf==3.9.1
pyparsing==2.4.2
python-crfsuite==0.9.6
python-dateutil==2.8.0
PyYAML==5.1.2
scikit-learn==0.21.3
scipy==1.3.0
six==1.12.0
sklearn-crfsuite==0.3.6
tabulate==0.8.3
tensorboard==1.14.0
tensorflow==1.14.0
tensorflow-estimator==1.14.0
tensorflow-tensorboard==1.5.1
termcolor==1.1.0
tqdm==4.32.2
Werkzeug==0.15.5
wrapt==1.11.2

@Giovanni1085 there's a few more things that will need adjusting, but would you like me submit a PR for usin gthe model with python 3.7.0?

[Giovanni1085]

@ivyleavedtoadflax excellent. Yes please, once you're happy with the results, feel welcome to add a how to section in the relevant README file (e.g., keras/README.md in this case) as a PR.

[ivyleavedtoadflax]

Great, will do 👍

[ivyleavedtoadflax]

Done in #7 . I'm going to continue to experiment with the other models, so probably push a fix for the CRF and multitask models soon too. Thanks for all your work on getting this out there @Giovanni1085 💯