Creating "threedod" venv takes long #86
Description
Problem description
When creating the virtual environment, running the python_venv_setup.sh script takes very long time: at least several hours (did not finish overnight). The long step is "Installing requirements" (pip install -r "${SCRIPT_DIR}/requirements.txt").
Details
My system-wide Python (3.9) is not supported by the required "open3d==0.13.0". The long run time was observed in a fresh Python 3.8 env created with conda create --name arkit_scenes python=3.8 – 3.8 is the newest Python version supported by the required open3d version, but the requirements.txt may have been "frozen" in an even older Python version (3.7/3.6? is this documented somewhere?). I suspect some of the dependencies may be unavailable/incompatible with 3.8. PIP fell back to backtracking, outputted messages like
INFO: pip is looking at multiple versions of importlib-metadata to determine which version is compatible with other requirements. This could take a while.
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
After that PIP downloaded and tried many versions of each package, and eventually got unbearably slow/stuck at importlib_metadata.
Solution
I did a cleanup of requirements.txt, keeping only the the packages which are directly-imported in benchmark_scripts.
My requirements.txt:
numpy==1.20.3
open3d==0.13.0
opencv-python==4.5.1.48
plyfile==0.7.4
scipy==1.7.1
vtk==9.0.3
After deleting the unfinished venv folder from the previous attempt, the bash python_venv_setup.sh command took just a few minutes and installed apparently all that's needed for running at least the data_prepare_offline.py script. I haven't thoroughly tested the outputs or ran other scripts, but think they should work as the direct dependencies' versions are kept, unless I overlooked some... If it proves helpful to others, it might be turned into a pull request.
The final "pip freeze" in the python=3.8 environment is attached, if anyone is interested in which packages PIP chose to install today.
requirements_38.txt
It might be helpful to either document which Python version was used, or make requirements.txt more universal.
P.S. Thanks to the authors for publishing this dataset along with the processing scripts. I am looking forward to working with it!