Deeplearning: Tensorflow-gpu 2.6.2 support in casa-dev singularity image needs hdf5 update

[nsouedet]

Hello,

It is now possible to use tensorflow in a casa-dev singularity image following the How to install tensorflow-gpu for BrainVISA processes.
To do so I needed to update container pyhdf5 package from version from 2.0.9 to 3.1.0 otherwise pyhdf5 was installed twice (version 2.0.9 in the container and version 3.1.0 during tensorflow-gpu installation) with incompatible hdf5 libraries. It resulted in the following error at BrainVISA starting::

Conflit de librairie HDF5 entre celle installée ici:
 
 /casa/home/.local/lib/python3.6/site-packages/h5py/../h5py.libs/

/usr/lib/x86_64-linux-gnu/hdf5/serial/libhdf5.so;/usr/lib/x86_64-linux-gnu/libpthread.so;/usr/lib/x86_64-linux-gnu/libsz.so;/usr/lib/x86_64-linux-gnu/libz.so;/usr/lib/x86_64-linux-gnu/libdl.so;/usr/lib/x86_64-linux-gnu/libm.so (found version "1.10.0.1") found components:  C HL

/casa/home/.local/lib/python3.6/site-packages/h5py/__init__.py:39: UserWarning: h5py is running against HDF5 1.10.0 when it was built against 1.12.0, this may cause problems
  '{0}.{1}.{2}'.format(*version.hdf5_built_version_tuple)
Warning! ***HDF5 library version mismatched error***
The HDF5 header files used to compile this application do not match
the version used by the HDF5 library to which this application is linked.
Data corruption or segmentation faults may occur if the application continues.
This can happen when an application was compiled by one version of HDF5 but
linked with a different version of static or shared HDF5 library.
You should recompile the application or check your shared library related
settings such as 'LD_LIBRARY_PATH'.
You can, at your own risk, disable this warning by setting the environment
variable 'HDF5_DISABLE_VERSION_CHECK' to a value of '1'.
Setting it to 2 or higher will suppress the warning messages totally.
Headers are 1.12.0, library is 1.10.0
            SUMMARY OF THE HDF5 CONFIGURATION
            =================================

General Information:
-------------------
                   HDF5 Version: 1.10.0-patch1
                  Configured on: Sun Aug 13 23:12:31 UTC 2017
                  Configured by: buildd@lgw01-21
                    Host system: x86_64-pc-linux-gnu
              Uname information: Linux lgw01-21 4.4.0-91-generic #114-Ubuntu SMP Tue Aug 8 11:56:56 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
                       Byte sex: little-endian
             Installation point: /usr
                    Flavor name: serial

Compiling Options:
------------------
                     Build Mode: production
              Debugging Symbols: no
                        Asserts: no
                      Profiling: no
             Optimization Level: high

Linking Options:
----------------
                      Libraries: static, shared
  Statically Linked Executables: 
                        LDFLAGS: -Wl,-Bsymbolic-functions -Wl,-z,relro
                     H5_LDFLAGS: -Wl,--version-script,$(top_srcdir)/debian/map_serial.ver
                     AM_LDFLAGS: 
                Extra libraries: -lpthread -lsz -lz -ldl -lm 
                       Archiver: ar
                         Ranlib: x86_64-linux-gnu-ranlib

Languages:
----------
                              C: yes
                     C Compiler: /usr/bin/gcc
                       CPPFLAGS: -Wdate-time -D_FORTIFY_SOURCE=2
                    H5_CPPFLAGS: -D_GNU_SOURCE -D_POSIX_C_SOURCE=200112L   -DNDEBUG -UH5_DEBUG_API
                    AM_CPPFLAGS: 
                        C Flags: -g -O2 -fdebug-prefix-map=/build/hdf5-X9JKIg/hdf5-1.10.0-patch1+docs=. -fstack-protector-strong -Wformat -Werror=format-security
                     H5 C Flags:   -std=c99 -pedantic -Wall -W -Wundef -Wshadow -Wpointer-arith -Wbad-function-cast -Wcast-qual -Wcast-align -Wwrite-strings -Wconversion -Waggregate-return -Wstrict-prototypes -Wmissing-prototypes -Wmissing-declarations -Wredundant-decls -Wnested-externs -Winline -finline-functions -s -O
                     AM C Flags: 
               Shared C Library: yes
               Static C Library: yes


                        Fortran: yes
               Fortran Compiler: /usr/bin/gfortran
                  Fortran Flags: -g -O2 -fdebug-prefix-map=/build/hdf5-X9JKIg/hdf5-1.10.0-patch1+docs=. -fstack-protector-strong
               H5 Fortran Flags:  -pedantic -Wall -Wextra -Wunderflow -Wimplicit-interface -Wsurprising -Wno-c-binding-type  -s -O2
               AM Fortran Flags: 
         Shared Fortran Library: yes
         Static Fortran Library: yes

                            C++: yes
                   C++ Compiler: /usr/bin/g++
                      C++ Flags: -g -O2 -fdebug-prefix-map=/build/hdf5-X9JKIg/hdf5-1.10.0-patch1+docs=. -fstack-protector-strong -Wformat -Werror=format-security
                   H5 C++ Flags:   -pedantic -Wall -W -Wundef -Wshadow -Wpointer-arith -Wcast-qual -Wcast-align -Wwrite-strings -Wconversion -Waggregate-return -Wredundant-decls -Winline -Wsign-promo -Woverloaded-virtual -Wold-style-cast -Weffc++ -Wreorder -Wnon-virtual-dtor -Wctor-dtor-privacy -Wabi -finline-functions -s -O
                   AM C++ Flags: 
             Shared C++ Library: yes
             Static C++ Library: yes

                           Java: yes
                  Java Compiler: /usr/bin/java (openjdk 1.8.0_141)


Features:
---------
                  Parallel HDF5: no
             High-level library: yes
                   Threadsafety: yes
            Default API mapping: v18
 With deprecated public symbols: yes
         I/O filters (external): deflate(zlib),szip(encoder)
                            MPE: no
                     Direct VFD: no
                        dmalloc: no
 Packages w/ extra debug output: none
                    API tracing: no
           Using memory checker: no
Memory allocation sanity checks: no
            Metadata trace file: no
         Function stack tracing: no
      Strict file format checks: no
   Optimization instrumentation: no
Bye...
Aborted (core dumped)

Could you confirm that I can update the install script with 3.1.0 version of pyhdf5 and that there is no known incompatibilities, please ?

[denisri]

I don't really know how the hdf5 lib works. We are not using it directly, but it is used in netcdf and libminc. Thus the right test I think is to check if .mnc volume files can still be read and written (the pyaims_io test should be fine).

[nsouedet]

Ok, thanks Denis, I will run pyaims_io test before commit.

[nsouedet]

Ok, soma and aims tests passed. So I commit the change.