Aims read slow down when readening .trm file on lustre system

[rbregand]

Hello,
I just want to highlight a slow down with the aims.read (18min) on .trm file when the file is located on the lustre system whereas the reading part is instant on local.
Aims.read on a nifti located on the lustre doesn't show any problems.

If anyone as any idea, don't hesitate to share.
Thanks

Configuration system : Mint 20.1

[sapetnioc]

.trm files are very small text files, this is very surprising. I would suggest to first check the file system access in the same condition as Aims. For instance, does a cat on the file is also slow ? Is there a difference in reading speeds with standard commands (i.e. cat or Python script) between the host and the casa_distro container ?

A very verbose debugging possibility is to use strace. It logs all the system calls of a given command. It could allow to find out the one(s) that cause this incredible delay (either finding one that is stuck or some that are called many many times).

[denisri]

I have already seen similar phenomenons, while parsing text files (sulci graphs for instance) on a network filesystem from a containerized environment a few years ago (on the neurospin cluster). I tried to investigate somewhat but could not find reliable reproducible conditions when it happens, and things had been fixed "by themselves" (filesystem update? or reboot?). It seemed to be linked to text files parsing, maybe there were network filesystem buffering problems, since we did not have such problems using large binary files. By that time the only fix I found was to copy such files locally before processing them, which is not a real solution I admit...