Unable to run selxavg3-sess freesurfer command on neurodesk

[adam-coates]

Hello,

Following on from my discussion regarding mkanalysis-sess (the FSFAST FreeSurfer workflow). I have been trying test out running a first-level analysis on neurodesk with not much luck. When trying to run selxavg3-sess inside of the FreeSurfer container with the following beforehand export FS_MCRROOT=/opt/MCR2019b/v97 I get the following error:

freesurfer-7.4.1:~/test$ selxavg3-sess -s sub-02 -analysis allimages_ses-1.sm 
--------------------------------------------------------------
selxavg3-sess logfile is /home/jovyan/test/log/selxavg3-sess-/ses-1/bold-allimages_ses-1.sm-250122090204.log
--------------------------------------------------------------
selxavg3-sess log file
/opt/freesurfer-7.4.1/fsfast/bin/selxavg3-sess
-s sub-02 -analysis allimages_ses-1.sm
selxavg3-sess 7.4.1
uid=1013(jovyan) gid=1014(jovyan) groups=1014(jovyan),65534(nobody)
/home/jovyan/test
Linux 53f426fd8c60 5.15.0-126-generic #136-Ubuntu SMP Wed Nov 6 10:38:22 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Wed Jan 22 09:02:05 UTC 2025
-------------------------------------------
1/1 /home/jovyan/test/sub-02 
Wed Jan 22 09:02:05 UTC 2025
anadir = /home/jovyan/test/sub-02//ses-1/bold/allimages_ses-1.sm
if: Expression Syntax.
if: Expression Syntax.
if: Expression Syntax.

I have tried the following

• Opening terminal, running ml matlab & ml freesurfer to see if there is some environment variables missing.
• Running selxavg3-sess inside of matlab
• Sourcing own installation of FreeSurfer inside of the matlab container

Now I am not sure how to proceed. I have additionally also tried running the command with the -bin flag. This in theory should force to use the binary, but get the same error above. To me it almost seems like a syntax problem, or that the csh inside the command is not being interpreted properly. Therefore I tried running the command inside csh but still get the same error as above.

Any help again would be appreciated!

Best wishes,
Adam

[adam-coates]

One way I think to make this work is to run it inside the matlab container and source my own version of FreeSurfer in my home directory. However, inside the matlab container there is currently no csh interpreter.

matlab-2022a:~$ export FREESURFER_HOME=/home/jovyan/freesurfer/
matlab-2022a:~$ source $FREESURFER_HOME/SetUpFreeSurfer.sh 
-------- freesurfer-linux-centos8_x86_64-7.4.1-20230613-7eb8460 --------
Setting up environment for FreeSurfer/FS-FAST (and FSL)
WARNING: /home/jovyan/freesurfer-subjects-dir does not exist
FREESURFER_HOME   /home/jovyan/freesurfer/
FSFAST_HOME       /home/jovyan/freesurfer//fsfast
FSF_OUTPUT_FORMAT nii.gz
SUBJECTS_DIR      /home/jovyan/freesurfer-subjects-dir
MNI_DIR           /home/jovyan/freesurfer//mni
matlab-2022a:~$ cd test
matlab-2022a:~/test$ selxavg3-sess -s sub-02 -analysis allimages_ses-1.sm 
bash: /home/jovyan/freesurfer//fsfast/bin/selxavg3-sess: /bin/csh: bad interpreter: No such file or directory

[stebo85]

Dear @adam-coates,

We have a couple of ideas to tackle this.

1. We could add octave to the freesurfer image - this should work and then wouldn't require a matlab license.
2. We can add csh to the matlab image, but it might need a few more dependencies.

Let's see what gets us closer to a solution.

[stebo85]

@adam-coates - we have now matlab 2022b with csh installed inside. Can you test if that gets you further?

[stebo85]

I also just fixed this in freesurfer 8.0.0: The image is currently building here: https://github.com/NeuroDesk/neurocontainers/actions/runs/12944747792

[stebo85]

this also fails earlier for the betas:

set beta = `stem2fname $anadir/beta`
stem2fname /home/jovyan/test/sub-002/ses-1/bold/allimages_ses-1.sm0/beta
if: Expression Syntax.

[adam-coates]

Okay this is not normal behaviour. The files created by selxavg3-sess, beta.nii.gz and allimages_vs_baseline/ces.nii.gz are not being created at all. I believe that there might be an error in the variable that should be created called $FSF_OUTPUT_FORMATS. Link to stem2fname code: https://surfer.nmr.mgh.harvard.edu/pub/dist/freesurfer/dev_binaries/centos7_x86_64/stem2fname

In stem2fname it uses the variable $FSF_OUTPUT_FORMATS. Currently in FreeSurfer/8.0.0 FSF_OUTPUT_FORMATS is the following:

freesurfer-8.0.0:~$ echo $FSF_OUTPUT_FORMAT 
nii.gz# mni env requirements

[adam-coates]

Here I have now exported the following variable:

So the "#" is erroneous.

export FSF_OUTPUT_FORMAT=.nii.gz

Now I get a new error which actually makes some progress:

freesurfer-8.0.0:~/test$ selxavg3-sess -s sub-002 -analysis allimages_ses-1.sm0 
--------------------------------------------------------------
selxavg3-sess logfile is /home/jovyan/test/log/selxavg3-sess-ses-1/bold-allimages_ses-1.sm0-250130102805.sub-002.log
--------------------------------------------------------------
selxavg3-sess log file
/opt/freesurfer-8.0.0/fsfast/bin/selxavg3-sess
-s sub-002 -analysis allimages_ses-1.sm0
selxavg3-sess 8.0.0-beta
uid=1013(jovyan) gid=1014(jovyan) groups=1014(jovyan),65534(nogroup)
/home/jovyan/test
Linux 1e8621d4dea8 5.15.0-126-generic #136-Ubuntu SMP Wed Nov 6 10:38:22 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Thu Jan 30 10:28:05 AM UTC 2025
-------------------------------------------
1/1 /home/jovyan/test/sub-002 
Thu Jan 30 10:28:05 AM UTC 2025
anadir = /home/jovyan/test/sub-002/ses-1/bold/allimages_ses-1.sm0
DoGLMFit = 1
DoContrasts = 1
UpdateNeeded = 1
    sub-002 Update needed
/home/jovyan/test/sub-002/ses-1/bold/allimages_ses-1.sm0/sxa3.config.txt
MLF is /tmp/selxavg3-sess-232832.m
------------------------------------------
% Thu Jan 30 10:28:05 AM UTC 2025  
% /home/jovyan/test  
%  allimages_ses-1.sm0  
which fast_selxavg3
which fast_ldanaflac
which MRIread
  
  
fast_selxavg3b('/home/jovyan/test/sub-002/ses-1/bold/allimages_ses-1.sm0/sxa3.config.txt')
if(~fast_fileexists('/tmp/selxavg3-sess-232832.ok')); quit; end
%--------------------------------
  
  
quit;
------------------------------------------
------------------------------------------
------- octave output --------------------
LD_LIBRARY_PATH :/usr/lib/x86_64-linux-gnu:/opt/MCR2014b/v84/runtime/glnxa64:/opt/MCR2014b/v84/bin/glnxa64:/opt/MCR2014b/v84/sys/os/glnxa64:/opt/MCR2014b/v84/extern/bin/glnxa64:/usr/lib/x86_64-linux-gnu:/opt/MCR2019b/v97/runtime/glnxa64:/opt/MCR2019b/v97/bin/glnxa64:/opt/MCR2019b/v97/sys/os/glnxa64:/opt/MCR2019b/v97/extern/bin/glnxa64:/usr/local/freesurfer/8.0.0-1/lib/qt/lib/:/usr/lib64/:/opt/freesurfer-8.0.0/MCRv97/runtime/glnxa64:/opt/freesurfer-8.0.0/MCRv97/bin/glnxa64:/opt/freesurfer-8.0.0/MCRv97/sys/os/glnxa64:/opt/freesurfer-8.0.0/MCRv97/sys/opengl/lib/glnxa64:/opt/freesurfer-8.0.0/MCRv97/extern/bin/glnxa64:/.singularity.d/libs:/usr/lib/x86_64-linux-gnu/octave/6.4.0
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-jovyan'
error: 'fast_selxavg3b' undefined near line 1, column 1
------------------------------------------
ERROR: fast_selxavg3() failed\n

[adam-coates]

The incorrect $FSF_OUTPUT_FORMATS Will also likely explain the error I was having when using MATLAB2022b container and exporting my own local FreeSurfer version.

The "#" comment is not being interpreted as such in csh. I think that is the reason behind why it kind of worked when I unzipped the files (perhaps .nii is default). However there is just an issue of read and write permissions inside the container to the /scratch directory.

[adam-coates]

To summarise the points so far:

1. selxavg3-sess fails, initially due to not having matlab/octave or a csh interpreter available in the freesurfer container
2. Including octave in the container and csh still lead to failures
3. the variable FSF_OUTPUT_FORMAT that is automatically created (presumably automatic) is not being interpreted correctly in stem2fname
   • currently in the freesurfer container: $FSF_OUTPUT_FORMAT = nii.gz# mni env requirements
4. setting the variable as such: export FSF_OUTPUT_FORMAT=.nii.gz removes errors regarding syntax (if: Expression syntax)
5. Now it seems that just fast_selxavg3b is undefined in the $LD_LIBRARY_PATH variable. (this is only true for freesurfer/8.0.0)

[adam-coates]

Okay I just tested using the matlab 2022b image. Which csh shows that it is there, which is good!! But it appears that there are some dependencies that are missing.

/home/jovyan/bsd-csh: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /home/jovyan/bsd-csh)
/home/jovyan/bsd-csh: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /home/jovyan/bsd-csh)

Thanks so much for your help with this!

[adam-coates]

Dear @stebo85

Okay I think the reason I mentioned this error is that I had a locally installed csh binary that I was trying out. Then in the selxavg3-sess script my shebang was pointing to that. After removing all that and for safe measure spawning a new neurodesk server.

which csh shows its in /usr/bin/csh

When I try the following:

1. start matlab in the neurodesk menu
2. run export FREESURFER_HOME=/home/jovyan/freesurfer
3. source $FREESURFER_HOME/SetUpFreesurfer.sh

Then if I run the command used to run the analysis: selxavg3-sess -s sub-002 -analysis allimages_ses-1.sm0 I get a new error that I haven't seen:

ERROR: /bin/bash: /scratch/tpf52fcc7e_0307_4928_9c77_d8d1f9bf55fd.load_nifti.m.nii: Read-only file system

ERROR: loading /home/jovyan/test/sub-002/ses-1/bold/001/fmcT1.nii.gz as analyze
ERROR: attempting to read /home/jovyan/test/sub-002/ses-1/bold/001/fmcT1
------------------------------------------
ERROR: fast_selxavg3() failed\n

This shows that at least now, the csh bad interpreter problem is solved but that the script seems to run into permission issues. Of course this is not the optimal way to run selxavg3-sess and so I think abandoning the idea of trying to make it run inside of a matlab container but sourcing a locally source freesurfer installation is probably for the best.

[stebo85]

good to hear the csh problem is solved. The error you get points to a problem with mounting the filesystem. Can you check if you have write permission in /scratch/ ? What filesystem have you mounted there?

[adam-coates]

Here I haven't mounted any file system. Just now I tried running chmod 775 /scratch and I got the same error.

I noticed though that the error looks like it is with creating a temporary nifti file. My files are .nii.gz

If I unzip the nifitis first I think this should run without errors.

I tried unzipping one nifit and the error is now with run 2 and not run 1 indicated by 001 or 002.

ERROR: /bin/bash: /scratch/tp9ef120c0_ca64_4e65_a080_95485b8dec21.load_nifti.m.nii: Read-only file system

ERROR: loading /home/jovyan/test/sub-002/ses-1/bold/002/fmcT1.nii.gz as analyze
ERROR: attempting to read /home/jovyan/test/sub-002/ses-1/bold/002/fmcT1
------------------------------------------
ERROR: fast_selxavg3() failed\n

[adam-coates]

I unzip all the files but ran into another problem again because some files are written as temporary files during selxavg3-sess. And I don't have permissions /scatch inside of the matlab container.

This has so be done when clicking on matlab from the neurodesk menu otherwise csh is not found. So If I did ml matlab/2022b I get the following error:

(base) jovyan@neurodesktop-:~/test$ selxavg3-sess -s sub-002 -analysis allimages_ses-1.sm0       
bash: /home/jovyan/freesurfer//fsfast/bin/selxavg3-sess: /bin/csh: bad interpreter: No such file or directory

[stebo85]

yes, that makes sense. Otherwise you are not in the matlab container context

[adam-coates]

I finally managed to get it working turns out we can use mcr and no need for octave or matlab at all!

I am using FreeSurfer/7.4.1.

I set export FSF_OUTPUT_FORMAT=.nii (Here I do not use .nii.gz, because currently there needs to be write permissions to /scratch inside the container which there isn't currently it is only read-only. Therefore, MATLAB can write in .nii without being zipped up to the users output directory without needing /scratch.

Use the command selxavg3-sess -s sub-002 -analysis allimages_ses-1.sm0 -mcr 1 with the mcr flag.

The whole main reason why this command didn't work is because of how FSF_OUT_FORMAT was being set to .nii.gz# mni env requirements. Therefore, we do not need octave or MATLAB in a freesurfer container since using mcr will work just fine (I cannot get this to run in FreeSurfer/8.0.0 because I wasn't able to work out which variables were needed to unset).

I suggest that FreeSurfer containers are built with a fix where FSF_OUTPUT_FORMAT is defined, either in $FREESURFER_HOME/FreeSurferEnv.sh or whereever it is defined and the possibility to make /scratch a writeable directory. Otherwise, we cannot use the extension .nii.gz.

[stebo85]

That's amazing! Great find!!!

I can set FSF_OUTPUT_FORMAT=.nii easily in the containers.

I can also make /scratch writable. Before I do this could you try if this fixes the problem even without setting FSF_OUTPUT_FORMAT?

to test run:

singularity shell --bind /tmp:/scratch /cvmfs/neurodesk.ardc.edu.au/containers/freesurfer_8.0.0_20250127/freesurfer_8.0.0_20250127.simg/

[adam-coates]

Great, yeah of course this makes sense to bind /tmp to scratch which should make it read and writeable. Here I still had to set FSF_OUTPUT_FORMAT=.nii.gz, otherwise I got the error if: expression syntax again. I think it would be better to stick with .nii.gz for the smaller file size in general. So setting containers to .nii.gz would be better.

I tried using freesurfer8.0.0 but I got this error:

LD_LIBRARY_PATH :/usr/lib/x86_64-linux-gnu:/opt/MCR2014b/v84/runtime/glnxa64:/opt/MCR2014b/v84/bin/glnxa64:/opt/MCR2014b/v84/sys/os/glnxa64:/opt/MCR2014b/v84/extern/bin/glnxa64:/usr/lib/x86_64-linux-gnu:/opt/MCR2019b/v97/runtime/glnxa64:/opt/MCR2019b/v97/bin/glnxa64:/opt/MCR2019b/v97/sys/os/glnxa64:/opt/MCR2019b/v97/extern/bin/glnxa64:/usr/local/freesurfer/8.0.0-1/lib/qt/lib/:/usr/lib64/:/opt/freesurfer-8.0.0/MCRv97/runtime/glnxa64:/opt/freesurfer-8.0.0/MCRv97/bin/glnxa64:/opt/freesurfer-8.0.0/MCRv97/sys/os/glnxa64:/opt/freesurfer-8.0.0/MCRv97/sys/opengl/lib/glnxa64:/opt/freesurfer-8.0.0/MCRv97/extern/bin/glnxa64:/.singularity.d/libs:/usr/lib/x86_64-linux-gnu/octave/6.4.0:/lib/x86_64-linux-gnu:1/runtime/glnxa64:1/bin/glnxa64:1/sys/os/glnxa64
-------------------------------------------
1/1 /home/jovyan/test/sub-002 
Fri Jan 31 10:38:15 AM UTC 2025
anadir = /home/jovyan/test/sub-002/ses-1/bold/allimages_ses-1.sm0
DoGLMFit = 1
DoContrasts = 1
UpdateNeeded = 1
    sub-002 Update needed
/home/jovyan/test/sub-002/ses-1/bold/allimages_ses-1.sm0/sxa3.config.txt
fast_selxavg3b.glnxa64 /home/jovyan/test/sub-002/ses-1/bold/allimages_ses-1.sm0/sxa3.config.txt
fast_selxavg3b.glnxa64: symbol lookup error: fast_selxavg3b.glnxa64: undefined symbol: _Z14standaloneMainRKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS5_EE
Apptainer>

However, with FreeSurfer 7.4.1 it completed successfully by binding /tmp to scratch and setting FSF_OUTPUT_FORMAT=.nii.gz. Therefore, I think there might be something specific to the freesurfer 8.0.0 container, whether having set to use octave instead of mcr (I suspect that the resason for this undefined symbol error).

Thanks so much for your help with this, it is very appreciated!! I think once/ if I have a bit of downtime I'd be happy to contribute to documentation about FreeSurfer FSFAST analysis but this won't be until a few months or so.

[adam-coates]

Okay, so the error is a result that /scratch still is being interpreted as a read-only file system when I open FreeSurfer 8.0.0 from within the menu entry. The only way to mitigate this is to run the container and bind /tmp to /scratch. I can confirm that hovering the mouse over the version is 20250203.

[adam-coates]

And I notice a bit of a mistake from my side. I think that FSF_OUTPUT_FORMAT should really be set to nii.gz without the first period. Otherwise output files contain a double period e.g. beta..nii.gz. That was my mistake.

I confirm that using the container and binding /tmp to scratch without needing to export any variables for the freesurfer_8.0.0_20250203 container worked fine (although output files are with double periods).

singularity shell --bind /tmp:/scratch /cvmfs/neurodesk.ardc.edu.au/containers/freesurfer_8.0.0_20250203/freesurfer_8.0.0_20250203.simg/

Therefore the only thing that still needs some tweaking is the ability for /scratch to be writeable and to remove the first period in the output format so export FSF_OUTPUT_FORMAT=nii.gz.

[stebo85]

great. Both things are fixed now in the version

Can you confirm if it's working now for you as well?

[adam-coates]

Hi @stebo85

Apologies for the late reply. I can confirm that it works now as intended with freesurfer 8.0.0 20250210.

Thanks so much for all your help with this it is very appreciated!! 🎉

[stebo85]

wonderful to hear :) Thank you for the great help in fixing this for everyone :)