recon-all SUBJECTS_DIR in neurocommand

[mschira]

Hi
I have installed neurocommand on a HPC system, it seems to run, when I load a freesurfer module and type >recon-all
I get the recon-all help.

Now when I try to use recon-all it tries to write into the module filesystem but finds it is read only.
i.e.
mkdir: cannot create directory '/usr/local/freesurfer/7.1.1-1/subjects/test': Read-only file system

That makes sense, normally for FS one sets a variable pointing to where the output should go:
i.e.
export SUBJECTS_DIR=/whereveryouwantyoursubjectsdir

So I set it, but it looks like the variable is not handed to the sandbox recon-all is in?
M

[stebo85]

Dear @mschira,

Yes, that looks like a bug. The environment variables should be passed inside the container, but if this error comes up then this didn't happen.

I have a look and see if I can reproduce the problem.

Thank you
Steffen

[mschira]

Sounds good.
We have been using the freesurfer_7.1.1_20211216 module and the behavior is consistent for both a)Neurodesktop via the Nectar cloud, and b) Neurocommand I installed on a HCP system. I am in very early days of trying this so I can't promise I didn't make some mistakes there.

Ideally, I would probably suggest that the variable
$SUBJECTS_DIR
should never be set to a place in /usr/local/freesurfer or any other place the container can't write.
I can imagine a useful default setting for $SUBJECTS_DIR could be something in ~/
i.e. ~/fs_subjects, though admittedly I am not 100% it is very wise to set that variable somewhere outside of the container.

I have been using freesurfer a bit on CVL@Massive and the first I do is right after loading the module is

export SUBJECTS_DIR=~/db18/fs_subjects
where db18 is where we have some space.
That works.

While talking about CVL, I checked a few things which I feel may help you.
Once one loads the module in CVL, i.e.

module load freesurfer/7.1.0
I typed
echo $SUBJECTS_DIR
and received:
/usr/local/freesurfer/7.1.0/subjects

I could change into that folder, it does contain a bunch of 'subjects' that are part of the basic freesurfer installation.
I have never needed them, but who am I to know what other people do :-)
while the directory exists and can be read one cannot write anything, so if one wants to use recon-all one has to set the variable $SUBJECTS_DIR first.

On the other hand for Neurodesktop on Nectar cloud:

ml freesurfer/7.1.1
echo $SUBJECTS_DIR does not return anything
also I can't browse to
/usr/local/freesurfer/7.1.1/subjects which is where recon-all tries to write the output to.

That is the same for all other freesurfer modules on Neurodesktop.
I suspect that is a larger problem.
Cheers
Mark

[stebo85]

Dear @mschira,

I had a look at this problem and identified the issue. The problem is caused by an update in singularity and singularity changed how they handle environment variables. In singularity versions older than 3.6 external variables overwrite internal variables. In newer versions this is not the case anymore and it has to be explicilty set.

A workaround for you for now is the following:
define a variable called export SINGULARITYENV_SUBJECTS_DIR=/home/mshira/fssubjects - this will then overwrite SUBJECTS_DIR inside freesurfer.

[stebo85]

Dear @mschira,

I documented this new behaviour now here: https://www.neurodesk.org/tutorials/structural_imaging/freesurfer/#freesurfer-example-using-module-load-eg-on-an-hpc

Could you test this on your end and let me know if this works?

Thank you so much for finding and reporting this bug :)

Cheers
Steffen

[mschira]

Hi Steffen
Not to worry, being able to point collaborators at UOW that want to do some MRI analysis to a virtual machine that is sorta all set up is a potential Godsend for me.
So far there were some hickups ...., but I am still hopeful.

So,
I've had a try on the nectar Neurodesktop (can't get to the HCP system right at this moment)
it's doing a recon-all at the moment, so that looks promising. Given that the HPC and Neurodesktop showed identical behavior, I am optimistic this will work on the HCP, too.

For clarification, it appears the two variables
$FS_LICENSE
and
$SINGULARITYENV_SUBJECTS_DIR
do make it into the singularity container. Can I asks why these variables make it into the container, while others apparently don't?
Cheers
Mark

P.S.
essentially there needed to be two things done before using the freesurfer module:
export FS_LICENSE=~ /to_where_your_license_is/license.txt
export SINGULARITYENV_SUBJECTS_DIR=~ /where_you_want_your_fsoutput

[stebo85]

the behaviour of these variables is a bit tricky:

1. If the variable is defined outside the container and it does NOT exist inside the container, then it's passed into the container (this is the case for FS_LICENSE)
2. If the variable is defined outside the container and does exist inside the container, then the variable inside the container is used (this was your problem for the SUBJECTS_DIR)
3. If the variable is prepended with SINGULARITYENV_ then it's always passed into the container and would overwrite an existing variable inside the container (This is the proposed solution

If you use freesurfer/7.3.2 I included a license and you do not have to set your own license. All older version will need the license set.

[mschira]

Thanks for clarification.
We are using 7.1.1 because we have experience with it and 7.3 is rather new.
I try to do ONE NEW THING at a time only 😇
If I had tried using 7.3 and Neurodesktop for the first time, I would not know if any problem is due to 7.3 or the container...
...
...
I'll let you know how we go.

[stebo85]

yes, good point and that's one of many reasons why we build neurodesk ... to make all tools available that one needs :) If you run into any issues with the FS_LICENSE variable you could also prepend it to be on the safe side, but so far I think it's not necessary.