Replies: 16 comments 5 replies
-
|
hey Ralf Thanks for trying bidspm and thanks for sending all these logs: helps with debugging. So the script you adapted this from (this one?) is run regularly as part of continuous integration, so I have good hopes that it should work even in your case. Let's see what how we can troubleshoot things. First let's make sure there is no path issue. When this runs, it should tell you what dataset is being indexed. A bit like this: I suspect that for you it should print something like: Is that correct? Also you can get more info when bidspm run by increasing the verbosity to 3 (the default is 2). So your smoothing command should be: bidspm(fmriprep_dir, output_dir, 'subject', ...
'action', 'smooth', ...
'participant_label', subject_label, ...
'task', task, ...
'space', space, ...
'fwhm', 8, ...
'versbosity', 3); |
Beta Was this translation helpful? Give feedback.
-
|
hey Remi,
exactly, I simply modified moae_fmriprep.m with my path settings,
because I want to analyze the data not in the same folder as bidspm.
When I run your modifications with verbosity level 3, I receive the
following output
```
Indexing dataset:
/server/fo2-17/data/NASAL_2018/DAGen_Bromo/BIDS/Tuebingen/moae/inputs/fmriprep
what you already suspected and waht is correct.
then I receive the following in the workspace
Warning:
Missing:
/server/fo2-17/data/NASAL_2018/DAGen_Bromo/BIDS/Tuebingen/moae/inputs/fmriprep/participants.tsv
To silence this warning, consider adding a "participants.tsv" to your dataset.
See the function: bids.util.create_participants_tsv
See also this section of the BIDS specification:
https://bids-specification.readthedocs.io/en/latest/03-modality-agnostic-files.html#participants-file
In bids.internal.error_handling (line 57)
In bids.layout>manage_tsv (line 684)
In bids.layout (line 140)
In getData (line 59)
In bidsCopyInputFolder (line 54)
In bidspm>copy (line 196)
In bidspm>smooth (line 272)
In bidspm>executeAction (line 138)
In bidspm (line 79)
Indexing subject: sub-01 [...]
Warning: This may fail if your dataset is not valid.
In bids.internal.return_file_index (line 17)
In bids.internal.return_file_info (line 20)
In bids.layout>manage_dependencies (line 715)
In bids.layout (line 205)
In getData (line 59)
In bidsCopyInputFolder (line 54)
In bidspm>copy (line 196)
In bidspm>smooth (line 272)
In bidspm>executeAction (line 138)
In bidspm (line 79)
[16:52:56] bidspm - INFO
Reference to non-existent field 'fmriprep'.
Error MATLAB:nonExistentField occurred:
"identifier": "MATLAB:nonExistentField",
"message": "Reference to non-existent field 'fmriprep'.",
"cause": [],
"stack": [
{
"file":
"/opt_prg/bidspm/lib/bids-matlab/+bids/+internal/return_file_index.m",
"name": "return_file_index",
"line": 15
},
{
"file":
"/opt_prg/bidspm/lib/bids-matlab/+bids/+internal/return_file_info.m",
"name": "return_file_info",
"line": 20
},
{
"file": "/opt_prg/bidspm/lib/bids-matlab/+bids/layout.m",
"name": "manage_dependencies",
"line": 715
},
{
"file": "/opt_prg/bidspm/lib/bids-matlab/+bids/layout.m",
"name": "layout",
"line": 205
},
{
"file": "/opt_prg/bidspm/src/IO/getData.m",
"name": "getData",
"line": 59
},
{
"file": "/opt_prg/bidspm/src/workflows/bidsCopyInputFolder.m",
"name": "bidsCopyInputFolder",
"line": 54
},
{
"file": "/opt_prg/bidspm/demos/MoAE/../../bidspm.m",
"name": "copy",
"line": 196
},
{
"file": "/opt_prg/bidspm/demos/MoAE/../../bidspm.m",
"name": "smooth",
"line": 272
},
{
"file": "/opt_prg/bidspm/demos/MoAE/../../bidspm.m",
"name": "executeAction",
"line": 138
},
{
"file": "/opt_prg/bidspm/demos/MoAE/../../bidspm.m",
"name": "bidspm",
"line": 79
}
]
}
}
```
any ideas?
|
Beta Was this translation helpful? Give feedback.
-
|
OK does not seem to be a path issue. So let's just run the thing that triggers the error, to make sure it can be reproduced outside of the whole bidspm "context".
should contain only one line with:
It is set up to run from the same folder where you ran % add the root path of bidspm to the path and activate bidspm
addpath(fullfile(pwd, '..', '..'));
bidspm();
fmriprep_dataset = '/server/fo2-17/data/NASAL_2018/DAGen_Bromo/BIDS/Tuebingen/moae/inputs/fmriprep';
% use bids-matlab directly to index the dataset
% without indexing extra file: should work
bids.layout(fmriprep_dataset, ...
'use_schema', false, ...
'index_dependencies', false, ...
'verbose', true)
% also index extra file: should fail if the error comes from here
bids.layout(fmriprep_dataset, ...
'use_schema', false, ...
'index_dependencies', true, ...
'verbose', true) |
Beta Was this translation helpful? Give feedback.
-
|
see my replies at the corresponding line
Zitat von Remi Gau ***@***.***>:
OK does not seem to be a path issue.
So let's just run the thing that triggers the error, to make sure it
can be reproduced outside of the whole bidspm "context".
1. Can you first confirm you have the right version of bids-matlab?
`/opt_prg/bidspm/lib/bids-matlab/version.txt`
should contain only one line with: `v0.1.0dev`
yes, there is only one line with v0.1.0dev
2. Do you think you could run the following?
It is set up to run from the same folder where you ran `moae_fmriprep_ralf.m`
```matlab
% add the root path of bidspm to the path and activate bidspm
addpath(fullfile(pwd, '..', '..'));
bidspm();
fmriprep_dataset =
'/server/fo2-17/data/NASAL_2018/DAGen_Bromo/BIDS/Tuebingen/moae/inputs/fmriprep';
% use bids-matlab directly to index the dataset
% without indexing extra file: should work
bids.layout(fmriprep_dataset, ...
'use_schema', false, ...
'index_dependencies', false, ...
'verbose', true)
this works fine.
```
struct with fields:
pth:
'/server/fo2-17/data/NASAL_2018/DAGen_Bromo/BIDS/Tuebingen/moae/inputs/fmriprep'
description: [1×1 struct]
sessions: {}
participants: []
subjects: [1×1 struct]
root: [1×3 struct]
is_datalad_ds: 0
```
and when I look at the structure for the subject
```
test.subjects
ans =
struct with fields:
name: 'sub-01'
path:
'/server/fo2-17/data/NASAL_2018/DAGen_Bromo/BIDS/Tuebingen/moae/inputs/fmriprep'
session: ''
sess: ''
scans: ''
anat: [28×1 struct]
figures: [14×1 struct]
func: [13×1 struct]
```
% also index extra file: should fail if the error comes from here
bids.layout(fmriprep_dataset, ...
'use_schema', false, ...
'index_dependencies', true, ...
'verbose', true)
```
the error comes exactly here. There migth be something with the
index_dependencies issue
```
Reference to non-existent field 'fmriprep'.
Error in bids.internal.return_file_index (line 15)
file_idx = strcmp(filename,
{BIDS.subjects(sub_idx).(modality).filename}');
Error in bids.internal.return_file_info (line 20)
file_info.file_idx = bids.internal.return_file_index(BIDS, ...
Error in bids.layout>manage_dependencies (line 715)
info_src = bids.internal.return_file_info(BIDS, file_list{iFile});
Error in bids.layout (line 205)
BIDS = manage_dependencies(BIDS, index_dependencies, verbose);
```
|
Beta Was this translation helpful? Give feedback.
-
|
OK. So I have update the code to give us more information when it fails. Assuming that you are in the bidspm root folder or that you have initialized bidspm, bidspm updateThis should work if you are on a linux environment or on Mac: which I think it is a the case as you are working on a server. You should see something a bit like this in the matlab prompt (matlab is just calling git to update things): bash tools/update.sh
you are on branch main
From https://github.com/cpp-lln-lab/bidspm
* [new branch] add_qa -> tmp/add_qa
* [new branch] dev -> tmp/dev
* [new branch] main -> tmp/main
* [new branch] olf -> tmp/olf
* [new branch] v0.1.1 -> tmp/v0.1.1
From https://github.com/cpp-lln-lab/bidspm
* branch main -> FETCH_HEADThen if you re run the indexing of the fmriprep dataset with the code we used before: % add the root path of bidspm to the path and activate bidspm
addpath(fullfile(pwd, '..', '..'));
bidspm();
fmriprep_dataset = '/server/fo2-17/data/NASAL_2018/DAGen_Bromo/BIDS/Tuebingen/moae/inputs/fmriprep';
% use bids-matlab directly to index the dataset
% and index extra file: should fail if the error comes from here
bids.layout(fmriprep_dataset, ...
'use_schema', false, ...
'index_dependencies', true, ...
'verbose', true) You should then see a warning like this appear before the error: Warning: An error occurred when processing
- dataset: /home/remi/Downloads/inputs/fmriprep
- subject: sub-01
- modality: anat
- file: sub-01_desc-aparcaseg_dseg.nii.gz
This may happen if your dataset is not valid.
> In bids.internal.return_file_index (line 18)
In bids.internal.return_file_info (line 20)
In bids.layout>manage_dependencies (line 714)
In bids.layout (line 204) |
Beta Was this translation helpful? Give feedback.
-
|
unfortunately, I was not able to run the update due to permission
issues (error: can not open .git/FETCH_HEAD: no permission). One
problem is that I'm working in a hospital network behind a firewall
which makes downloading via proxy server more complicate with some
restrictions to users. However, I was able to download bidspm again
using git clone and the dev version (which sould solve the problem?).
Unfortunately, there were no additional warning messages as you
assumed. The errors were the same as before.
|
Beta Was this translation helpful? Give feedback.
-
|
Actually just clone from I probably need to remove that Sorry for the confusion. |
Beta Was this translation helpful? Give feedback.
-
|
Now I installed bidspm from the main branch and followed your
suggestions. Now there is a warning message
Warning: An error occurred when processing
- dataset:
/server/fo2-17/data/NASAL_2018/DAGen_Bromo/BIDS/Tuebingen/moae/inputs/fmriprep
- subject: sub-01
- modality: fmriprep
- file: sub-01_task-auditory_events.tsv
I would expect modality is func and the file is
sub-01_task-auditory_space-T1w_desc-preproc_bold.nii (or in MNI space).
The ..._events.tsv file is in the input folder.
|
Beta Was this translation helpful? Give feedback.
-
|
OK that narrows things down. I just checked what was the content of the fmriprep data from OSF by running the following (in the bidspm folder). mkdir -p demos/MoAE/inputs/
cd demos/MoAE/inputs/
wget https://osf.io/vufjs/download
unzip download
mv moae_fmriprep fmriprepI get the content below where there is no file: Do you know how this file could have got there? It is not a typical file from fmriprep. If suspect / hope that if this file is removed, then things should be able to run. In any case it would be better if bids-matlab threw a warning rather than an error for something like this especially when working with derivatives dataset. Will open an issue on the bids-matlab and try to push a fix soon. |
Beta Was this translation helpful? Give feedback.
-
|
you are right, seems to be my fault (some stupid copying), the output
of bids.layout is now
```
ans =
struct with fields:
pth:
'/server/fo2-17/data/NASAL_2018/DAGen_Bromo/BIDS/Tuebingen/moae/inputs/fmriprep'
description: [1×1 struct]
sessions: {}
participants: []
subjects: [1×1 struct]
root: [1×2 struct]
is_datalad_ds: 0
Now, I tried the smoothing (without copying) using
bidspm(fmriprep_dataset, deri_dataset, 'subject', ...
'action', 'smooth', ...
'participant_label', {'01'}, ...
'space',{'individual', 'IXI549Space'}, ...
'task', {}, ...
'verbosity', 3, ...
'fwhm', 6)
unfortunately, I got another error
Indexing subject: sub-01 [...]
[13:32:10] bidspm - INFO getData
WILL WORK ON SUBJECTS
- 01
[13:32:10] bidspm - INFO
No data found for this query:
- desc: {preproc, brain}
- modality: {anat}
- space: {individual, IXI549Space}
- suffix: {T1w, bold, mask}
- sub: {01}
```
in dataset:
/server/fo2-17/data/NASAL_2018/DAGen_Bromo/BIDS/Tuebingen/moae/inputs/fmriprep
in this folder is the sub-01 folder with the subfolders anat, func and
figures (as in your example).
some other indexing problem?
|
Beta Was this translation helpful? Give feedback.
-
|
random copying errors has happened to me more than I care to count: on the bright side it helped highlight some sub optimal behavior in bids-matlab, so definitely useful. The command you used won't run because there is no file in the fmriprep dataset with the space 'individual' or 'IXI549Space'. I think it will be better to mention the task: 'auditory' So maybe something more like that: bidspm(fmriprep_dataset, deri_dataset, 'subject', ...
'action', 'smooth', ...
'participant_label', {'01'}, ...
'space',{'MNI152NLin6Asym', 'T1w'}, ...
'task', {'auditory'}, ...
'verbosity', 3, ...
'fwhm', 6) |
Beta Was this translation helpful? Give feedback.
-
|
things move forward, but still not working
there might some error with copying and/or packing/unpacking first in
the anat folder and then in the func folder
```
copying
/server/fo2-17/data/NASAL_2018/DAGen_Bromo/BIDS/Tuebingen/moae/inputs/fmriprep/sub-01/anat/sub-01_space-MNI152NLin6Asym_desc-preproc_T1w.nii.gz --> /server/fo2-17/data/NASAL_2018/DAGen_Bromo/BIDS/Tuebingen/moae/outputs/derivatives/bidspm-preproc/../bidspm-preproc/sub-01/anat/sub-01_space-MNI152NLin6Asym_desc-preproc_T1w.nii.gzgzip: invalid option --
'k'
Try `gzip --help' for more information.
Warning:
copying
/server/fo2-17/data/NASAL_2018/DAGen_Bromo/BIDS/Tuebingen/moae/inputs/fmriprep/sub-01/func/sub-01_task-auditory_space-MNI152NLin6Asym_desc-brain_mask.nii.gz --> /server/fo2-17/data/NASAL_2018/DAGen_Bromo/BIDS/Tuebingen/moae/outputs/derivatives/bidspm-preproc/../bidspm-preproc/sub-01/func/sub-01_task-auditory_space-MNI152NLin6Asym_desc-brain_mask.nii.gzgzip: invalid option --
'k'
Try `gzip --help' for more information.
```
actually, the correct files and names were copied, but the nifti files
were empty (0KB) not the .json files and the .tsv files.
```
14:33:29] bidspm - INFO getData
WILL WORK ON SUBJECTS
- 01
[14:33:31] bidspm - INFO printWorkflowName
WORKFLOW: SMOOTHING DATA
[14:33:31] bidspm - INFO printProcessingSubject
PROCESSING SUBJECT No.: 1 SUBJECT LABEL : 01
[14:33:31] bidspm - DEBUG printBatchName
BUILDING JOB: smoothing anat images
[14:33:31] bidspm - DEBUG getAnatFilename
selecting anat file:
-
/server/fo2-17/data/NASAL_2018/DAGen_Bromo/BIDS/Tuebingen/moae/outputs/derivatives/bidspm-preproc/sub-01/anat/sub-01_space-MNI152NLin6Asym_desc-preproc_T1w.nii
[14:33:31] bidspm - DEBUG printBatchName
BUILDING JOB: smoothing images
[14:33:31] bidspm - DEBUG saveMatlabBatch
Saving job in:
/server/fo2-17/data/NASAL_2018/DAGen_Bromo/BIDS/Tuebingen/moae/outputs/derivatives/bidspm-preproc/jobs/auditory
```
and when running smooth the anat and not the func image was selected,
but even that file is empty
```
Error using read_hdr (line 39)
Error reading header file
"/server/fo2-17/data/NASAL_2018/DAGen_Bromo/BIDS/Tuebingen/moae/outputs/derivatives/bidspm-preproc/sub-01/anat/sub-01_space-MNI152NLin6Asym_desc-preproc_T1w.nii".
```
|
Beta Was this translation helpful? Give feedback.
-
This is the normal behavior: bidspm "smooth" will smooth all modalities (unless you explicitly tell it to work only on some modalities). The idea is that when you visualize your data, it is is more appropriate to do it on some data (usually anatomical) that was smoothed the same way that your functional was: otherwise it is easy to fool yourself thinking that you have more anatomical precision that you actually do. The empty file thing is not new to me though. Let me try to figure this one out. |
Beta Was this translation helpful? Give feedback.
-
|
The copy / gunzip / empty file error seems to be coming from more or less here: in a terminal (not the matlab prompt) can you type the following to see which version of gunzip you have. I am putting the output I get on my local machine to compare gunzip --version
gunzip (gzip) 1.10
Copyright (C) 2007, 2011-2018 Free Software Foundation, Inc.
This is free software. You may redistribute copies of it under the terms of
the GNU General Public License <https://www.gnu.org/licenses/gpl.html>.
There is NO WARRANTY, to the extent permitted by law.
Written by Paul Eggert.gunzip --help
Usage: /usr/bin/gunzip [OPTION]... [FILE]...
Uncompress FILEs (by default, in-place).
Mandatory arguments to long options are mandatory for short options too.
-c, --stdout write on standard output, keep original files unchanged
-f, --force force overwrite of output file and compress links
-k, --keep keep (don't delete) input files
-l, --list list compressed file contents
-n, --no-name do not save or restore the original name and timestamp
-N, --name save or restore the original name and timestamp
-q, --quiet suppress all warnings
-r, --recursive operate recursively on directories
-S, --suffix=SUF use suffix SUF on compressed files
--synchronous synchronous output (safer if system crashes, but slower)
-t, --test test compressed file integrity
-v, --verbose verbose mode
--help display this help and exit
--version display version information and exit
With no FILE, or when FILE is -, read standard input.
Report bugs to <[email protected]>. |
Beta Was this translation helpful? Give feedback.
-
|
we have gunzip 1.5
Copyright (C) 2007, 2011-2012 Free Software Foundation, Inc.
the Linux OS is CentOS7
|
Beta Was this translation helpful? Give feedback.
-
|
and I suspect that |
Beta Was this translation helpful? Give feedback.
-
|
thats true.
|
Beta Was this translation helpful? Give feedback.
-
|
OK that's the issue here: it may be better if you could ask for a more recent version of gunzip to be installed. From what I gather the version you have was released 10 years ago. I will make a note to add the proper warnings in place if this error is encountered. |
Beta Was this translation helpful? Give feedback.
-
|
Now I moved to a server with gzip 1.12 and finally the smoothing worked.
Now I'll move to the first level analyses and I hope not to bother you
any further ;-)
|
Beta Was this translation helpful? Give feedback.
-
|
no worries: by reporting issues like that you can help improve things for others too! |
Beta Was this translation helpful? Give feedback.
-
|
will close this discussion but repoen if you encounter more problems |
Beta Was this translation helpful? Give feedback.
-
Dear community,
I just started to test your bidspm tool for first level analyses after finished preprocessing with fmriprep. Therefore, I downloaded the MoAe.zip dataset and tried to run the smoothing. Unfortunately, the script stops with the following message
see error log file (env deleted)
error_2023-04-20T14-51_modified.log
The data tree is as follows.
MOAe_data_tree.txt.
The dir was correctly set in the m-file
moae_fmriprep_ralf.zip
It's probably a silly mistake, but I can't get any further.
Best,
Ralf
Beta Was this translation helpful? Give feedback.
All reactions