Topaz 2D Training -> Nan Loss + Error

[3P1L]

Dear TOPAZ Enthusiasts,

I'm trying to train a denoising model in Topaz 0.25a.

After preprocessing the 1/2 micrographs [4kx4k] using the standard parameters form the online GUI,

I get nan's for loss and loss_val. [and the model training fails]

The result is the same ± preprocessing.

Do you have any suggestions?

Thanks,

Phil.

The command for preprocessing was:

topaz preprocess PATH_TO_IMAGES/*.mrc --scale 4 --sample 1 --num-workers -1 --format mrc,png --device 0 --niters 100 --alpha 900 --beta 1 --verbose --destdir PATH_TO_PREPROCESSED_IMAGES

(done for both half-sets)

The command for training was:

topaz denoise --dir-a A_PATH --dir-b B_PATH --method noise2noise --arch unet --optim adagrad --lr 0.001 --criteria L2 --crop 800 --batch-size 4 --num-epochs 100 --device 0 --num-workers 8 --save-prefix MODEL_PATH

[tbepler]

Do you have a minimal set of micrographs you can share with us that produce the error so we can reproduce it?

[3P1L]

Hi Tristan, Sure - how many would you need? Maybe there’s something special about them that I haven’t found yet. It is my first dataset with Falcon4i 😅 Best, Philipp. Philipp Sebastian Erdmann Research Group Leader Erdmann Group, Structural Biology Research Centre Human Technopole Palazzo Italia Viale Rita Levi Montalcini, 1 20157 Milan, Italy M. +39 3452453710 E. ***@***.*** Prima di stampare questa e-mail, pensa all’ambiente. Before printing this e-mail, think of the environment Legal Disclaimer The information contained in or accompanying this email and any attachment thereto, is intended solely for the use of the stated recipient(s) and may contain information that is confidential and/or privileged. Any dissemination, distribution or copying of this email and any attachment by anyone who is not the stated recipient is strictly prohibited. If you receive this message in error, please notify the sender immediately and delete the message and any attachment from your system without retaining a copy. On 8 May 2023, at 07:51, Tristan Bepler ***@***.******@***.***>> wrote: Do you have a minimal set of micrographs you can share with us that produce the error so we can reproduce it? — Reply to this email directly, view it on GitHub<#166 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ALUQRH7DVX5PX72A2CHLMYTXFCCWFANCNFSM6AAAAAAXYMAF4A>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[tbepler]

If you can reproduce the error with just one micrograph, then that should be enough. If you have a small set (say, <10) that produce the error, we can work with that too. Ideally, the smaller the better so we know which ones to look at.