Exact zeros in dff traces

[alexpiet]

@matchings @jeromelecoq
I've noticed something odd, and I'm not sure if its a bug, but I can't think of a reason why it might be true. Some sessions have df/f values that are exactly equal to zero. These values do not happen at the start or end of the session, and are randomly intermixed. Its extremely odd to me that the values could be exactly 0.

import numpy as np
from visual_behavior.ophys.io.convert_level_1_to_level_2 import convert_level_1_to_level_2
from visual_behavior.ophys.dataset.visual_behavior_ophys_dataset import VisualBehaviorOphysDataset
from visual_behavior.ophys.response_analysis.response_analysis import ResponseAnalysis
experiment_id = 787498309
cache_dir = r'/allen/programs/braintv/workgroups/nc-ophys/visual_behavior/visual_behavior_production_analysis'
dataset = VisualBehaviorOphysDataset(experiment_id, cache_dir=cache_dir)
dff = dataset.dff_traces
np.sum(dff == 0)
413594
float(np.sum(dff == 0))/np.prod(np.shape(dff))
0.009877768012380826
np.sum((dff[0:-1] == 0) & (dff[1:] == 0))
4170
float(np.sum((dff[0:-1] == 0) & (dff[1:] == 0)))/np.prod(np.shape(dff))
9.959112707541222e-05
0.00987*0.00987
9.741690000000001e-05

This last check demonstrates that the exact 0s are randomly intermixed

[jeromelecoq]

Could you check whether these 0 happens where the motion correction X,Y values are NaN or similar? This is stored in a dedicated files associated with motion correction. I am suspecting these are frames where the also has issues

…

On Jun 12, 2019, at 11:41 AM, Alex Piet ***@***.***> wrote: @matchings <https://github.com/matchings> @jeromelecoq <https://github.com/jeromelecoq> I've noticed something odd, and I'm not sure if its a bug, but I can't think of a reason why it might be true. Some sessions have df/f values that are exactly equal to zero. These values do not happen at the start or end of the session, and are randomly intermixed. Its extremely odd to me that the values could be exactly 0. import numpy as np from visual_behavior.ophys.io.convert_level_1_to_level_2 import convert_level_1_to_level_2 from visual_behavior.ophys.dataset.visual_behavior_ophys_dataset import VisualBehaviorOphysDataset from visual_behavior.ophys.response_analysis.response_analysis import ResponseAnalysis experiment_id = 787498309 cache_dir = r'/allen/programs/braintv/workgroups/nc-ophys/visual_behavior/visual_behavior_production_analysis' dataset = VisualBehaviorOphysDataset(experiment_id, cache_dir=cache_dir) dff = dataset.dff_traces np.sum(dff == 0) 413594 float(np.sum(dff == 0))/np.prod(np.shape(dff)) 0.009877768012380826 np.sum((dff[0:-1] == 0) & (dff[1:] == 0)) 4170 float(np.sum((dff[0:-1] == 0) & (dff[1:] == 0)))/np.prod(np.shape(dff)) 9.959112707541222e-05 0.00987*0.00987 9.741690000000001e-05 This last check demonstrates that the exact 0s are randomly intermixed — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#577?email_source=notifications&email_token=AATAHT2QCIOT46N6N5CM6MLP2E7NPA5CNFSM4HXOQQ5KYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4GZEK4AQ>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AATAHT5SOPDLUGPHZQULNA3P2E7NPANCNFSM4HXOQQ5A>.

[alexpiet]

@jeromelecoq I'm not sure if this is what you meant, but the motion correction values in the ophys dataset object do not have NaNs.

np.sum(pd.isnull(dataset.motion_correction.x_corr))
0
np.sum(pd.isnull(dataset.motion_correction.y_corr))
0

[alexpiet]

@jeromelecoq and I just talked in person, and we also checked that the x_corr and y_corr values are not exact 0s. Jerome suspects the exact 0s in df/f originate in the segmentation algorithm which at some point, to prevent numerical instabilities rounds numbers close to zero to exactly 0. If so, that is probably OK, but I just want to make sure that is the source of the exact 0s instead of something unexpected.

[nickponvert]

@alexpiet can we close this? Should we submit a ticket to technology to verify that this is the cause of this behavior?

[alexpiet]

I think we should submit a ticket to technology to verify the source of this behavior