Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add data preparation functions for the Kr map computation #865

Open
wants to merge 53 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
53 commits
Select commit Hold shift + click to select a range
655182a
Add Strickness symbol
bpalmeiro Mar 1, 2024
10affe1
Add ValueOutOfRange exception
bpalmeiro Mar 1, 2024
528ad25
Add check_if_values_in_interval function
bpalmeiro Mar 1, 2024
b64f9e5
Add tests for check_if_values_in_interval function
bpalmeiro Mar 1, 2024
3075a1a
Add type_of_signal enumerate
bpalmeiro Mar 1, 2024
cd737bc
Add apply_geo_correction function
bpalmeiro Mar 5, 2024
e1538aa
Add test for apply_geo_correction function
bpalmeiro Mar 5, 2024
00a3b14
Add seleccion_nS_mask_and_checking function
bpalmeiro Mar 1, 2024
e646b80
Add tests for selection_nS_mask_and_checking
bpalmeiro Mar 4, 2024
73157bd
Add band_selector_and_check function
bpalmeiro Mar 6, 2024
378dcb9
Add selection_in_band function
bpalmeiro Mar 7, 2024
56f0f70
Add scikit-learn to enviroment
bpalmeiro Dec 26, 2024
645cafc
Change pandas managing for clarity
bpalmeiro Dec 27, 2024
fd339ef
Remove leftover imports
bpalmeiro Dec 27, 2024
2ad45b6
Change Z to DT in naming
bpalmeiro Dec 27, 2024
c59a58e
Add silent to Strictness
bpalmeiro Dec 27, 2024
bd5c5ce
Add silent option to check_if_values_in_interval
bpalmeiro Dec 27, 2024
65f0947
Unify raises style
bpalmeiro Dec 27, 2024
ae385b5
Test warning
bpalmeiro Dec 27, 2024
4d6bfbd
Factorize selection dst
bpalmeiro Dec 30, 2024
96f29e5
Remove unneeded line
bpalmeiro Dec 30, 2024
5a8cd65
Add sigma estimation function
bpalmeiro Dec 30, 2024
dc0b30b
Generalize test to include both signals
bpalmeiro Dec 30, 2024
81d0f56
Refactor test to be less implementation dependent
bpalmeiro Dec 30, 2024
e013211
Factorize to remove hypotesis and comply with style
bpalmeiro Dec 30, 2024
4d3d0e8
Add test for sigma estimator function
bpalmeiro Dec 30, 2024
4977fc0
Change name to all_in_range
bpalmeiro Dec 30, 2024
94ab349
Remove import
bpalmeiro Dec 30, 2024
fe37a5b
Small refactor to all_in_range
bpalmeiro Dec 30, 2024
de11a09
Unify stile for raises
bpalmeiro Dec 30, 2024
1c0dbd3
Remove hypothesis fof random test
bpalmeiro Dec 30, 2024
9871e66
Cosmetics and types
bpalmeiro Dec 30, 2024
82b9160
Update typehints
bpalmeiro Dec 30, 2024
fb81f7a
Change naming for strictness type
bpalmeiro Jan 14, 2025
15ac9f7
Update all_in_range docsting
bpalmeiro Jan 14, 2025
0e64342
Add explicit strictness in tests
bpalmeiro Jan 14, 2025
769968b
Cosmetics
bpalmeiro Jan 14, 2025
582b1a9
Add match to warns enviroment
bpalmeiro Jan 14, 2025
b8a251a
Remove unused import
bpalmeiro Jan 14, 2025
c76c73c
Reword apply_geo_correction docstring
bpalmeiro Jan 14, 2025
c745cc4
Reword docstings in correction tests
bpalmeiro Jan 14, 2025
59386dd
Change function names to imperative
bpalmeiro Jan 14, 2025
0c635d0
Change variable naming for clarity
bpalmeiro Jan 14, 2025
61005e3
Update docstrings with right types
bpalmeiro Jan 14, 2025
3bed7d9
Remove wordines in given for tests
bpalmeiro Jan 14, 2025
7d12409
Change floats to ints
bpalmeiro Jan 14, 2025
8b584bb
Replace np.sum for np.count_nonzero
bpalmeiro Jan 14, 2025
85f3834
Add robust seeds
bpalmeiro Jan 16, 2025
273ce0b
Remove mixing hypothesis and np.random
bpalmeiro Jan 16, 2025
c35d18a
Include extreme cases
bpalmeiro Jan 16, 2025
7b5f2a9
Simplify logic
bpalmeiro Jan 16, 2025
91b69ac
Cosmetics and style
bpalmeiro Jan 22, 2025
0ac5006
Fix Optionals both for typehings and docstrings
bpalmeiro Jan 22, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 61 additions & 0 deletions invisible_cities/core/core_functions.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,19 @@
This module includes utility functions.
"""
import time
import warnings

import numpy as np

from typing import Sequence
from typing import Tuple
from typing import Optional


from . exceptions import ValueOutOfRange
from ..types.symbols import NormMode
from ..types.symbols import Strictness


def timefunc(f):
"""
Expand Down Expand Up @@ -77,6 +83,61 @@ def in_range(data, minval=-np.inf, maxval=np.inf, left_closed=True, right_closed
return lower_bound & upper_bound


def all_in_range(data : np.ndarray ,
minval : float ,
maxval : float ,
display_name : Optional[str] = '' ,
strictness : Optional[Strictness] = Strictness.raise_error,
**kwargs)->bool:
"""
Checks whether input values are all inside the interval (minval, maxval).

Parameters
----------
data : np.ndarray
Input array to check.
minval: float
Lower limit of the interval.
maxval: float
Upper limit of the interval.
display_name: string
Label to be displayed in case of warning or exception.
strictness: Strictness
Describes the behaviour when the output is `False`. If `strictness` is:
- `Strictness.raise_error`: an exception is raised.
- `Strictness.warning`: a warning is raised before returning.
- `Strictness.silent`: the function returns quietly.

**kwargs:
Optional arguments being passed to `in_range`.
Returns
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Returns
Returns

add blank line before Returns

-------
True if values are in the interval. False otherwise.

Raises
------
ValueOutOfRange: if the output is `False` and `strictness` is `Strictness.raise_error`
"""

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove blank line.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still there

values_in_interval = in_range(data, minval, maxval, **kwargs)
outliers = data[~values_in_interval]
n_outliers = len(outliers)

if values_in_interval.all():
return True
elif strictness is Strictness.silent:
return False

text = f'Variable {display_name} has {n_outliers} values out of bounds ({minval}, {maxval}\n'
text += f'Outliers: {outliers}'

if strictness is Strictness.warning:
warnings.warn(text, UserWarning)
return False
else:
raise ValueOutOfRange(text)


def weighted_mean_and_var(data : Sequence,
weights : Sequence,
unbiased : bool = False,
Expand Down
34 changes: 34 additions & 0 deletions invisible_cities/core/core_functions_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@
from pytest import approx
from pytest import mark
from pytest import raises
from pytest import warns


from flaky import flaky
from hypothesis import given
Expand Down Expand Up @@ -111,6 +113,38 @@ def test_in_range_right_close_interval(data):
assert all(output)


@given(random_length_float_arrays(min_length = 1,
min_value = 0,
max_value = 100))
def test_all_in_range_when_all_fall_inside(data):
assert core.all_in_range(data, 0, 100, left_closed=True,right_closed=True,
strictness=core.Strictness.raise_error)


@given(random_length_float_arrays(min_length = 1,
min_value = 0,
max_value = 100))
def test_all_in_range_when_some_fall_outside_silent(data):
minvalue = np.min(data)
assert not core.all_in_range(data, minvalue+1, 100, strictness=core.Strictness.silent)

@given(random_length_float_arrays(min_length = 1,
min_value = 0,
max_value = 100))
def test_all_in_range_when_some_fall_outside_warns(data):
minvalue = np.min(data)
with warns(UserWarning, match="values out of bounds"):
core.all_in_range(data, minvalue+1, 100, strictness=core.Strictness.warning)

@given(random_length_float_arrays(min_length = 1,
min_value = 0,
max_value = 100))
def test_all_in_range_when_some_fall_outside_exception(data):
minvalue = np.min(data)
with raises(core.ValueOutOfRange, match="values out of bounds"):
core.all_in_range(data, minvalue+1, 100, strictness=core.Strictness.raise_error)


@mark.parametrize(" first second norm_mode expected".split(),
(( 1 , 2 , core.NormMode.first , -1 ),
( 4 , 2 , core.NormMode.second, 1 ),
Expand Down
3 changes: 3 additions & 0 deletions invisible_cities/core/exceptions.py
Original file line number Diff line number Diff line change
Expand Up @@ -55,3 +55,6 @@ class MCEventNotFound(ICException):

class SensorIDMismatch(ICException):
pass

class ValueOutOfRange(ICException):
pass
34 changes: 34 additions & 0 deletions invisible_cities/reco/corrections.py
Original file line number Diff line number Diff line change
Expand Up @@ -243,6 +243,40 @@ def get_normalization_factor(map_e0 : ASectorMap,

return norm_value

def apply_geo_correction(map_e0 : ASectorMap,
norm_strat : NormStrategy = NormStrategy.max,
norm_value : Optional[float] = None
) -> Callable:
"""
Given a map and a correction strategy, it returns a
function that provides a geometric only correction factor
for a given hit collection when (x,y) is provided.

Parameters
----------
map_e0 : AsectorMap
Correction map for geometric effects.
norm_strat : NormStrategy
Provides the desired normalization to be used.
norm_value : Float(optional)
If norm_strat is selected to be custom, user must provide the
desired scale.
Comment on lines +259 to +263
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
norm_strat : NormStrategy
Provides the desired normalization to be used.
norm_value : Float(optional)
If norm_strat is selected to be custom, user must provide the
desired scale.
norm_strat : NormStrategy, optional
Provides the desired normalization to be used. Default: NormStrategy.max.
norm_value : float, optional
If norm_strat is selected to be custom, user must provide the
desired scale.

Returns
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add blank line before Returns

-------
A function that returns geometric correction factor without passing a map.
"""
normalization = get_normalization_factor(map_e0, norm_strat, norm_value)
get_xy_corr_fun = maps_coefficient_getter(map_e0.mapinfo, map_e0.e0)

def geo_correction_factor(x : np.ndarray,
y : np.ndarray)-> np.ndarray:
geo_factor = correct_geometry_(get_xy_corr_fun(x,y))
factor = geo_factor * normalization
return factor

return geo_correction_factor



def apply_all_correction_single_maps(map_e0 : ASectorMap,
map_lt : ASectorMap,
Expand Down
34 changes: 22 additions & 12 deletions invisible_cities/reco/corrections_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
from . corrections import get_normalization_factor
from . corrections import apply_all_correction_single_maps
from . corrections import apply_all_correction
from . corrections import apply_geo_correction

from pytest import fixture
from numpy.testing import assert_allclose
Expand Down Expand Up @@ -268,21 +269,30 @@ def test_apply_all_correction_single_maps_raises_exception_when_invalid_map(map_

@few_examples
@given(float_arrays(size = 1,
min_value = -198,
max_value = +198),
float_arrays(size = 1,
min_value = -198,
max_value = +198),
float_arrays(size = 1,
min_value = 0,
max_value = 5e2),
min_value = -198,
max_value = +198),
float_arrays(size = 1,
min_value = 0,
max_value = 1e5))
min_value = -198,
max_value = +198))
def test_apply_geo_correction_properly(map_filename, x, y):
Comment on lines 271 to +277
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Display each float_arrays call as a single line like the test below

"""
The input map is homogeneous, therefore the geometric
correction factor must be 1.
"""
maps = read_maps(map_filename)
load_corr = apply_geo_correction(map_e0=maps, norm_strat=NormStrategy.max)
corr = load_corr(x, y)
assert_allclose (corr, np.ones_like(corr))

@few_examples
@given(float_arrays(size = 1, min_value = -198, max_value = 198),
float_arrays(size = 1, min_value = -198, max_value = 198),
float_arrays(size = 1, min_value = 0, max_value = 500),
float_arrays(size = 1, min_value = 0, max_value = 100*1000))
Comment on lines +288 to +291
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove unnecessary spaces after "size"

def test_apply_all_correction_single_maps_properly(map_filename, x, y, z, t):
"""
Due the map taken as input, the geometric correction
factor must be 1, the temporal correction 1 and the
The input map is homogeneous, therefore the geometric
correction factor must be 1, the temporal correction 1 and the
lifetime one: exp(Z/5000).
"""
maps = read_maps(map_filename)
Expand Down
Loading