Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix the two sided test on IBMA Fishers and Stouffers estimators #903

Closed
wants to merge 7 commits into from

Conversation

JulioAPeraza
Copy link
Collaborator

@JulioAPeraza JulioAPeraza commented Sep 10, 2024

Closes None.

Changes proposed in this pull request:

  • As noticed in PyMARE, combination tests only take z-scores from one-sided p-values. Before, we were performing a concordant test, but that returned the values of the left tail to positive as well.
  • To be able to handle both tails, we perform the test for each tail separately and take the values with minimum p-value as output.

Summary by Sourcery

Fix the two-sided test implementation for IBMA Fishers and Stouffers estimators by running separate tests for each tail and using the minimum p-value to ensure correct handling of both tails.

Bug Fixes:

  • Fix the two-sided test for IBMA Fishers and Stouffers estimators by separately handling each tail and selecting the minimum p-value.

@JulioAPeraza JulioAPeraza added bug Issues noting problems and PRs fixing those problems. ibma Issues/PRs pertaining to image-based meta-analysis labels Sep 10, 2024
Copy link
Contributor

sourcery-ai bot commented Sep 10, 2024

Reviewer's Guide by Sourcery

This pull request fixes the implementation of two-sided tests for Fisher's and Stouffer's estimators in the IBMA (Image-Based Meta-Analysis) module. The changes modify how z-scores and p-values are calculated to properly handle both positive and negative tails in two-sided tests.

File-Level Changes

Change Details Files
Implement separate calculations for positive and negative tails in two-sided tests
  • Remove the '_mode' attribute from both Fishers and Stouffers classes
  • Add separate estimators for positive and negative tails
  • Implement logic to choose the minimum p-value between positive and negative tails
  • Create z-map based on which tail has the smaller p-value
  • Update the return values to use the new calculation method
nimare/meta/ibma.py
Modify Fisher's method to handle two-sided tests correctly
  • Add conditional logic for two-sided tests
  • Create separate PyMARE datasets for positive and negative tails
  • Implement minimum p-value selection for two-sided tests
  • Update z-map calculation based on selected p-values
nimare/meta/ibma.py
Update Stouffer's method to properly handle two-sided tests
  • Remove single estimator approach and implement separate estimators for each tail
  • Add conditional logic for two-sided tests
  • Create separate PyMARE datasets for positive and negative tails
  • Implement minimum p-value selection for two-sided tests
  • Update z-map calculation based on selected p-values
nimare/meta/ibma.py

Tips
  • Trigger a new Sourcery review by commenting @sourcery-ai review on the pull request.
  • Continue your discussion with Sourcery by replying directly to review comments.
  • You can change your review settings at any time by accessing your dashboard:
    • Enable or disable the Sourcery-generated pull request summary or reviewer's guide;
    • Change the review language;
  • You can always contact us if you have any questions or feedback.

Copy link
Contributor

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @JulioAPeraza - I've reviewed your changes - here's some feedback:

Overall Comments:

  • Consider adding explanatory comments for the two-sided test implementation to improve code readability and maintainability.
  • To reduce code duplication, consider extracting the common logic for positive and negative tails into a separate function.
Here's what I looked at during the review
  • 🟡 General issues: 1 issue found
  • 🟢 Security: all looks good
  • 🟢 Testing: all looks good
  • 🟢 Complexity: all looks good
  • 🟢 Documentation: all looks good

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment to tell me if it was helpful.

est = pymare.estimators.FisherCombinationTest(mode=self._mode)
est.fit_dataset(pymare_dset)
est_summary = est.summary()
# Run Stouffers for right tail
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion (performance): Consider performance implications of running two separate tests for two-sided cases

The new implementation runs two separate tests when two_sided is True, which could potentially double the computation time. Consider if there's a way to optimize this, especially for large datasets.

Copy link

codecov bot commented Sep 10, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 88.25%. Comparing base (2c9b52a) to head (23865c5).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #903      +/-   ##
==========================================
+ Coverage   88.23%   88.25%   +0.02%     
==========================================
  Files          48       48              
  Lines        6382     6394      +12     
==========================================
+ Hits         5631     5643      +12     
  Misses        751      751              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@JulioAPeraza
Copy link
Collaborator Author

@jdkent
Copy link
Member

jdkent commented Sep 11, 2024

I'm making a pull request here to fix the underlying bug in pymare: neurostuff/PyMARE#125

@JulioAPeraza
Copy link
Collaborator Author

Thanks! Closing this in favor of #904.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Issues noting problems and PRs fixing those problems. ibma Issues/PRs pertaining to image-based meta-analysis
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants