Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replaced numpy.testing.assert_almost_equal to numpy.testing.assert_allclose #4438

Merged
merged 20 commits into from
Feb 24, 2024

Conversation

aditya292002
Copy link
Contributor

@aditya292002 aditya292002 commented Jan 24, 2024

Fixes #

Changes made in this Pull Request:

  • Replaced numpy.testing.assert_almost_equal to numpy.testing.assert_allclose for more consistent floating point comparisons. as mentioned in the issue #3743
  • Changes are made in mdanalysis/testsuite/MDAnalysisTests/analysis/test_align.py and /mdanalysis/package/MDAnalysis/visualization/streamlines_3D.py

PR Checklist

Developers certificate of origin


📚 Documentation preview 📚: https://mdanalysis--4438.org.readthedocs.build/en/4438/

…tent floating point comparisons

e.g.         # assert_almost_equal(rms.rmsd(first_frame, first_frame), 0.0, 5,
        #                     err_msg="error: rmsd(X,X) should be 0")

        assert_allclose(rms.rmsd(first_frame, first_frame), 0.0, rtol=0, atol=1.5*1e-5,
                        err_msg="error: rmsd(X,X) should be 0")

1.5 factor in atol bcz,in assert_almost_equal
The test verifies that the elements of actual and desired satisfy.
abs(desired-actual) < float64(1.5 * 10**(-decimal))
So, to keep behaviour consistent in assert_allclose introduced factor 1.5 in atol
@pep8speaks
Copy link

pep8speaks commented Jan 24, 2024

Hello @aditya292002! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

Line 248:80: E501 line too long (83 > 79 characters)
Line 249:43: E127 continuation line over-indented for visual indent
Line 249:80: E501 line too long (111 > 79 characters)

Line 200:80: E501 line too long (85 > 79 characters)
Line 207:80: E501 line too long (93 > 79 characters)
Line 210:80: E501 line too long (100 > 79 characters)
Line 210:101: W291 trailing whitespace
Line 320:80: E501 line too long (84 > 79 characters)
Line 360:44: E251 unexpected spaces around keyword / parameter equals
Line 360:46: E251 unexpected spaces around keyword / parameter equals
Line 396:35: E127 continuation line over-indented for visual indent
Line 439:80: E501 line too long (87 > 79 characters)
Line 446:29: E127 continuation line over-indented for visual indent
Line 454:29: E127 continuation line over-indented for visual indent

Line 254:80: E501 line too long (81 > 79 characters)

Line 265:80: E501 line too long (87 > 79 characters)
Line 282:80: E501 line too long (87 > 79 characters)
Line 307:80: E501 line too long (83 > 79 characters)
Line 367:61: E231 missing whitespace after ','
Line 367:80: E501 line too long (81 > 79 characters)
Line 413:80: E501 line too long (84 > 79 characters)
Line 435:80: E501 line too long (84 > 79 characters)

Line 49:80: E501 line too long (84 > 79 characters)
Line 57:80: E501 line too long (84 > 79 characters)
Line 65:80: E501 line too long (84 > 79 characters)
Line 94:80: E501 line too long (81 > 79 characters)
Line 102:80: E501 line too long (87 > 79 characters)
Line 179:80: E501 line too long (89 > 79 characters)

Line 156:80: E501 line too long (98 > 79 characters)
Line 176:80: E501 line too long (103 > 79 characters)
Line 195:53: E231 missing whitespace after ','
Line 195:80: E501 line too long (93 > 79 characters)
Line 277:80: E501 line too long (88 > 79 characters)
Line 368:38: E231 missing whitespace after ','
Line 379:42: E231 missing whitespace after ','
Line 379:80: E501 line too long (83 > 79 characters)
Line 399:80: E501 line too long (87 > 79 characters)
Line 400:80: E501 line too long (85 > 79 characters)
Line 416:29: E127 continuation line over-indented for visual indent
Line 416:80: E501 line too long (128 > 79 characters)
Line 440:17: E251 unexpected spaces around keyword / parameter equals
Line 440:19: E251 unexpected spaces around keyword / parameter equals
Line 441:17: E251 unexpected spaces around keyword / parameter equals
Line 441:19: E251 unexpected spaces around keyword / parameter equals

Comment last updated at 2024-02-24 09:35:19 UTC

Copy link

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello there first time contributor! Welcome to the MDAnalysis community! We ask that all contributors abide by our Code of Conduct and that first time contributors introduce themselves on GitHub Discussions so we can get to know you. You can learn more about participating here. Please also add yourself to package/AUTHORS as part of this PR.

Copy link

github-actions bot commented Jan 24, 2024

Linter Bot Results:

Hi @aditya292002! Thanks for making this PR. We linted your code and found the following:

Some issues were found with the formatting of your code.

Code Location Outcome
main package ⚠️ Possible failure
testsuite ⚠️ Possible failure

Please have a look at the darker-main-code and darker-test-code steps here for more details: https://github.com/MDAnalysis/mdanalysis/actions/runs/8029691872/job/21936262886


Please note: The black linter is purely informational, you can safely ignore these outcomes if there are no flake8 failures!

Copy link

codecov bot commented Jan 24, 2024

Codecov Report

Attention: Patch coverage is 0% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 93.37%. Comparing base (3a5339c) to head (e1f2f4e).

Files Patch % Lines
package/MDAnalysis/visualization/streamlines_3D.py 0.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #4438      +/-   ##
===========================================
- Coverage    93.38%   93.37%   -0.01%     
===========================================
  Files          171      183      +12     
  Lines        21744    22823    +1079     
  Branches      4014     4014              
===========================================
+ Hits         20305    21311    +1006     
- Misses         952     1025      +73     
  Partials       487      487              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Member

@tylerjereddy tylerjereddy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had one or two comments, apart from those most cases seem to match this idiom:

Before: abs(desired-actual) < float64(1.5 * 10**(-decimal))
New: abs(desired-actual) < atol + rtol * abs(desired) with 0 for rtol reducing to abs(desired-actual) < atol, so you just set atol to 1.5 * 10**(-decimal) to match the original behavior.

I suppose that's hard to argue with in terms of being equivalent. There probably are some cases where you do want to allow an rtol, but I don't have a huge problem with the literal translations for now I suppose.

@@ -167,7 +167,8 @@ def test_subselection_alignto(self, universe, reference, subselection, expectati

with expectation:
rmsd = align.alignto(universe, reference, subselection=subselection)
assert_almost_equal(rmsd[1], 0.0, decimal=9)
# assert_almost_equal(rmsd[1], 0.0, decimal=9)
assert_allclose(rmsd[1], 0.0, rtol=0, atol=1.5*1e-9)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can remove the commented old version


def test_rmsd_custom_mass_weights(self, universe, reference):
last_atoms_weight = universe.atoms.masses
A = universe.trajectory[0]
B = reference.trajectory[-1]
rmsd = align.alignto(universe, reference,
weights=reference.atoms.masses)
weights=reference.atoms.masses)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some of the whitespace changes don't seem useful?

err_msg="assert_allclose failed for positions")
assert_allclose(avg.results.rmsd, rmsd,
rtol=0, atol=1.5*(1e-4),
err_msg="assert_allclose failed for rmsd")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fwiw, I don't think these custom messages are worth it in this context, but that may just be based on the style of the projects I usually maintain

rtol=0, atol=1.5*(1e-4),
err_msg="assert_allclose failed for positions")
assert_allclose(avg.results.rmsd, rmsd,
rtol=0, atol=1.5*(1e-4),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't you want 1e-7 since assert_almost_equal is using default arguments in the original here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, that's a mistake. I will correct this.

@aditya292002
Copy link
Contributor Author

Hi, @tylerjereddy, I've made all the suggested changes. Should I proceed with applying the same modifications to all the files? There are nearly 840 occurrences of just the 'assert_almost_close'. I'm thinking of creating a Python script to automate these changes. Should I go ahead with that approach?

Also, could you please explain why not all the test cases are passing? I didn't make any changes to the '/mdanalysis/testsuite/MDAnalysisTests/parallelism/test_multiprocessing.py' file, but the 'def test_creating_multiple_universe_without_offset(temp_xtc, ncopies=3)' test failed for Ubuntu with Python 3.12.

Thank you.

@IAlibay
Copy link
Member

IAlibay commented Jan 25, 2024

@aditya292002 this issue is really meant to be taken in small chunks and not all on one go. I would suggest doing a handful of files so that it's both manageable to yourself and reviewers.

Regarding the test failures, this is a known issue and not related to this PR.

@aditya292002
Copy link
Contributor Author

@IAlibay, I've made additional changes to the files: test_base, test_bat, test_contracts, test_dihedrals, and test_encore.

Please review these modifications and merge this pull request for the time being. I'd like to further explore this project and address any other issues.
Thank you for your assistance.

@aditya292002
Copy link
Contributor Author

aditya292002 commented Jan 26, 2024

@IAlibay, I've made additional changes to the files: test_base, test_bat, test_contracts, test_dihedrals, and test_encore.

Please review these modifications and merge this pull request for the time being. I'd like to further explore this project and address any other issues. Thank you for your assistance.

@IAlibay
Copy link
Member

IAlibay commented Jan 26, 2024

@aditya292002 - whilst I appreciate your work here, I would like to ask for your patience. The MDAnalysis developers are not full time hired on the project and have their own separate jobs / lives. I will add this to my task list and will attempt to review, althoughtit may take at least a few days (there are already a few other reviews pending my attention).

@aditya292002
Copy link
Contributor Author

@aditya292002 - whilst I appreciate your work here, I would like to ask for your patience. The MDAnalysis developers are not full time hired on the project and have their own separate jobs / lives. I will add this to my task list and will attempt to review, althoughtit may take at least a few days (there are already a few other reviews pending my attention).

Apologies for any impatience displayed. I am currently a university student, and this marks my initial involvement in contributing to an open-source project. Rest assured, I will take note of this experience for future contributions.

@IAlibay
Copy link
Member

IAlibay commented Jan 31, 2024

I won't be able to get to reviewing this any time soon - can one of the current @MDAnalysis/coredevs take this on instead please?

@orbeckst
Copy link
Member

@tylerjereddy would you be able review this PR, given that you already had some eyes on it?

@tylerjereddy
Copy link
Member

@aditya292002 please remove the testsuite/MDAnalysisTests/analysis/__pycache__/test_gnm.cpython-310-pytest-7.4.4.pyc.1853 file from your branch, we don't want Python bytecode files under version control.

After you check that your diff only has the changes that are needed, feel free to ping me again and I'll take another look.

@aditya292002
Copy link
Contributor Author

@aditya292002 please remove the testsuite/MDAnalysisTests/analysis/__pycache__/test_gnm.cpython-310-pytest-7.4.4.pyc.1853 file from your branch, we don't want Python bytecode files under version control.

After you check that your diff only has the changes that are needed, feel free to ping me again and I'll take another look.

@tylerjereddy, I have removed the specified bytecode file.

assert_almost_equal(rmsd, desired, decimal=5,
err_msg="frame {0:d} of fit does not have "
"expected RMSD".format(frame))
fitted.trajectory[frame]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You still have unwanted whitespace changes here and elsewhere, please re-read the entire diff line-by-line and make sure those changes are not present (since I'm spending time doing the same).

Copy link
Member

@orbeckst orbeckst left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Please fix any whitespace issues: if you didn't touch a line then it shouldn't change in the diff.
  2. Write floats in engineering notation and not as products (see comment).
  3. Remove assert_almost_equal from the import line in any file where you replaced it with assert_allclose.

np.testing.assert_almost_equal(array_cube_vertex_distances_from_centroid.min(),
array_cube_vertex_distances_from_centroid.max(), decimal=4,
np.testing.assert_allclose(array_cube_vertex_distances_from_centroid.min(),
array_cube_vertex_distances_from_centroid.max(), rtol=0, atol=1.5*(1e-4),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Write floats as 1.5e-4 and not as 1.5*(1e-4).

This applies to all floats with powers of ten throughout.


self._assert_rmsd(reference, fitted, 0, 0.0,
weights=universe.atoms.masses)
weights=universe.atoms.masses)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

undo indent

self._assert_rmsd(reference, fitted, -1, 6.929083032629219,
weights=universe.atoms.masses)
weights=universe.atoms.masses)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

undo indent

@@ -27,7 +27,7 @@

import numpy as np

from numpy.testing import assert_equal, assert_almost_equal
from numpy.testing import assert_equal, assert_almost_equal, assert_allclose
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove assert_almost_equal: the whole point of the PR is to get rid of it :-)

@orbeckst orbeckst self-assigned this Feb 10, 2024
assert_almost_equal(result_value, expected_value, decimal=-3,
err_msg="Unexpected value for Harmonic Ensemble Similarity: {0:f}. Expected {1:f}.".format(result_value, expected_value))
assert_allclose(result_value, expected_value, rtol=0, atol=1.5*(1e3),
err_msg="Unexpected value for Harmonic Ensemble Similarity: {0:f}. Expected {1:f}.".format(result_value, expected_value))

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please check if I have got this right

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it's a very loose test!

err_msg="Unexpected average value for bootstrapped samples in Clustering Ensemble similarity")
assert_almost_equal(stdev, expected_stdev, decimal=0,

assert_almost_equal(stdev, expected_stdev, decimal=0,
err_msg="Unexpected standard daviation for bootstrapped samples in Clustering Ensemble similarity")
Copy link
Contributor Author

@aditya292002 aditya292002 Feb 13, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for this, "assert_almost_equal" statement
the value of parameter decimal = 0,

which means abs(x,y) should be < 1.5 (which i found by running some examples),

but, for assert_allclose(x,y, rtol=0, atol=0) means the abs(x-y) = 0,

To resolve this should I go ahead and replace the assert_almost_equal which a if condition.
Please suggest some alternate ways to handle it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The equivalent for decimal=0 is atol=1.5e0 which is just atol=1.5:

assert_allclose(stdev, expected_stdev, rtol=0, atol=1.5)

Does the code above work?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Btw, can you please fix the spelling mistake: "daviation" --> "deviation"?

removed all the extra widespaces and correct the format for atol=1.5e-x
@aditya292002
Copy link
Contributor Author

@tylerjereddy, could you please take a look at the changes I made?

@aditya292002
Copy link
Contributor Author

@orbeckst , could you please take a look at the changes I made?

Copy link
Member

@orbeckst orbeckst left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks almost done:

  1. fix the last assert_almost_equal with decimal=0 (see comment)

  2. add yourself to the AUTHORS file

  3. add an entry to CHANGELOG:

    "updated tests that used assert_almost_equal(..., decimal={N}) with equivalent assert_allclose(..., rtol=0, atol=1.5e-{N}) (issue modernize testing code #3743, PR Replaced numpy.testing.assert_almost_equal to numpy.testing.assert_allclose #4438)"

    (Normally we don't include changes to the tests in CHANGELOG, but this one may make some tests fail and we want to have documentation that alerts us to the fact that we changed test evaluation.)

assert_equal,
assert_array_equal,
assert_array_almost_equal
assert_array_almost_equal,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

assert_array_almost_equal() could also be replaced with assert_allclose().

assert_almost_equal(result_value, expected_value, decimal=-3,
err_msg="Unexpected value for Harmonic Ensemble Similarity: {0:f}. Expected {1:f}.".format(result_value, expected_value))
assert_allclose(result_value, expected_value, rtol=0, atol=1.5*(1e3),
err_msg="Unexpected value for Harmonic Ensemble Similarity: {0:f}. Expected {1:f}.".format(result_value, expected_value))

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it's a very loose test!

err_msg="Unexpected average value for bootstrapped samples in Clustering Ensemble similarity")
assert_almost_equal(stdev, expected_stdev, decimal=0,

assert_almost_equal(stdev, expected_stdev, decimal=0,
err_msg="Unexpected standard daviation for bootstrapped samples in Clustering Ensemble similarity")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The equivalent for decimal=0 is atol=1.5e0 which is just atol=1.5:

assert_allclose(stdev, expected_stdev, rtol=0, atol=1.5)

Does the code above work?

and
- changed assert_array_almost_equal with assert_allclose
Copy link
Member

@orbeckst orbeckst left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please fix CHANGELOG, otherwise looking good!

UPDATE: There are now tests that are failing. Please have a look at them and propose adjustments to the tolerances to make them pass again.

24/02/23 aditya292002
* "updated tests that used assert_almost_equal(..., decimal={N}) with equivalent assert_allclose(..., rtol=0, atol=1.5e-{N}) (issue modernize testing code #3743, PR Replaced numpy.testing.assert_almost_equal to numpy.testing.assert_allclose #4438)"

/workspaces/mdanalysis/package/CHANGELOG
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove this line!

Comment on lines 3340 to 3341
24/02/23 aditya292002
* "updated tests that used assert_almost_equal(..., decimal={N}) with equivalent assert_allclose(..., rtol=0, atol=1.5e-{N}) (issue modernize testing code #3743, PR Replaced numpy.testing.assert_almost_equal to numpy.testing.assert_allclose #4438)"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add information to the top of the file under 2.8.0.

  • add your handle after ??/??/?? IAlibay, HeetVekariya, marinegor, lilyminium
  • add the text under "Changes"

package/CHANGELOG Outdated Show resolved Hide resolved
package/CHANGELOG Outdated Show resolved Hide resolved
@aditya292002
Copy link
Contributor Author

@orbeckst Do I need to do anything more in this PR.

@orbeckst orbeckst merged commit 2c1aa4b into MDAnalysis:develop Feb 24, 2024
22 of 24 checks passed
@orbeckst
Copy link
Member

Congratulations on your first contribution @aditya292002 ! 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants