Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

E2e test of Session with AADForest takes up to one hour #121

Open
hombit opened this issue Jul 10, 2023 · 1 comment
Open

E2e test of Session with AADForest takes up to one hour #121

hombit opened this issue Jul 10, 2023 · 1 comment
Labels
bug Something isn't working performance Performance improvements and benchmarks

Comments

@hombit
Copy link
Member

hombit commented Jul 10, 2023 •
edited
Loading

When Github Action runs test_non_anomalous_outliers variant for AADForest it takes ~20 seconds for macOS runners, and up to ~4000 seconds for Linux and Windows runners (while accidentally it could be 20 seconds as well). I'm going to mark this test as "long" in #119 which would prevent it from running in CI. However, we need to investigate what is happening there, because I think I can reproduce the issue at Linux. cProfile shows that it spends all the time in minimize:

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   1507/1    0.045    0.000 4078.951 4078.951 {built-in method builtins.exec}
        1    0.000    0.000 4078.951 4078.951 <string>:1(<module>)
        1    0.000    0.000 4078.951 4078.951 <frozen runpy>:201(run_module)
        1    0.000    0.000 4078.812 4078.812 <frozen runpy>:65(_run_code)
        1    0.000    0.000 4078.812 4078.812 /usr/local/lib/python3.11/site-packages/pytest/__main__.py:1(<module>)
        1    0.000    0.000 4078.812 4078.812 /usr/local/lib/python3.11/site-packages/_pytest/config/__init__.py:182(console_main)
        1    0.000    0.000 4078.812 4078.812 /usr/local/lib/python3.11/site-packages/_pytest/config/__init__.py:134(main)
   479/72    0.000    0.000 4078.789   56.650 /usr/local/lib/python3.11/site-packages/pluggy/_manager.py:103(_hookexec)
   479/72    0.003    0.000 4078.788   56.650 /usr/local/lib/python3.11/site-packages/pluggy/_callers.py:30(_multicall)
    286/2    0.001    0.000 4078.786 2039.393 /usr/local/lib/python3.11/site-packages/pluggy/_hooks.py:427(__call__)
        1    0.000    0.000 4077.793 4077.793 /usr/local/lib/python3.11/site-packages/_pytest/main.py:316(pytest_cmdline_main)
        1    0.000    0.000 4077.793 4077.793 /usr/local/lib/python3.11/site-packages/_pytest/main.py:257(wrap_session)
        1    0.000    0.000 4077.754 4077.754 /usr/local/lib/python3.11/site-packages/_pytest/main.py:320(_main)
       16    0.000    0.000 4077.742  254.859 /usr/local/lib/python3.11/site-packages/_pytest/runner.py:318(from_call)
        1    0.000    0.000 4076.376 4076.376 /usr/local/lib/python3.11/site-packages/_pytest/main.py:337(pytest_runtestloop)
        3    0.000    0.000 4076.375 1358.792 /usr/local/lib/python3.11/site-packages/_pytest/runner.py:111(pytest_runtest_protocol)
        3    0.000    0.000 4076.374 1358.791 /usr/local/lib/python3.11/site-packages/_pytest/runner.py:119(runtestprotocol)
        9    0.000    0.000 4076.374  452.930 /usr/local/lib/python3.11/site-packages/_pytest/runner.py:219(call_and_report)
        9    0.000    0.000 4076.372  452.930 /usr/local/lib/python3.11/site-packages/_pytest/runner.py:247(call_runtest_hook)
        9    0.000    0.000 4076.371  452.930 /usr/local/lib/python3.11/site-packages/_pytest/runner.py:262(<lambda>)
        3    0.000    0.000 4076.366 1358.789 /usr/local/lib/python3.11/site-packages/_pytest/runner.py:160(pytest_runtest_call)
        3    0.000    0.000 4076.366 1358.789 /usr/local/lib/python3.11/site-packages/_pytest/python.py:1786(runtest)
        3    0.000    0.000 4076.366 1358.789 /usr/local/lib/python3.11/site-packages/_pytest/python.py:187(pytest_pyfunc_call)
        3    0.003    0.001 4076.297 1358.766 /usr/local/lib/python3.11/site-packages/coniferest/session/__init__.py:130(run)
        2    0.000    0.000 4074.363 2037.182 /coniferest/tests/test_session.py:59(test_non_anomalous_outliers)
       50    0.004    0.000 4058.721   81.174 /usr/local/lib/python3.11/site-packages/coniferest/aadforest.py:183(fit_known)
       46    0.001    0.000 4057.288   88.202 /usr/local/lib/python3.11/site-packages/scipy/optimize/_minimize.py:51(minimize)
       46    0.002    0.000 4057.286   88.202 /usr/local/lib/python3.11/site-packages/scipy/optimize/_trustregion_krylov.py:6(_minimize_trust_krylov)
       46  529.811   11.518 4057.284   88.202 /usr/local/lib/python3.11/site-packages/scipy/optimize/_trustregion.py:118(_minimize_trust_region)
  1788434  100.853    0.000 2000.168    0.001 /usr/local/lib/python3.11/site-packages/scipy/optimize/_trlib/__init__.py:7(subproblem_factory)
 21457483    8.136    0.000 1898.687    0.000 /usr/local/lib/python3.11/site-packages/scipy/optimize/_trustregion.py:63(jac)
  1788434    8.219    0.000 1890.551    0.001 /usr/local/lib/python3.11/site-packages/scipy/optimize/_differentiable_functions.py:270(grad)
  1788480    2.492    0.000 1808.688    0.001 /usr/local/lib/python3.11/site-packages/scipy/optimize/_differentiable_functions.py:254(_update_grad)
  1788259    2.374    0.000 1806.196    0.001 /usr/local/lib/python3.11/site-packages/scipy/optimize/_differentiable_functions.py:166(update_grad)
  1788259   10.396    0.000 1803.822    0.001 /usr/local/lib/python3.11/site-packages/scipy/optimize/_differentiable_functions.py:162(grad_wrapped)
  1788259    3.339    0.000 1775.780    0.001 /usr/local/lib/python3.11/site-packages/coniferest/aadforest.py:227(jac)
  1788259  671.409    0.000 1772.440    0.001 /usr/local/lib/python3.11/site-packages/coniferest/aadforest.py:75(loss_gradient)
  3576612 1662.970    0.000 1769.759    0.000 /usr/local/lib/python3.11/site-packages/coniferest/aadforest.py:21(score_samples)
  8941951    4.618    0.000  927.250    0.000 /usr/local/lib/python3.11/site-packages/scipy/optimize/_trustregion.py:56(fun)
  1788434    4.683    0.000  922.632    0.001 /usr/local/lib/python3.11/site-packages/scipy/optimize/_differentiable_functions.py:264(fun)
  1788480    1.968    0.000  884.608    0.000 /usr/local/lib/python3.11/site-packages/scipy/optimize/_differentiable_functions.py:249(_update_fun)
  1788259    1.627    0.000  882.640    0.000 /usr/local/lib/python3.11/site-packages/scipy/optimize/_differentiable_functions.py:154(update_fun)
  1788259    8.763    0.000  881.013    0.000 /usr/local/lib/python3.11/site-packages/scipy/optimize/_differentiable_functions.py:132(fun_wrapped)
  1788259    2.796    0.000  859.280    0.000 /usr/local/lib/python3.11/site-packages/coniferest/aadforest.py:224(fun)
  1788259   87.575    0.000  856.484    0.000 /usr/local/lib/python3.11/site-packages/coniferest/aadforest.py:44(loss)
  1788388    3.986    0.000  408.667    0.000 /usr/local/lib/python3.11/tempfile.py:305(mkstemp)
  1788390   13.562    0.000  384.958    0.000 /usr/local/lib/python3.11/tempfile.py:243(_mkstemp_inner)
  1788394  308.731    0.000  308.731    0.000 {built-in method posix.open}
  3574954    4.941    0.000   99.082    0.000 /usr/local/lib/python3.11/site-packages/scipy/optimize/_trustregion.py:77(hessp)
  1788388   34.921    0.000   94.742    0.000 /usr/local/lib/python3.11/site-packages/scipy/optimize/_trustregion.py:53(__call__)
  3574954   19.286    0.000   94.141    0.000 /usr/local/lib/python3.11/site-packages/scipy/optimize/_trustregion.py:22(function_wrapper)
  5386025   65.965    0.000   80.817    0.000 /usr/local/lib/python3.11/site-packages/numpy/lib/function_base.py:1324(diff)
  3576868   41.699    0.000   79.492    0.000 /usr/local/lib/python3.11/site-packages/numpy/core/numeric.py:2378(array_equal)
  8958722   26.097    0.000   71.486    0.000 /usr/local/lib/python3.11/site-packages/numpy/core/fromnumeric.py:71(_wrapreduction)
 14427940   68.878    0.000   68.895    0.000 {method 'reduce' of 'numpy.ufunc' objects}
  5365005   11.432    0.000   61.633    0.000 /usr/local/lib/python3.11/site-packages/numpy/core/fromnumeric.py:2322(any)
  7151518    5.984    0.000   53.279    0.000 /usr/local/lib/python3.11/site-packages/numpy/lib/function_base.py:873(copy)
  3576745    4.068    0.000   51.972    0.000 /usr/local/lib/python3.11/site-packages/scipy/optimize/_trustregion.py:83(jac_mag)
  1786577   17.040    0.000   47.459    0.000 /usr/local/lib/python3.11/site-packages/scipy/linalg/_misc.py:17(norm)
  7232863   47.386    0.000   47.386    0.000 {built-in method numpy.array}
  5363445    7.098    0.000   45.984    0.000 {method 'all' of 'numpy.ndarray' objects}
  3574954    5.686    0.000   42.766    0.000 /usr/local/lib/python3.11/site-packages/coniferest/aadforest.py:230(hessp)
  5363537    4.658    0.000   38.886    0.000 /usr/local/lib/python3.11/site-packages/numpy/core/_methods.py:61(_all)
  3574954   37.080    0.000   37.080    0.000 /usr/local/lib/python3.11/site-packages/coniferest/aadforest.py:94(loss_hessian)
1789549/1789530    2.604    0.000   34.889    0.000 {built-in method builtins.next}
@hombit hombit added bug Something isn't working performance Performance improvements and benchmarks labels Jul 10, 2023
@hombit hombit mentioned this issue Jul 11, 2023
Merged
@hombit
Copy link
Member Author

hombit commented Mar 12, 2024

This should be reevaluated

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working performance Performance improvements and benchmarks
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@hombit