Testing gets stuck at test_cli_multiprocess

[YoihenBachu]

============================================================= test session starts ==============================================================
platform linux -- Python 3.10.14, pytest-8.1.1, pluggy-1.5.0 -- /home/thdam/miniconda3/envs/histoqc/bin/python
cachedir: .pytest_cache
rootdir: /home/thdam/work/HistoQC
configfile: pyproject.toml
collected 19 items

histoqc/tests/test_config_cli.py::test_config_cli_list_configurations PASSED                                                             [  5%]
histoqc/tests/test_config_cli.py::test_config_cli_show_configuration[default] PASSED                                                     [ 10%]
histoqc/tests/test_config_cli.py::test_config_cli_show_configuration[clinical] PASSED                                                    [ 15%]
histoqc/tests/test_config_cli.py::test_config_cli_show_configuration[first] PASSED                                                       [ 21%]
histoqc/tests/test_config_cli.py::test_config_cli_show_configuration[light] PASSED                                                       [ 26%]
histoqc/tests/test_config_cli.py::test_config_cli_incorrect_configuration PASSED                                                         [ 31%]
histoqc/tests/test_config_cli.py::test_config_cli_print_usage PASSED                                                                     [ 36%]
histoqc/tests/test_data_cli.py::test_data_cli_write_output PASSED                                                                        [ 42%]
histoqc/tests/test_data_cli.py::test_data_cli_incorrect_output PASSED                                                                    [ 47%]
histoqc/tests/test_pipeline_cli.py::test_cli_no_args PASSED                                                                              [ 52%]
histoqc/tests/test_pipeline_cli.py::test_cli_defaults PASSED                                                                             [ 57%]
histoqc/tests/test_pipeline_cli.py::test_cli_multiprocess ^CFAILED                                                                         [ 63%]
histoqc/tests/test_pipeline_cli.py::test_cli_multiprocess_batching PASSED                                                                [ 68%]
histoqc/tests/test_pipeline_cli.py::test_cli_extraction PASSED                                                                           [ 73%]
histoqc/tests/test_ui_cli.py::TestUIServer::test_ui_server PASSED                                                                        [ 78%]
histoqc/tests/test_ui_cli.py::TestUIServer::test_ui_server_data_directory_mount PASSED                                                   [ 84%]
histoqc/tests/test_ui_cli.py::TestUIServer::test_ui_server_invalid_path PASSED                                                           [ 89%]
histoqc/tests/test_ui_cli.py::test_ui_server_deploy PASSED                                                                               [ 94%]
histoqc/tests/test_ui_cli.py::test_ui_server_deploy_no_dir PASSED                                                                        [100%]

=================================================================== FAILURES ===================================================================
____________________________________________________________ test_cli_multiprocess _____________________________________________________________

multi_svs_dir = PosixPath('/tmp/pytest-of-thdam/pytest-0/histoqc_test_multi0')
tmp_path = PosixPath('/tmp/pytest-of-thdam/pytest-0/test_cli_multiprocess0')

    def test_cli_multiprocess(multi_svs_dir, tmp_path):
        assert main(['-n', '2', '-p', os.fspath(multi_svs_dir), '-o', os.fspath(tmp_path), '*.svs']) == 0
>       assert _filenames_in(tmp_path) == _filenames_in(multi_svs_dir).union(['error.log', 'results.tsv'])
E       AssertionError: assert {'1.svs', '2...., 'error.log'} == {'1.svs', '2....'results.tsv'}
E
E         Extra items in the right set:
E         'results.tsv'
E
E         Full diff:
E           {
E               '1.svs',...
E
E         ...Full output truncated (4 lines hidden), use '-vv' to show

histoqc/tests/test_pipeline_cli.py:75: AssertionError
------------------------------------------------------------- Captured stdout call -------------------------------------------------------------
2
------------------------------------------------------------- Captured stderr call -------------------------------------------------------------
2024-05-01 23:48:09,752 - INFO - moving fileHandler '/home/thdam/work/HistoQC/error.log' to '/tmp/pytest-of-thdam/pytest-0/test_cli_multiprocess0/error.log'
2024-05-01 23:48:09,792 - INFO - -----Working on:       /tmp/pytest-of-thdam/pytest-0/histoqc_test_multi0/1.svs         1 of 2
2024-05-01 23:48:09,792 - INFO - -----Working on:       /tmp/pytest-of-thdam/pytest-0/histoqc_test_multi0/2.svs         2 of 2
2024-05-01 23:48:09,796 - INFO - 2.svs -        getMag
2024-05-01 23:48:09,796 - INFO - 1.svs -        getMag
2024-05-01 23:48:09,995 - INFO - 1.svs -        getBasicStats
2024-05-01 23:48:09,995 - INFO - 1.svs -        ClassificationModule.byExample: pen_markings
2024-05-01 23:48:09,997 - INFO - 1.svs - Training model ClassificationModule.byExample:pen_markings
2024-05-01 23:48:10,003 - INFO - 2.svs -        getBasicStats
2024-05-01 23:48:10,003 - INFO - 2.svs -        ClassificationModule.byExample: pen_markings
2024-05-01 23:48:17,226 - INFO - 1.svs - Training model ClassificationModule.byExample:pen_markings....done
2024-05-01 23:48:17,346 - INFO - 1.svs -        ClassificationModule.byExample: coverslip_edge
2024-05-01 23:48:17,347 - INFO - 1.svs - Training model ClassificationModule.byExample:coverslip_edge
2024-05-01 23:48:17,366 - INFO - 2.svs -        ClassificationModule.byExample: coverslip_edge
2024-05-01 23:48:24,805 - INFO - 1.svs - Training model ClassificationModule.byExample:coverslip_edge....done
2024-05-01 23:48:24,993 - INFO - 1.svs -        LightDarkModule.getIntensityThresholdPercent:    bright
2024-05-01 23:48:24,996 - INFO - 1.svs -        LightDarkModule.getIntensityThresholdPercent:    dark
2024-05-01 23:48:24,998 - INFO - 1.svs -        BubbleRegionByRegion.detectSmoothness
2024-05-01 23:48:25,002 - INFO - 1.svs -        removeFatlikeTissue
2024-05-01 23:48:25,006 - INFO - 1.svs -        fillSmallHoles
2024-05-01 23:48:25,010 - INFO - 1.svs -        removeSmallObjects
2024-05-01 23:48:25,012 - INFO - 1.svs -        identifyBlurryRegions
2024-05-01 23:48:25,045 - INFO - 2.svs -        LightDarkModule.getIntensityThresholdPercent:    bright
2024-05-01 23:48:25,048 - INFO - 2.svs -        LightDarkModule.getIntensityThresholdPercent:    dark
2024-05-01 23:48:25,050 - INFO - 2.svs -        BubbleRegionByRegion.detectSmoothness
2024-05-01 23:48:25,055 - INFO - 2.svs -        removeFatlikeTissue
2024-05-01 23:48:25,060 - INFO - 2.svs -        fillSmallHoles
2024-05-01 23:48:25,063 - INFO - 2.svs -        removeSmallObjects
2024-05-01 23:48:25,065 - INFO - 2.svs -        identifyBlurryRegions
2024-05-01 23:48:25,232 - WARNING - 1.svs - After BlurDetectionModule.identifyBlurryRegions NO tissue remains detectable! Downstream modules likely to be incorrect/fail
2024-05-01 23:48:25,232 - INFO - 1.svs -        finalProcessingSpur
2024-05-01 23:48:25,235 - WARNING - 1.svs - After BasicModule.finalProcessingSpur NO tissue remains detectable! Downstream modules likely to be incorrect/fail
2024-05-01 23:48:25,235 - INFO - 1.svs -        finalProcessingArea
2024-05-01 23:48:25,236 - WARNING - 1.svs - After BasicModule.finalProcessingArea NO tissue remains detectable! Downstream modules likely to be incorrect/fail
2024-05-01 23:48:25,236 - INFO - 1.svs -        compareToTemplates
2024-05-01 23:48:25,236 - WARNING - 1.svs - HistogramModule.compareToTemplates NO tissue remains detectable in mask!
2024-05-01 23:48:25,236 - INFO - 1.svs -        getHistogram
2024-05-01 23:48:25,267 - WARNING - 2.svs - After BlurDetectionModule.identifyBlurryRegions NO tissue remains detectable! Downstream modules likely to be incorrect/fail
2024-05-01 23:48:25,267 - INFO - 2.svs -        finalProcessingSpur
2024-05-01 23:48:25,269 - WARNING - 2.svs - After BasicModule.finalProcessingSpur NO tissue remains detectable! Downstream modules likely to be incorrect/fail
2024-05-01 23:48:25,269 - INFO - 2.svs -        finalProcessingArea
2024-05-01 23:48:25,270 - WARNING - 2.svs - After BasicModule.finalProcessingArea NO tissue remains detectable! Downstream modules likely to be incorrect/fail
2024-05-01 23:48:25,270 - INFO - 2.svs -        compareToTemplates
2024-05-01 23:48:25,271 - WARNING - 2.svs - HistogramModule.compareToTemplates NO tissue remains detectable in mask!
2024-05-01 23:48:25,271 - INFO - 2.svs -        getHistogram
X connection to :0 broken (explicit kill or server shutdown).
X connection to :0 broken (explicit kill or server shutdown).
=========================================================== short test summary info ============================================================
FAILED histoqc/tests/test_pipeline_cli.py::test_cli_multiprocess - AssertionError: assert {'1.svs', '2...., 'error.log'} == {'1.svs', '2....'results.tsv'}
=================================================== 1 failed, 18 passed in 217.98s (0:03:37) ===================================================

This is the terminal output when I keyboard interrupt. It also shows "NO tissue remains detectable" as in the case of #278. All but the test_cli_multiprocess passed. The same test module gets stuck and never shows any error or debuggable message until interrupted.

I have been trying to create a new build with updated module versions, particularly scikit-image>0.20.0. I have also run the same build for the testing images put up in the conftest.py. All images except the CMU-1-Small-Region.svs passed. The failed image showed the same error as given above. Please help me regarding this.

When I tried to test it using the original build by cloning the repo once again, the UI failed. I believe the latest commit is not yet stable.

============================================================= test session starts ==============================================================
platform linux -- Python 3.10.14, pytest-7.1.3, pluggy-1.5.0 -- /home/thdam/miniconda3/envs/histo/bin/python
cachedir: .pytest_cache
rootdir: /home/thdam/work/original_histoqc/HistoQC, configfile: pyproject.toml
collected 14 items / 1 error

==================================================================== ERRORS ====================================================================
________________________________________________ ERROR collecting histoqc/tests/test_ui_cli.py _________________________________________________
ImportError while importing test module '/home/thdam/work/original_histoqc/HistoQC/histoqc/tests/test_ui_cli.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../../miniconda3/envs/histo/lib/python3.10/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
histoqc/tests/test_ui_cli.py:10: in <module>
    from histoqc.ui import _create_server
E   ImportError: cannot import name '_create_server' from 'histoqc.ui' (/home/thdam/miniconda3/envs/histo/lib/python3.10/site-packages/histoqc/ui/__init__.py)
=========================================================== short test summary info ============================================================
ERROR histoqc/tests/test_ui_cli.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
=============================================================== 1 error in 0.29s ===============================================================

[CielAl]

Currently the pytests test cases (including test_cli_multiprocess) simply check whether any exception is caught and can't be handled by HistoQC. That means even if there is no remaining tissue in the mask, the HistoQC does not consider it as an error, but rather give you a warning/notice and assume these are genuine results.

By default these tests case will use the config.ini under the config folder. You may try tweaking the parameters and see if it can detect anything. For instance, the test case uses an extremely small sample WSI but the gaussian filter radius in the identityBlurrRegions is quite huge.

[YoihenBachu]

Currently the pytests test cases (including test_cli_multiprocess) simply check whether any exception is caught and can't be handled by HistoQC. That means even if there is no remaining tissue in the mask, the HistoQC does not consider it as an error, but rather give you a warning/notice and assume these are genuine results.

By default these tests case will use the config.ini under the config folder. You may try tweaking the parameters and see if it can detect anything. For instance, the test case uses an extremely small sample WSI but the gaussian filter radius in the identityBlurrRegions is quite huge.

Thank you so much for your kind clarification. I understand now that the pytests test cases do not necessarily indicate the failure of HistoQC. I will surely try to tweak the config.ini.

The purpose of building a new HistoQC with bumped-up module versions is that many deprecated arguments are being used. And, recent software usually uses libraries, particularly scikit-image, with the newer versions and hence causes many dependency issues in the pipeline when I integrate HistoQC. Is it possible for me to bump the versions using the current build that I have tested?

Below are the versions tested

dill==0.3.8
matplotlib==3.8.4
numpy==1.26.4
openslide-python==1.2.0
pillow==10.3.0
pytest==8.1.1
scikit-image==0.23.2
scikit-learn==1.4.2
scipy==1.13.0
shapely==2.0.4

P.S. HistoQC is a wonderful software and we use it daily.

[CielAl]

Currently the pytests test cases (including test_cli_multiprocess) simply check whether any exception is caught and can't be handled by HistoQC. That means even if there is no remaining tissue in the mask, the HistoQC does not consider it as an error, but rather give you a warning/notice and assume these are genuine results.
By default these tests case will use the config.ini under the config folder. You may try tweaking the parameters and see if it can detect anything. For instance, the test case uses an extremely small sample WSI but the gaussian filter radius in the identityBlurrRegions is quite huge.

Thank you so much for your kind clarification. I understand now that the pytests test cases do not necessarily indicate the failure of HistoQC. I will surely try to tweak the config.ini.

The purpose of building a new HistoQC with bumped-up module versions is that many deprecated arguments are being used. And, recent software usually uses libraries, particularly scikit-image, with the newer versions and hence causes many dependency issues in the pipeline when I integrate HistoQC. Is it possible for me to bump the versions using the current build that I have tested?

Below are the versions tested

dill==0.3.8
matplotlib==3.8.4
numpy==1.26.4
openslide-python==1.2.0
pillow==10.3.0
pytest==8.1.1
scikit-image==0.23.2
scikit-learn==1.4.2
scipy==1.13.0
shapely==2.0.4

P.S. HistoQC is a wonderful software and we use it daily.

Hi,

I have introduced certain forward compatibility in the recent PRs which removes the deprecated signatures/methods and uses the latest/recommended way to invoke them. However this procedure may take sometime and we need to be cautious since there could be underlying implementation changes and therefore introduce discrepancy to numeric outputs. A typical example is that the Hessian computation in skimage.filters.frangi is rapidly changed from 0.19.x to 0.23.x and the corresponding frangi output of each version from 0.19.x to 0.23.x is different.

Currently the packages specified in requirements are mostly either not faraway from the latest version or did not break backward compatibility so it should be OK to use packages with higher versions. (But perhaps we can migrate python major versions from 3.7-3.8 to 3.8-3.10 since 3.7 is slowly being retired? @choosehappy )

[YoihenBachu]

Hi,

I have introduced certain forward compatibility in the recent PRs which removes the deprecated signatures/methods and uses the latest/recommended way to invoke them. However this procedure may take sometime and we need to be cautious since there could be underlying implementation changes and therefore introduce discrepancy to numeric outputs. A typical example is that the Hessian computation in skimage.filters.frangi is rapidly changed from 0.19.x to 0.23.x and the corresponding frangi output of each version from 0.19.x to 0.23.x is different.

Currently the packages specified in requirements are mostly either not faraway from the latest version or did not break backward compatibility so it should be OK to use packages with higher versions. (But perhaps we can migrate python major versions from 3.7-3.8 to 3.8-3.10 since 3.7 is slowly being retired? @choosehappy )

Can you provide me with the PRs which are being mentioned here? Also, what is the procedure recommended for migrating Python to 3.10?

[CielAl]

Hi,
I have introduced certain forward compatibility in the recent PRs which removes the deprecated signatures/methods and uses the latest/recommended way to invoke them. However this procedure may take sometime and we need to be cautious since there could be underlying implementation changes and therefore introduce discrepancy to numeric outputs. A typical example is that the Hessian computation in skimage.filters.frangi is rapidly changed from 0.19.x to 0.23.x and the corresponding frangi output of each version from 0.19.x to 0.23.x is different.
Currently the packages specified in requirements are mostly either not faraway from the latest version or did not break backward compatibility so it should be OK to use packages with higher versions. (But perhaps we can migrate python major versions from 3.7-3.8 to 3.8-3.10 since 3.7 is slowly being retired? @choosehappy )

Can you provide me with the PRs which are being mentioned here? Also, what is the procedure recommended for migrating Python to 3.10?

#290.

So when you migrating from py3.7 or 3.8 to 3.10, since that the big changes are really typing and structural pattern matching:
(1) For structural pattern matching (match case), there isn't anything you can do. But you can refactor certain blocks to a multiple if elif case which makes it easier for future refactoring to match cases.
(2) Use from future import annotations and typing_extensions to introduce py310 typing features into 3.8 first. This would make it easier when you finally migrate your python to 3.10.