Print output to debug the TestEvalModel unit test #21

rwest · 2021-04-09T14:23:50Z

This unit test is failing on Travis only on Python 3.6 but not Python 3.5 on PR #20,
and we can't make sense of why. This is mostly a "no change" pull request to trigger
the CI tests, but I added a print(output) statement to help debug.

This unit test is failing on Travis only on Python 3.6 but not Python 3.5 on PR #20, and we can't make sense of why. This is mostly a "no change" pull request to trigger the CI tests, but I added a print(output) statement to help debug.

rwest · 2021-04-09T14:51:39Z

On the Python 3.6 stack in Travis the tests fail .

______________________________ TestEvalModel.test ______________________________
2495
2496self = <pyteck.tests.test_eval_model.TestEvalModel object at 0x7fd809071a58>
2497
2498    def test(self):
2499        """Test overall evaluation of model.
2500        """
2501    
2502        with TemporaryDirectory() as temp_dir:
2503            output = eval_model.evaluate_model(
2504                model_name='h2o2.cti',
2505                spec_keys_file=self.relative_location('spec_keys.yaml'),
2506                dataset_file=self.relative_location('dataset_file.txt'),
2507                data_path=self.relative_location(''),
2508                model_path='',
2509                results_path=temp_dir,
2510                num_threads=1,
2511                skip_validation=True
2512                )
2513            print(output)
2514>           assert numpy.isclose(output['average error function'], 58.78211242028232, rtol=1.e-3)
2515E           assert False
2516E            +  where False = <function isclose at 0x7fd80f60f048>(58.882651168057734, 58.78211242028232, rtol=0.001)
2517E            +    where <function isclose at 0x7fd80f60f048> = numpy.isclose
2518
2519pyteck/tests/test_eval_model.py:223: AssertionError

We get this output:
{'model': 'h2o2.cti', 'datasets': [{'dataset': 'testfile_st.yaml', 'dataset_id': 0, 'standard deviation': 0.1, 'datapoints': [{'experimental ignition delay': '0.00047154 second', 'simulated ignition delay': '0.000999806876854139 second', 'temperature': '1164.48 kelvin', 'pressure': '220000.0 pascal', 'composition': [{'InChI': '1S/H2/h1H', 'species-name': 'H2', 'amount': '0.00444'}, {'InChI': '1S/O2/c1-2', 'species-name': 'O2', 'amount': '0.00556'}, {'InChI': '1S/Ar', 'species-name': 'Ar', 'amount': '0.99'}], 'composition type': 'mole fraction'}, {'experimental ignition delay': '0.00044803 second', 'simulated ignition delay': '0.0009972202043260416 second', 'temperature': '1164.97 kelvin', 'pressure': '220000.0 pascal', 'composition': [{'InChI': '1S/H2/h1H', 'species-name': 'H2', 'amount': '0.00444'}, {'InChI': '1S/O2/c1-2', 'species-name': 'O2', 'amount': '0.00556'}, {'InChI': '1S/Ar', 'species-name': 'Ar', 'amount': '0.99'}], 'composition type': 'mole fraction'}, {'experimental ignition delay': '0.00029157 second', 'simulated ignition delay': '0.0005734834427541362 second', 'temperature': '1264.2 kelvin', 'pressure': '220000.0 pascal', 'composition': [{'InChI': '1S/H2/h1H', 'species-name': 'H2', 'amount': '0.00444'}, {'InChI': '1S/O2/c1-2', 'species-name': 'O2', 'amount': '0.00556'}, {'InChI': '1S/Ar', 'species-name': 'Ar', 'amount': '0.99'}], 'composition type': 'mole fraction'}, {'experimental ignition delay': '0.00020593 second', 'simulated ignition delay': '0.000421523255121319 second', 'temperature': '1332.57 kelvin', 'pressure': '220000.0 pascal', 'composition': [{'InChI': '1S/H2/h1H', 'species-name': 'H2', 'amount': '0.00444'}, {'InChI': '1S/O2/c1-2', 'species-name': 'O2', 'amount': '0.00556'}, {'InChI': '1S/Ar', 'species-name': 'Ar', 'amount': '0.99'}], 'composition type': 'mole fraction'}, {'experimental ignition delay': '8.811e-05 second', 'simulated ignition delay': '0.0002116988585442861 second', 'temperature': '1519.18 kelvin', 'pressure': '220000.0 pascal', 'composition': [{'InChI': '1S/H2/h1H', 'species-name': 'H2', 'amount': '0.00444'}, {'InChI': '1S/O2/c1-2', 'species-name': 'O2', 'amount': '0.00556'}, {'InChI': '1S/Ar', 'species-name': 'Ar', 'amount': '0.99'}], 'composition type': 'mole fraction'}], 'error function': 58.882651168057734, 'absolute deviation': 7.64207193590874}], 'average error function': 58.882651168057734, 'error function standard deviation': 0.0, 'average deviation function': 7.64207193590874}

This adds a flag to pytest when running on Travis so that it reports the captured stdout even on passing unit tests, so we can compare.

rwest · 2021-04-09T16:52:59Z

The passing version, on Python 3.5, has this output:
{'datasets': [{'datapoints': [{'simulated ignition delay': '0.0009996030198605602 second', 'experimental ignition delay': '0.00047154 second', 'pressure': '220000.0 pascal', 'temperature': '1164.48 kelvin', 'composition': [{'species-name': 'H2', 'InChI': '1S/H2/h1H', 'amount': '0.00444'}, {'species-name': 'O2', 'InChI': '1S/O2/c1-2', 'amount': '0.00556'}, {'species-name': 'Ar', 'InChI': '1S/Ar', 'amount': '0.99'}], 'composition type': 'mole fraction'}, {'simulated ignition delay': '0.0009954820845162022 second', 'experimental ignition delay': '0.00044803 second', 'pressure': '220000.0 pascal', 'temperature': '1164.97 kelvin', 'composition': [{'species-name': 'H2', 'InChI': '1S/H2/h1H', 'amount': '0.00444'}, {'species-name': 'O2', 'InChI': '1S/O2/c1-2', 'amount': '0.00556'}, {'species-name': 'Ar', 'InChI': '1S/Ar', 'amount': '0.99'}], 'composition type': 'mole fraction'}, {'simulated ignition delay': '0.0005738392264714885 second', 'experimental ignition delay': '0.00029157 second', 'pressure': '220000.0 pascal', 'temperature': '1264.2 kelvin', 'composition': [{'species-name': 'H2', 'InChI': '1S/H2/h1H', 'amount': '0.00444'}, {'species-name': 'O2', 'InChI': '1S/O2/c1-2', 'amount': '0.00556'}, {'species-name': 'Ar', 'InChI': '1S/Ar', 'amount': '0.99'}], 'composition type': 'mole fraction'}, {'simulated ignition delay': '0.0004209878446795384 second', 'experimental ignition delay': '0.00020593 second', 'pressure': '220000.0 pascal', 'temperature': '1332.57 kelvin', 'composition': [{'species-name': 'H2', 'InChI': '1S/H2/h1H', 'amount': '0.00444'}, {'species-name': 'O2', 'InChI': '1S/O2/c1-2', 'amount': '0.00556'}, {'species-name': 'Ar', 'InChI': '1S/Ar', 'amount': '0.99'}], 'composition type': 'mole fraction'}, {'simulated ignition delay': '0.0002114426078483022 second', 'experimental ignition delay': '8.811e-05 second', 'pressure': '220000.0 pascal', 'temperature': '1519.18 kelvin', 'composition': [{'species-name': 'H2', 'InChI': '1S/H2/h1H', 'amount': '0.00444'}, {'species-name': 'O2', 'InChI': '1S/O2/c1-2', 'amount': '0.00556'}, {'species-name': 'Ar', 'InChI': '1S/Ar', 'amount': '0.99'}], 'composition type': 'mole fraction'}], 'dataset': 'testfile_st.yaml', 'absolute deviation': 7.634451187255257, 'dataset_id': 0, 'error function': 58.75871579871307, 'standard deviation': 0.1}], 'model': 'h2o2.cti', 'average deviation function': 7.634451187255257, 'error function standard deviation': 0.0, 'average error function': 58.75871579871307}

and here's the diff. annoyingly, the dictionaries are unsorted

1,10c1,5
< broken {
< 'model': 'h2o2.cti'
<  'datasets': [{
< 'dataset': 'testfile_st.yaml'
<  'dataset_id': 0
<  'standard deviation': 0.1
<  'datapoints': [{
< 'experimental ignition delay': '0.00047154 second'
<  'simulated ignition delay': '0.000999806876854139 second'
<  'temperature': '1164.48 kelvin'
---
> working {
> 'datasets': [{
> 'datapoints': [{
> 'simulated ignition delay': '0.0009996030198605602 second'
>  'experimental ignition delay': '0.00047154 second'
11a7
>  'temperature': '1164.48 kelvin'
13,14c9,10
< 'InChI': '1S/H2/h1H'
<  'species-name': 'H2'
---
> 'species-name': 'H2'
>  'InChI': '1S/H2/h1H'
18,19c14,15
< 'InChI': '1S/O2/c1-2'
<  'species-name': 'O2'
---
> 'species-name': 'O2'
>  'InChI': '1S/O2/c1-2'
23,24c19,20
< 'InChI': '1S/Ar'
<  'species-name': 'Ar'
---
> 'species-name': 'Ar'
>  'InChI': '1S/Ar'
30,32c26,27
< 'experimental ignition delay': '0.00044803 second'
<  'simulated ignition delay': '0.0009972202043260416 second'
<  'temperature': '1164.97 kelvin'
---
> 'simulated ignition delay': '0.0009954820845162022 second'
>  'experimental ignition delay': '0.00044803 second'
33a29
>  'temperature': '1164.97 kelvin'
35,36c31,32
< 'InChI': '1S/H2/h1H'
<  'species-name': 'H2'
---
> 'species-name': 'H2'
>  'InChI': '1S/H2/h1H'
40,41c36,37
< 'InChI': '1S/O2/c1-2'
<  'species-name': 'O2'
---
> 'species-name': 'O2'
>  'InChI': '1S/O2/c1-2'
45,46c41,42
< 'InChI': '1S/Ar'
<  'species-name': 'Ar'
---
> 'species-name': 'Ar'
>  'InChI': '1S/Ar'
52,54c48,49
< 'experimental ignition delay': '0.00029157 second'
<  'simulated ignition delay': '0.0005734834427541362 second'
<  'temperature': '1264.2 kelvin'
---
> 'simulated ignition delay': '0.0005738392264714885 second'
>  'experimental ignition delay': '0.00029157 second'
55a51
>  'temperature': '1264.2 kelvin'
57,58c53,54
< 'InChI': '1S/H2/h1H'
<  'species-name': 'H2'
---
> 'species-name': 'H2'
>  'InChI': '1S/H2/h1H'
62,63c58,59
< 'InChI': '1S/O2/c1-2'
<  'species-name': 'O2'
---
> 'species-name': 'O2'
>  'InChI': '1S/O2/c1-2'
67,68c63,64
< 'InChI': '1S/Ar'
<  'species-name': 'Ar'
---
> 'species-name': 'Ar'
>  'InChI': '1S/Ar'
74,76c70,71
< 'experimental ignition delay': '0.00020593 second'
<  'simulated ignition delay': '0.000421523255121319 second'
<  'temperature': '1332.57 kelvin'
---
> 'simulated ignition delay': '0.0004209878446795384 second'
>  'experimental ignition delay': '0.00020593 second'
77a73
>  'temperature': '1332.57 kelvin'
79,80c75,76
< 'InChI': '1S/H2/h1H'
<  'species-name': 'H2'
---
> 'species-name': 'H2'
>  'InChI': '1S/H2/h1H'
84,85c80,81
< 'InChI': '1S/O2/c1-2'
<  'species-name': 'O2'
---
> 'species-name': 'O2'
>  'InChI': '1S/O2/c1-2'
89,90c85,86
< 'InChI': '1S/Ar'
<  'species-name': 'Ar'
---
> 'species-name': 'Ar'
>  'InChI': '1S/Ar'
96,98c92,93
< 'experimental ignition delay': '8.811e-05 second'
<  'simulated ignition delay': '0.0002116988585442861 second'
<  'temperature': '1519.18 kelvin'
---
> 'simulated ignition delay': '0.0002114426078483022 second'
>  'experimental ignition delay': '8.811e-05 second'
99a95
>  'temperature': '1519.18 kelvin'
101,102c97,98
< 'InChI': '1S/H2/h1H'
<  'species-name': 'H2'
---
> 'species-name': 'H2'
>  'InChI': '1S/H2/h1H'
106,107c102,103
< 'InChI': '1S/O2/c1-2'
<  'species-name': 'O2'
---
> 'species-name': 'O2'
>  'InChI': '1S/O2/c1-2'
111,112c107,108
< 'InChI': '1S/Ar'
<  'species-name': 'Ar'
---
> 'species-name': 'Ar'
>  'InChI': '1S/Ar'
117,118c113,117
<  'error function': 58.882651168057734
<  'absolute deviation': 7.64207193590874
---
>  'dataset': 'testfile_st.yaml'
>  'absolute deviation': 7.634451187255257
>  'dataset_id': 0
>  'error function': 58.75871579871307
>  'standard deviation': 0.1
120c119,120
<  'average error function': 58.882651168057734
---
>  'model': 'h2o2.cti'
>  'average deviation function': 7.634451187255257
122,123c122,123
<  'average deviation function': 7.64207193590874
< }
\ No newline at end of file
---
>  'average error function': 58.75871579871307
> }
(default) Richards-iMac:Desktop rwest$

rwest · 2021-04-09T17:02:54Z

Converted them to json and used http://www.jsondiff.com
The simulated ignition delays are subtly different. Not a bug, just a precision difference.

I suggest we just relax the tolerance on the test?

Print output to debug the TestEvalModel unit test

fbc2879

This unit test is failing on Travis only on Python 3.6 but not Python 3.5 on PR #20, and we can't make sense of why. This is mostly a "no change" pull request to trigger the CI tests, but I added a print(output) statement to help debug.

rwest mentioned this pull request Apr 9, 2021

Fix for inf error when no ignition found #20

Open

rwest added 2 commits April 9, 2021 10:54

TemporaryMake pytest report captured stdout for passing tests in Travis

8d16f32

This adds a flag to pytest when running on Travis so that it reports the captured stdout even on passing unit tests, so we can compare.

fixup last commit

901332f

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Print output to debug the TestEvalModel unit test #21

Print output to debug the TestEvalModel unit test #21

rwest commented Apr 9, 2021

rwest commented Apr 9, 2021

rwest commented Apr 9, 2021

rwest commented Apr 9, 2021

Print output to debug the TestEvalModel unit test #21

Are you sure you want to change the base?

Print output to debug the TestEvalModel unit test #21

Conversation

rwest commented Apr 9, 2021

rwest commented Apr 9, 2021

rwest commented Apr 9, 2021

rwest commented Apr 9, 2021