Tesseract 5.0.1 test_LSTM_choices(...) fails

[simonflueckiger]

I compiled tesserocr 2.5.2 with Tesseract 5.0.1 on Windows. When executing tesserocr\tests\test_api.py I get the following exception for test_LSTM_choices(...):

FAIL: test_LSTM_choices (tests.test_api.TestTessBaseApi)
Test GetBestLSTMSymbolChoices.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tesserocr\tests\test_api.py", line 201, in test_LSTM_choices
    self.assertLessEqual(alternative[1], 2.0)
AssertionError: 3.621181011199951 not less than or equal to 2.0

Very similar to this #147 (comment). It passes when built with Tesseract 4.1.3. Does this also pass on Travis for Tesseract 5.x? I get a 404 when trying to access the build pipeline.

[srinivas1746]

Hi @simonflueckiger , can you please provide the instructions on how to build tesserocr with tesseract 5 .

[simonflueckiger]

@srinivas1746 you can have a look at my appveyor.yml from my repository tesserocr-windows_build.

[srinivas1746]

Hi @simonflueckiger , thank you for the update. In my case I want to build on UBUNTU system. Please let me know if there is any way?

[simonflueckiger]

CMake and vcpkg work very similarly on Linux, so if you want to build Tesseract 5.x from source you can use most of appveyor.yml with minor adaptations. To build tesserocr on Linux please refer to this README.

[srinivas1746]

@simonflueckiger thank you for you respose. I am able to build tesserocr + tesseract 5 with instructions in this link.
https://github.com/sirfz/tesserocr/blob/master/Windows.build.md#tesseract-build-and-installation

[simonflueckiger]

@sirfz has this been changed/fixed in the meantime?

[sirfz]

@simonflueckiger I saw the last comment about the working windows build, now I realize the issue is unrelated so I'm reopening it. Anyone with knowledge about GetBestLSTMSymbolChoices can chip in with the proper "fix" for this test case?

[makra89]

I also encounter a strange behavior when using tesserocr 2.5.2 together with tesseract 5.0.1
Besides the OCR result itself I also collect the LSTM symbol choices via GetBestLSTMSymbolChoices() to use them in a separate trie.
This works fine for tesserocr 2.5.1 and tesseract 4

After the switch I get strange confidence values, the symbol choices themselves seem to be okay.
An example:

• Best candidate string: 08/2004 , confidence 89.6
• Lstm choices: [[('0', 14.880019187927246), ('O', 32.32719802856445)], [('8', 0.5799692869186401), ('B', 30.601856231689453), ('3', 30.436098098754883)], [('/', 1.3585673570632935)], [('2', 1.2053536176681519), ('3', 34.48374557495117)], [('0', 1.0320600271224976)], [('0', 1.3039389848709106), ('9', 14.587406158447266), ('O', 22.922658920288086)], [('4', 1.349197268486023)]]

[sirfz]

I think we should change the test case, we shouldn't be testing tesseract's correctness, instead just test that tesserocr's API wrapping works.

It's not wrong to get different results (especially confidence scores) from different models/versions (like in this case with tesseract 5 vs 4) so we shouldn't expect it to be a static value or range

[makra89]

They scores didn't just change, I think they are wrong. If you look at the values I posted above, you can see that they increase with getting more unlikely. I first thought this could be a log-prob., but this neither makes sense.

What you are saying is that this may be a bug in Tesseract itself?

[ngeraks]

popos 22.04, tesseract 5.2, tesserocr main branch

======================================================================
ERROR: test_init (tests.test_api.TestTessBaseApi)
Test Init calls with different lang and oem.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ni/tesserocr/tests/test_api.py", line 98, in test_init
    self._api.Init(oem=tesserocr.OEM.TESSERACT_ONLY)
  File "tesserocr.pyx", line 1485, in tesserocr.PyTessBaseAPI.Init
    self._init_api(cpath, clang, oem, NULL, 0, NULL, NULL, False, PSM_AUTO)
  File "tesserocr.pyx", line 1233, in tesserocr.PyTessBaseAPI._init_api
    raise RuntimeError('Failed to init API, possibly an invalid tessdata path: {}'.format(path))
RuntimeError: Failed to init API, possibly an invalid tessdata path: /home/ni/tesseract-main/share/tessdata/

======================================================================
FAIL: test_LSTM_choices (tests.test_api.TestTessBaseApi)
Test GetBestLSTMSymbolChoices.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ni/tesserocr/tests/test_api.py", line 201, in test_LSTM_choices
    self.assertLessEqual(alternative[1], 2.0)
AssertionError: 3.578106641769409 not less than or equal to 2.0

======================================================================
FAIL: test_detect_os (tests.test_api.TestTessBaseApi)
Test DetectOS and DetectOrientationScript (tesseract v4+).
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ni/tesserocr/tests/test_api.py", line 235, in test_detect_os
    self.assertEqual(orientation["orientation"], 0)
AssertionError: 3 != 0

----------------------------------------------------------------------
Ran 24 tests in 7.739s

FAILED (failures=2, errors=1)
Test failed: <unittest.runner.TextTestResult run=24 errors=1 failures=2>
error: Test failed: <unittest.runner.TextTestResult run=24 errors=1 failures=2>

reinstalled and tried to use tesseract 4.1

======================================================================
ERROR: test_LSTM_choices (tests.test_api.TestTessBaseApi)
Test GetBestLSTMSymbolChoices.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ni/tesserocr/tests/test_api.py", line 202, in test_LSTM_choices
    chosen_symbol = timestep[0][0]
IndexError: list index out of range

======================================================================
ERROR: test_init (tests.test_api.TestTessBaseApi)
Test Init calls with different lang and oem.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ni/tesserocr/tests/test_api.py", line 98, in test_init
    self._api.Init(oem=tesserocr.OEM.TESSERACT_ONLY)
  File "tesserocr.pyx", line 1485, in tesserocr.PyTessBaseAPI.Init
    self._init_api(cpath, clang, oem, NULL, 0, NULL, NULL, False, PSM_AUTO)
  File "tesserocr.pyx", line 1245, in tesserocr.PyTessBaseAPI._init_api
    raise RuntimeError('Failed to init API, possibly an invalid tessdata path: {}'.format(path))
RuntimeError: Failed to init API, possibly an invalid tessdata path: /home/ni/tesseract-4.1/share/tessdata/

======================================================================
FAIL: test_detect_os (tests.test_api.TestTessBaseApi)
Test DetectOS and DetectOrientationScript (tesseract v4+).
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ni/tesserocr/tests/test_api.py", line 235, in test_detect_os
    self.assertEqual(orientation["orientation"], 0)
AssertionError: 3 != 0

----------------------------------------------------------------------

[makra89]

This seems to be an issue with a changed API in tesseract itself. See tesseract-ocr/tesseract#3706

In order to convert the scores to confidences you have to first fetch a variable:
lstm_rating = self._api.GetDoubleVariable('lstm_rating_coefficient')

Then you get the scores from tesserocr:
character_candidates = api.GetBestLSTMSymbolChoices()

Then you can convert the scores:
conf = 100 - lstm_rating * score