pyinstaller hidden-imports for spacy deployment (windows)

[DarkSoliditi]

Hey, I'm having a difficult time finding information for making pyinstaller deploy spacy dependencies for windows. When I run
pyinstaller main.py --noconsole and run that everything in my app works except for the spacy functionality. I don't get an error or anything it just hits the spacy lines of code and silently fails.

I've tried to make a hidden-imports .py file. (code attached) but when I run pyinstaller main.py --additional-hooks-dir=. myhooks.py
it says pyinstaller: error: unrecognized arguments: myhooks.py

I'm not sure what I'm doing wrong or if I'm on the right path to get this working or not. I'm not looking for anyone to do the work for me but some direction would be appreciated.

Thanks in advance.

HOOK FILE FOR SPACY

from PyInstaller.utils.hooks import collect_all

----------------------------- SPACY -----------------------------

data = collect_all('spacy')

datas = data[0]
binaries = data[1]
hiddenimports = data[2]

----------------------------- THINC -----------------------------

data = collect_all('thinc')

datas += data[0]
binaries += data[1]
hiddenimports += data[2]

----------------------------- CYMEM -----------------------------

data = collect_all('cymem')

datas += data[0]
binaries += data[1]
hiddenimports += data[2]

----------------------------- PRESHED -----------------------------

data = collect_all('preshed')

datas += data[0]
binaries += data[1]
hiddenimports += data[2]

----------------------------- BLIS -----------------------------

data = collect_all('blis')

datas += data[0]
binaries += data[1]
hiddenimports += data[2]

----------------------------- OTHER ----------------------------

hiddenimports += ['srsly.msgpack.util']

from PyInstaller.utils.hooks import collect_data_files
datas = collect_data_files("en_core_web_trf")

[rokm]

When I run
pyinstaller main.py --noconsole and run that everything in my app works except for the spacy functionality. I don't get an error or anything it just hits the spacy lines of code and silently fails.

Can you try building without --noconsole, so you can see if there are any error messages in the console?

I've tried to make a hidden-imports .py file. (code attached) but when I run pyinstaller main.py --additional-hooks-dir=. myhooks.py
it says pyinstaller: error: unrecognized arguments: myhooks.py

If you want to create custom hooks, the naming convention is hook-<modulename>.py, and you need to pass the directory containing the extra hook files to --additional-hooks-dir.

pyinstaller main.py --additional-hooks-dir=. myhooks.py

You have a space between . and myhooks.py, so they ended up being treated as two separate arguments; and since script name was already passed (main.py), the myhooks.py argument is unrecognized.

---

That said, recent versions of pyinstaller-hooks-contrib (from v.2021.2 on; pyinstaller/pyinstaller-hooks-contrib#128) should contain hooks for spacy.

What version of pyinstaller and pyinstaller-hooks-contrib are you using?

[DarkSoliditi]

In this case, you might want to try --collect-submodules en_core_web_trf --collect-data en_core_web_trf --copy-metadata en_core_web_trf.

This got me progress! I have a new error code now though.

Do I need to run the collect submodules, collect data, & copy metadata for spacy-transformers or on all the available factories? or is there another way to package all these?

Exception Callback attached:

Exception in Tkinter callback
Traceback (most recent call last):
File "tkinter_init_.py", line 1892, in call
File "main.py", line 78, in tesseract
File "spacy_init_.py", line 51, in load
File "spacy\util.py", line 321, in load_model
File "spacy\util.py", line 354, in load_model_from_package
File "en_core_web_trf_init_.py", line 10, in load
File "spacy\util.py", line 514, in load_model_from_init_py
File "spacy\util.py", line 389, in load_model_from_path
File "spacy\util.py", line 426, in load_model_from_config
File "spacy\language.py", line 1715, in from_config
File "spacy\language.py", line 777, in add_pipe
File "spacy\language.py", line 641, in create_pipe
ValueError: [E002] Can't find factory for 'transformer' for language English (en
). This usually happens when spaCy calls nlp.create_pipe with a custom compone
nt name that's not registered on the current language class. If you're using a T
ransformer, make sure to install 'spacy-transformers'. If you're using a custom
component, make sure you've added the decorator @Language.component (for funct
ion components) or @Language.factory (for class components).

Available factories: attribute_ruler, tok2vec, merge_noun_chunks, merge_entities
, merge_subtokens, token_splitter, parser, beam_parser, entity_linker, ner, beam
_ner, entity_ruler, lemmatizer, tagger, morphologizer, senter, sentencizer, text
cat, spancat, textcat_multilabel, en.lemmatizer

[rokm]

You need to add --collect-submodules spacy_transformers --copy-metadata spacy_transformers - the former because these (sub)modules are not imported directly but rather via the indirect factory mechanism (so they fly under our radar), and the latter because the factory mechanism depends on the packages' metadata.

But I see that installing en_core_web_trf also pulls in transformers, which will cause additional problems...

[rokm]

For transformers, see #6033.

[rokm]

So for a basic program:

import spacy

nlp = spacy.load("en_core_web_trf")
doc = nlp("Apple is looking at buying U.K. startup for $1 billion")
for token in doc:
    print(token.text, token.pos_, token.dep_)

the following is needed:

pyinstaller --clean --noconfirm program.py  ^
  --collect-submodules en_core_web_trf --collect-data en_core_web_trf --copy-metadata en_core_web_trf ^
  --collect-submodules spacy_transformers --copy-metadata spacy_transformers ^
  --copy-metadata tqdm --copy-metadata regex --copy-metadata sacremoses --copy-metadata requests --copy-metadata packaging --copy-metadata filelock --copy-metadata numpy --copy-metadata tokenizers ^
  --collect-data sacremoses ^
  --collect-all transformers

The first series of collects is what you already know; the second one is for spacy_transformers package. The third series is metadata for packages whose versions transformers validates at runtime; plus sacremoses needs it data files.

The last, brute-force collect-all is there in lieu of collect_data_files with include_py_files=True (for which we don't have command-line switch), because transformers' torch backend requires the source files.

[DarkSoliditi]

You are a miracle worker! This got it working for me! I can't thank you enough! You are awesome!