Skip dependencies in SOK installation

[edknv]

gpu-multi / tensorflow (pull_request) in the CI fails with:

FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/lib/python3.8/dist-packages/horovod/openmpi_dist/bin/mpirun'

in #1110 for example, because SOK requires horovod and reinstalls it, overwriting the horovod installation in the ci-runner. This PR fixes that by adding --no-deps so that SOK does not install dependencies. We run setup.py in development mode because python install setup.py does not support --no-deps.

[github-actions]

Documentation preview

https://nvidia-merlin.github.io/models/review/pr-1138

[oliverholworthy]

if horovod is installed already in the container, I wonder why this is trying to install horovod again. does it require a higher version that what we have already?

[edknv]

I wasn't accurate when I said it's trying to install horovod again. setup.py detects that horovod is already installed from site_packages and skips installing the core horovod library it doesn't re-install horovod per se. But it does install some entrypoint scripts:

Installing horovodrun script to /home/gha-user/gha3/models/models/.tox/py38-multi-gpu/bin
Installing mpiexec script to /home/gha-user/gha3/models/models/.tox/py38-multi-gpu/bin
Installing mpirun script to /home/gha-user/gha3/models/models/.tox/py38-multi-gpu/bin
Installing ompi_info script to /home/gha-user/gha3/models/models/.tox/py38-multi-gpu/bin
Installing orted script to /home/gha-user/gha3/models/models/.tox/py38-multi-gpu/bin
Installing orterun script to /home/gha-user/gha3/models/models/.tox/py38-multi-gpu/bin

These are all installed in the tox environment .tox/py38-multi-gpu/bin, and horovod being from site_packages (i.e., /usr/local/bin) causes the issue. So, this issue is due to tox and mixing external commands inside the tox environment, and this shouldn't happen in normal environments.

[oliverholworthy]

thanks for the explanation @edknv