libiconv missing from uwsgi dependencies

[lsetiawan]

I get an error when trying to deploy wofpy via uwsgi.

$ uwsgi --socket 0.0.0.0:8080 --protocol=http -w wsgi
uwsgi: error while loading shared libraries: libiconv.so.2: cannot open shared object file: No such file or directory

To solve this, I had to do conda install libiconv. Seems like this dependency is missing from uwsgi, should it be included in wofpy?

[lsetiawan]

@ocefpaf Do you know if libiconv is included in uwsgi conda feedstock?

[ocefpaf]

It is the dependency of a dependency (libxml2 I guess) .

The problem is probably a bad mix of packages from defaults and conda-forge. I believe that defaults does not add libiconv to their stack.

Maybe conda-forge/uwsgi-feedstock#10 will fix it. (But things will only get stable when conda-forge moves to conda-build 3.)

[lsetiawan]

Thanks for figuring things out 😄

[emiliom]

Thanks @ocefpaf. Does this mean that it's safer for @lsetiawan to add that dependency explicitly in WOFpy requirements until conda-forge moves to conda-build 3? Or that we should wait (while crossing our fingers) to see if conda-forge/uwsgi-feedstock#10 will take care of it?

[ocefpaf]

Does this mean that it's safer for @lsetiawan to add that dependency explicitly in WOFpy requirements until conda-forge moves to conda-build 3?

In a way, yes :-( b/c even with conda-forge/uwsgi-feedstock#10, if defaults updates their version too, there will still be conda solver related problems like that and we cannot predict when defaults will updated.

[emiliom]

Ok, thanks. So be it.

[lsetiawan]

I tried deploying WOFpy again with uwsgi and I am still getting a libiconv error similar to above.

[ocefpaf]

I tried deploying WOFpy again with uwsgi and I am still getting a libiconv error similar to above.

That means you are using the conda-forge channel for deploying, right?
For Travis-CI we split the coding standards tests, that uses conda-forge to get those flake8 plugins, to avoid the conflict with the odm2 and defaults channels.

If you are using conda-forge I guess that the best solution would be to re-build the odm2 channel inside the conda-forge context. If you are not using conda-forge you should be able to use only defaults and odm2, but not the 3 of them.

The reason for this incompatibility is b/c the the odm2 channel is built against defaults packages.

PS: can you send me the details on how you are creating the deployment env?

[lsetiawan]

I did this:

1. conda create -n wofpytest -c odm2 python=2.7 --file requirements.txt -- file requirements-dev.txt

[ocefpaf]

What is your .condarc?

[lsetiawan]

I do not have one.

[ocefpaf]

Can you send me a conda list for that env? If you have conda-forge packages in there than it is related to the issue I mentioned above, if not than it is an Anaconda issue and we should report it upstream.

PS: if you do have conda-forge packages you somehow have a .condarc somewhere 😄

[lsetiawan]

I don't have any conda-forge packages within this environment. Only defaults and odm2. Should I be using conda-forge also? I thought I can't use all 3?

Conda List

# packages in environment at /home/lsetiawan/Miniconda/envs/wofpytest:
#
# Name                    Version                   Build  Channel
asn1crypto                0.24.0                   py27_0  
attrs                     17.4.0                   py27_0  
automat                   0.6.0            py27hfed452d_0  
ca-certificates           2017.08.26           h1d4fec5_0  
certifi                   2018.1.18                py27_0  
cffi                      1.11.4           py27h9745a5d_0  
chardet                   3.0.4            py27hfa10054_1  
click                     6.7              py27h4225b90_0  
configparser              3.5.0            py27h5117587_0  
constantly                15.1.0                   py27_0  
cryptography              2.1.4            py27h6697b16_0  
django                    1.11.8           py27hd476221_0  
docopt                    0.6.2                    py27_0  
enum34                    1.1.6            py27h99a27e9_1  
flask                     0.12.2           py27h6d5c1cd_0  
funcsigs                  1.0.2            py27h83f16ab_0  
geoalchemy-odm2           0.7.4                    py27_1    odm2
geos                      3.6.2                heeff764_2  
hyperlink                 17.1.1                   py27_0  
icu                       58.2                 h9c2bf20_1  
idna                      2.6              py27h5722d68_1  
incremental               17.5.0           py27h1923969_0  
intel-openmp              2018.0.0             hc7b2577_8  
ipaddress                 1.0.19                   py27_0  
itsdangerous              0.24             py27hb8295c1_1  
jinja2                    2.10             py27h4114e70_0  
krb5                      1.14.2               hcdc1b81_6  
libedit                   3.1                  heed3624_0  
libffi                    3.2.1                hd88cf55_4  
libgcc-ng                 7.2.0                h7cc24e2_2  
libgfortran-ng            7.2.0                h9f7466a_2  
libpq                     9.6.6                h4e02ad2_0  
libsodium                 1.0.15               hf101ebd_0  
libstdcxx-ng              7.2.0                h7a57d05_2  
libxml2                   2.9.7                h26e45fe_0  
libxslt                   1.1.32               h1312cb7_0  
lxml                      4.1.1            py27hf71bdeb_1  
markupsafe                1.0              py27h97b2822_1  
mkl                       2018.0.1             h19d6760_4  
msgpack-python            0.5.1            py27h6bb024c_0  
ncurses                   6.0                  h9df7e31_2  
nose                      1.3.7            py27heec2199_2  
numpy                     1.9.3            py27h7e35acb_3  
odm2api                   0.7.0                    py27_1    odm2
openssl                   1.0.2n               hb7f436b_0  
pandas                    0.21.1           py27h38cdd7d_0  
pastedeploy               1.5.2                    py27_1  
pcre                      8.41                 hc27e229_1  
pip                       9.0.1            py27ha730c48_4  
pluggy                    0.6.0            py27h1f4f128_0  
psycopg2                  2.7.3.2                  py27_0    odm2
py                        1.5.2            py27h203d672_0  
pycodestyle               2.3.1            py27h904819d_0  
pycparser                 2.18             py27hefa08c5_1  
pymysql                   0.7.11           py27ha7233d9_0  
pyodbc                    4.0.22           py27hf484d3e_0  
pyopenssl                 17.5.0           py27hcee3be0_0  
pyparsing                 2.2.0            py27hf1513f8_1  
pyramid                   1.5.7                    py27_0  
pysocks                   1.6.7            py27he2db6d2_1  
pytest                    3.3.2                    py27_0  
python                    2.7.14              h1571d57_29  
python-dateutil           2.6.0                    py27_1    odm2
pytz                      2017.3           py27h001bace_0  
pyyaml                    3.12             py27h2d70dd7_1  
pyzmq                     16.0.3           py27hc579512_0  
readline                  7.0                  ha6073c6_4  
repoze.lru                0.6                      py27_0  
requests                  2.18.4           py27hc5b0589_1  
setuptools                38.4.0                   py27_0  
shapely                   1.5.15               np19py27_0    odm2
simplejson                3.12.0           py27h264bab0_0  
six                       1.11.0           py27h5f960f1_1  
spyne                     2.12.14                  py27_2    odm2
sqlalchemy                1.2.1                    py27_0    odm2
sqlite                    3.22.0               h1bed415_0  
suds-jurko                0.6                      py27_1    odm2
tk                        8.6.7                hc745277_3  
translationstring         1.3                      py27_0  
twisted                   17.5.0                   py27_0  
unixodbc                  2.3.4                hc36303a_1  
urllib3                   1.22             py27ha55213b_0  
uwsgi                     2.0.15                   py27_0    odm2
venusian                  1.0                      py27_0  
webob                     1.4.1                    py27_0  
werkzeug                  0.14.1                   py27_0  
wheel                     0.30.0           py27h2bc6bb2_1  
WOFpy                     2.3.0+9.g2d9c633           <pip>
xz                        5.2.3                h55aa19d_2  
yaml                      0.1.7                had09818_2  
zeromq                    4.2.2                hbedb6e5_2  
zlib                      1.2.11               ha838bed_2  
zope                      1.0                      py27_0  
zope.deprecation          4.1.2                    py27_1  
zope.interface            4.1.2                    py27_1 

[ocefpaf]

This is what I feared 😒

Let me go over all the packages in odm2 to be sure where is contamination is coming from but I think the issue is in defaults.

[lsetiawan]

Trying to understand here. So should the defaults channel say defaults in the channel column? I don't really see anywhere that specify libiconv to be installed. How come when I do conda install libiconv things are fixed?

[ocefpaf]

Trying to understand here. So should the defaults channel say defaults in the channel column?

Only if you have the option show_channels true, they decided that the implicit channel is always default 😒

I don't really see anywhere that specify libiconv to be installed. How come when I do conda install libiconv things are fixed?

The thing is, uwsgi is built against libxml2 and at the time of the build libxml2 from defautls linked libxml2 with to libiconv. However, it seems that they no longer do that and uwsgi is broken unless you force install libiconv.

TL;DR we need to rebuild uwsgi to make it link with the current versions of libxml2 and libiconv.

Issues like this one is the reason we created conda-forge, so the stack does not change under us.

[emiliom]

Issues like this one is the reason we created conda-forge, so the stack does not change under us.

Yup, I caught that ... I'll follow up with the ODM2 gang soon.

[lsetiawan]

Thanks for the explanations @ocefpaf. That was helpful!