add support for pip install of odoo and addons #62

JordiBForgeFlow · 2018-01-16T16:29:48Z

This is an initial attempt to make the project compatible with the pip install approach proposed here: https://odoo-community.org/blog/our-blog-1/post/installing-oca-addons-the-easy-way-32

I was not able to successfully make it work, but hopefully this can be an starting point for an ongoing discussion.

sebalix · 2018-01-17T11:19:38Z

Hi Jordi, thank you for this first PR!
Travis is not happy because the LXD PPA has been deprecated and removed, I'm trying to fix that at first.

sebalix · 2018-01-17T11:22:00Z

vars/main.yml

@@ -14,6 +14,7 @@ odoo_pypi_packages:
    - psycogreen

 odoo_buildout_venv_cmd: "virtualenv --no-setuptools {{ odoo_buildout_venv_path }}"
+odoo_venv_cmd: "virtualenv {{ odoo_venv_path }}"


Rename to odoo_pip_venv_cmd

sebalix · 2018-01-17T11:22:24Z

defaults/main.yml

@@ -105,6 +105,9 @@ odoo_config_custom: {}
    #your_option1: value1
    #your_option2: value2

+# Pip installation options (odoo_install_type == 'pip')
+odoo_venv_path: "{{ odoo_workdir }}/sandbox"


Rename to odoo_pip_venv_path

JordiBForgeFlow · 2018-01-18T11:17:03Z

@sebalix Thanks! I have applied the fixes as suggested.

sebalix · 2018-01-19T14:53:36Z

@jbeficent can you remove .idea/* files and rebase on master? Tests under LXD are fixed now.

JordiBForgeFlow · 2018-01-21T08:48:56Z

This is now looking better. I added tests for the pip install method. Still have to resolve why in some versions is failing

sebalix · 2018-01-22T08:57:28Z

Nice work!
About failures on Jessie and Trusty, I think it is due to the version of setuptools copied into the virtualenv:

5.5 on Jessie: https://packages.debian.org/jessie/python-setuptools
3.3 on Trusty: https://packages.ubuntu.com/trusty/python-setuptools

EDIT: maybe not needed, see the next comment
We can add a task after the one which generates the virtualenv to upgrade setuptools as pinned by the requirements.txt file, something like this:
pip install --upgrade=setuptools -r requirements.txt (don't know if it works, need test)

sebalix · 2018-01-22T09:01:16Z

Or try to generate the virtualenv without setuptools (--no-setuptools) in the odoo_pip_venv_cmd option, then setuptools will be installed in the right version I think.

sebalix · 2018-01-22T09:14:45Z

templates/odoo-10.0.conf

@@ -1,5 +1,5 @@
 [options]
-addons_path = {{ odoo_config_addons_path.__class__.__name__ == 'list' and odoo_config_addons_path | join(',') or odoo_config_addons_path }}
+addons_path = {{ odoo_config_addons_path.__class__.__name__ == 'list' and odoo_config_addons_path | join(',') or odoo_install_type == 'standard' and odoo_config_addons_path or ''}}


addons_path can be used as a list with the standard install type:

addons_path: - /path/1 - /path/2

Or with a string as usual.
To fix that with pip install type, I would prefer to be more explicit:

{% if odoo_install_type == 'pip' %} addons_path = {% else %} addons_path = {{ odoo_config_addons_path.__class__.__name__ == 'list' and odoo_config_addons_path | join(',') or odoo_config_addons_path }} {% endif %}

sebalix · 2018-01-22T09:15:31Z

templates/odoo-11.0.conf

@@ -1,5 +1,5 @@
 [options]
-addons_path = {{ odoo_config_addons_path.__class__.__name__ == 'list' and odoo_config_addons_path | join(',') or odoo_config_addons_path }}
+addons_path = {{ odoo_config_addons_path.__class__.__name__ == 'list' and odoo_config_addons_path | join(',') or odoo_install_type == 'standard' and odoo_config_addons_path or ''}}


Same comment.

sebalix · 2018-01-22T09:15:33Z

templates/odoo-9.0.conf

@@ -1,5 +1,5 @@
 [options]
-addons_path = {{ odoo_config_addons_path.__class__.__name__ == 'list' and odoo_config_addons_path | join(',') or odoo_config_addons_path }}
+addons_path = {{ odoo_config_addons_path.__class__.__name__ == 'list' and odoo_config_addons_path | join(',') or odoo_install_type == 'standard' and odoo_config_addons_path or ''}}


Same comment.

sebalix · 2018-01-22T09:16:19Z

templates/odoo-8.0.conf

@@ -1,5 +1,5 @@
 [options]
-addons_path = {{ odoo_config_addons_path.__class__.__name__ == 'list' and odoo_config_addons_path | join(',') or odoo_config_addons_path }}
+addons_path = {{ odoo_config_addons_path.__class__.__name__ == 'list' and odoo_config_addons_path | join(',') or odoo_install_type == 'standard' and odoo_config_addons_path or ''}}


Same comment.

sebalix · 2018-01-22T09:17:03Z

vars/main.yml

@@ -14,6 +14,7 @@ odoo_pypi_packages:
    - psycogreen

 odoo_buildout_venv_cmd: "virtualenv --no-setuptools {{ odoo_buildout_venv_path }}"
+odoo_pip_venv_cmd: "virtualenv {{ odoo_pip_venv_path }}"


To try with --no-setuptools?

sebalix · 2018-01-22T09:18:15Z

vars/Ubuntu-16_Odoo-8.yml

@@ -52,6 +72,8 @@ odoo_buildout_build_dependencies:
    - libfreetype6-dev
    - liblcms2-dev
    - libwebp-dev
+    - libssl-dev
+    - python-wrapt


Why these packages are added to odoo_buildout_build_dependencies? (especially python-wrapt)

sebalix · 2018-01-22T09:18:49Z

vars/Ubuntu-16_Odoo-11.yml

@@ -87,6 +107,7 @@ odoo_wkhtmltox_depends:
    - libjpeg-turbo8

 odoo_buildout_venv_cmd: "virtualenv --no-setuptools --python=python3 {{ odoo_buildout_venv_path }}"
+odoo_pip_venv_cmd: "virtualenv --python=python3 {{ odoo_pip_venv_path }}"


To try with --no-setuptools.

sebalix · 2018-01-22T09:20:28Z

vars/Debian-9_Odoo-11.yml

@@ -66,6 +86,7 @@ odoo_buildout_build_dependencies:
    - libwebp-dev

 odoo_buildout_venv_cmd: "python3 -m virtualenv --no-setuptools --python=python3 {{ odoo_buildout_venv_path }}"
+odoo_pip_venv_cmd: "python3 -m virtualenv --python=python3 {{ odoo_pip_venv_path }}"


To try with --no-setuptools (if it works we will reduce the dependency against an old version of setuptools provided by Debian Stretch).

sebalix · 2018-01-22T09:20:57Z

tests/test_pip_changed.yml

+    - ansible-odoo
+  vars:
+    odoo_install_type: pip
+    odoo_repo_type: pip


Is odoo_repo_type useful?

sebalix · 2018-01-22T09:21:04Z

tests/test_pip.yml

+    - ansible-odoo
+  vars:
+    odoo_install_type: pip
+    odoo_repo_type: pip


Is odoo_repo_type useful?

The problem is that we do not want to execute the task 'Clone project repository (Git)' https://github.com/osiell/ansible-odoo/blob/master/tasks/install.yml#L52. How can we ensure that this part is skipped when we install with pip?

Oh ok,
Then you can replace the line you point by this:

when: odoo_install_type != 'pip' and odoo_repo_type == 'git' and odoo_repo_url

And do the same with https://github.com/osiell/ansible-odoo/blob/master/tasks/install.yml#L37 for Mercurial repositories.

JordiBForgeFlow · 2018-01-22T16:05:09Z

@sebalix I pushed one more fix and now Travis complains everywhere. Not sure why :S

sebalix · 2018-01-22T16:15:26Z

vars/Debian-8_Odoo-10.yml

+    - setuptools==30.0.0
+    - odoo-autodiscover==2.0.0
+    - setuptools-git==1.2
+    - setuptools-odoo==2.0.2.post1


Travis fails because the standard install type is using odoo_pypi_packages to install some packages (not available as a Debian package) at the system level with pip.
Re-using this for others purposes is not ideal, you should use another variable for that, e.g odoo_pip_dependencies (in order to be consistent with odoo_pip_build_dependencies)

sebalix · 2018-01-22T16:47:25Z

@jbeficent I wonder why you can't put the content of odoo_pip_dependencies in the requirements.txt file as it was done before? You encountered some problems to do it that way?

The problem I see on Travis is that Ansible installs dependencies at first from odoo_pip_dependencies, then it applies the requirements.txt file (which can upgrade some packages to a different version). So, if we run the playbook a second time, it will downgrade packages when applying odoo_pip_dependencies, to re-upgrade them with the requirements.txt, resulting in a "changed" flag (not idempotent).

sebalix · 2018-01-23T14:16:01Z

@jbeficent done

JordiBForgeFlow · 2018-01-23T14:23:00Z

This is the error without wrapt

failed: [localhost] (item=odoo-autodiscover==2.0.0) => {"changed": false, "cmd": "/home/odoo/odoo/sandbox/bin/pip2 install odoo-autodiscover==2.0.0", "item": "odoo-autodiscover==2.0.0", "msg": "stdout: Downloading/unpacking odoo-autodiscover==2.0.0\n Downloading odoo-autodiscover-2.0.0.tar.gz\n Running setup.py (path:/tmp/pip-build-loNmKv/odoo-autodiscover/setup.py) egg_info for package odoo-autodiscover\n /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires'\n warnings.warn(msg)\n your setuptools is too old (<12)\n setuptools_scm functionality is degraded\n \nDownloading/unpacking wrapt (from odoo-autodiscover==2.0.0)\n Downloading wrapt-1.10.11.tar.gz\n Running setup.py (path:/tmp/pip-build-loNmKv/wrapt/setup.py) egg_info for package wrapt\n \nInstalling collected packages: odoo-autodiscover, wrapt\n Running setup.py install for odoo-autodiscover\n /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires'\n warnings.warn(msg)\n your setuptools is too old (<12)\n setuptools_scm functionality is degraded\n changing mode of build/scripts-2.7/odoo-autodiscover.py from 644 to 755\n \n changing mode of /home/odoo/odoo/sandbox/bin/odoo-autodiscover.py to 755\n Installing openerp-server-autodiscover script to /home/odoo/odoo/sandbox/bin\n Installing odoo-server-autodiscover script to /home/odoo/odoo/sandbox/bin\n Running setup.py install for wrapt\n Traceback (most recent call last):\n File \"/home/odoo/odoo/sandbox/lib/python2.7/site.py\", line 703, in <module>\n main()\n File

JordiBForgeFlow · 2018-01-23T14:26:07Z

I have now pushed again, with wrapt.

sebalix · 2018-01-23T14:26:30Z

Okay... Very strange, it tries to install it from odoo-autodiscover but this results in an error :/
Thank you!

JordiBForgeFlow · 2018-01-23T14:27:07Z

Can you cancel prior travis builds?

sebalix · 2018-01-23T14:27:28Z

Yep, done

sebalix · 2018-01-23T14:56:39Z

To speed up tests, we could download Odoo sources from a .tar.gz (e.g. https://github.com/odoo/odoo/archive/10.0.tar.gz#egg=odoo , ~100Mb) which is must faster than downloading the whole Odoo git repository (2,5Go I think?).
The drawback is that we can not have the package in a editable format, but we don't care for CI.

- -e git+https://github.com/odoo/[email protected]#egg=odoo
+ https://github.com/odoo/odoo/archive/10.0.tar.gz#egg=odoo

JordiBForgeFlow · 2018-01-23T15:01:12Z

Then we'd need a requirements.txt file just for the tests, right?

sebalix · 2018-01-23T15:01:27Z

If Odoo 10.0 deployment is working on Debian Stretch but not on Debian Jessie, maybe it's due to the pip version.
pip must be upgraded inside the virtualenv before doing any other operation.

JordiBForgeFlow · 2018-01-23T15:02:26Z

We can try, but perhaps we loose the idempotency. We can see what is the result of the current testing

sebalix · 2018-01-23T15:04:26Z

@jbeficent the ones you wrote are enough (templates/odoo-X-pip-requirements.txt) no?
In real cases, users will wrote their own requirements.txt file, editable mode can be used there.

sebalix · 2018-01-23T15:18:34Z

Regarding pip, we can add pip==9.0.1 in odoo_pip_dependencies for all Debian Jessie and Ubuntu Trusty environments (and maybe Ubuntu Xenial as it supplies pip==8.1.1) to ensure the idempotency.

The great advantage with a recent version of pip is the use of Wheel package format which also speed up the installation of some packages (e.g. lxml, psycopg2, gevent, greenlet...). I'm using this in the buildout example recipe used by this role: https://github.com/osiell/odoo-buildout-example/blob/8.0/buildout.cfg
The commit which allows the use of wheel package format (not available for old package versions): osiell/odoo-buildout-example@f35a6f9

JordiBForgeFlow · 2018-01-23T15:50:28Z

OK, I have applied the changes you suggest, including downloading odoo from tar.gz. Let's see the results now..

sebalix · 2018-01-23T15:58:57Z

tasks/install_pip.yml

+        cache_valid_time={{ odoo_apt_cache_valid_time }}
+  with_items: "{{ odoo_debian_packages }}"
+  tags:
+    - odoo_packages


I didn't see this part, can we remove it? There is only python-dev which seems useful, and it is already included in odoo_pip_build_dependencies.

Done. Please cancel the previous build. I am not yet familiar with tags and what they are useful for.

Sorry, but I was referring to the whole "Install Odoo dependencies" task which process odoo_debian_packages :)
About Ansible tags, it allows to execute only some parts of the role via the ansible-playbook command with the -t option, for instance -t odoo_config,odoo_postgresql.

Ohh, Ok. I just removed it. Somehow it was requiring on a previous test. But let's see now.

sebalix · 2018-01-23T16:19:24Z

I know I'm annoying, there is only the .gitignore and .idea folder to remove and this PR will be almost ready to be merged.
I still have some ideas to improve this feature, like the possibility to keep the task to clone a project repository which could contains a requirements.txt file + customer addons (to have only one repo to manage all the customer specific code), but that can be done in another PR!

JordiBForgeFlow · 2018-01-23T16:53:30Z

@sebalix Ok, fixed! I wonder why https://travis-ci.org/osiell/ansible-odoo/jobs/332368800 failed. Seems not really related to the configuration, but a generic server error.

sebalix · 2018-01-23T16:57:46Z

Yes, it is not related to the PR, this is the LXD image mirror which was down.
Unable to connect to: uk.images.linuxcontainers.org:443
I'm waiting the end of the other jobs to check the total amount of time it takes to run now. I'll try to reduce the time it needs in another PR, at least on Travis CI, but pip install could be the fastest way to deploy Odoo.

sebalix · 2018-01-23T17:01:11Z

vars/Ubuntu-16_Odoo-10.yml

@@ -67,6 +90,7 @@ odoo_buildout_build_dependencies:
    - libfreetype6-dev
    - liblcms2-dev
    - libwebp-dev
+    - python-wrapt


Ah, I forgot this one ;) To remove please.

JordiBForgeFlow · 2018-01-23T17:13:36Z

@sebalix People can also specify https://wheelhouse.odoo-community.org/oca-simple/ in their requirements.txt file, and that would also speed up the install significantly.

Another interesting idea is to maintain a set of already packaged wheels for the python packages often used, like in https://wheelhouse.acsone.eu/odoo-requirements-ubuntu.

This would help to greatly improve the speed to install.

JordiBForgeFlow · 2018-01-23T17:14:25Z

The problem I see with https://wheelhouse.acsone.eu/odoo-requirements-ubuntu/ is that you would have to trust the sources. With the pip install method it is not easy to verify if the packages are safe.

sebalix · 2018-01-23T17:17:52Z

Great to know, there is space for improvements!
Anyway, this is a nice contribution, thank you! Can I merge the PR?

JordiBForgeFlow · 2018-01-23T17:29:17Z

Good to merge for me! Thanks for your review!!!!!

sebalix reviewed Jan 17, 2018

View reviewed changes

JordiBForgeFlow force-pushed the odoo_install_type_pip branch 15 times, most recently from 7820447 to fa36629 Compare January 21, 2018 06:36

JordiBForgeFlow force-pushed the odoo_install_type_pip branch from fa36629 to 75f1434 Compare January 21, 2018 14:44

JordiBForgeFlow closed this Jan 21, 2018

JordiBForgeFlow reopened this Jan 21, 2018

sebalix requested changes Jan 22, 2018

View reviewed changes

JordiBForgeFlow force-pushed the odoo_install_type_pip branch from 75f1434 to 6d681a3 Compare January 22, 2018 15:29

sebalix reviewed Jan 22, 2018

View reviewed changes

JordiBForgeFlow force-pushed the odoo_install_type_pip branch from cc3e892 to 4c78794 Compare January 23, 2018 14:25

JordiBForgeFlow force-pushed the odoo_install_type_pip branch from 4c78794 to 233c7a2 Compare January 23, 2018 15:49

sebalix reviewed Jan 23, 2018

View reviewed changes

JordiBForgeFlow force-pushed the odoo_install_type_pip branch 2 times, most recently from 2b81d21 to 5a82524 Compare January 23, 2018 16:07

JordiBForgeFlow force-pushed the odoo_install_type_pip branch from 5a82524 to d3bdb4a Compare January 23, 2018 16:52

sebalix reviewed Jan 23, 2018

View reviewed changes

add support for pip install of odoo and addons

440147c

JordiBForgeFlow force-pushed the odoo_install_type_pip branch from d3bdb4a to 440147c Compare January 23, 2018 17:04

sebalix merged commit c1f8496 into OCA:master Jan 23, 2018

add support for pip install of odoo and addons #62

add support for pip install of odoo and addons #62

Conversation

JordiBForgeFlow commented Jan 16, 2018

sebalix commented Jan 17, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JordiBForgeFlow commented Jan 18, 2018

sebalix commented Jan 19, 2018

JordiBForgeFlow commented Jan 21, 2018

sebalix commented Jan 22, 2018 • edited Loading

sebalix commented Jan 22, 2018

sebalix Jan 22, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sebalix Jan 22, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JordiBForgeFlow commented Jan 22, 2018

Choose a reason for hiding this comment

sebalix commented Jan 22, 2018 • edited Loading

sebalix commented Jan 23, 2018

JordiBForgeFlow commented Jan 23, 2018

JordiBForgeFlow commented Jan 23, 2018

sebalix commented Jan 23, 2018

JordiBForgeFlow commented Jan 23, 2018

sebalix commented Jan 23, 2018

sebalix commented Jan 23, 2018 • edited Loading

JordiBForgeFlow commented Jan 23, 2018

sebalix commented Jan 23, 2018

JordiBForgeFlow commented Jan 23, 2018

sebalix commented Jan 23, 2018

sebalix commented Jan 23, 2018

JordiBForgeFlow commented Jan 23, 2018

sebalix Jan 23, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sebalix commented Jan 23, 2018 • edited Loading

JordiBForgeFlow commented Jan 23, 2018

sebalix commented Jan 23, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JordiBForgeFlow commented Jan 23, 2018

JordiBForgeFlow commented Jan 23, 2018

sebalix commented Jan 23, 2018

JordiBForgeFlow commented Jan 23, 2018

sebalix commented Jan 22, 2018 •

edited

Loading

sebalix Jan 22, 2018 •

edited

Loading

sebalix Jan 22, 2018 •

edited

Loading

sebalix commented Jan 22, 2018 •

edited

Loading

sebalix commented Jan 23, 2018 •

edited

Loading

sebalix Jan 23, 2018 •

edited

Loading

sebalix commented Jan 23, 2018 •

edited

Loading

sebalix commented Jan 23, 2018 •

edited

Loading