-
-
Notifications
You must be signed in to change notification settings - Fork 69
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Docker /etc/cont.init.d/ scripts for installation of Jython and the Helper Libraries for openHAB Scripted Automation #215
base: master
Are you sure you want to change the base?
Conversation
You should use relative links Alternatively, putting the libs inside the container would cause them to be updated when you update the container. You would then be making links to a location only available in the container so wouldn't need relative. |
I haven't been here for a while, as i'm too busy setting up a solid Kubernetes cluster. However today i've been writing my own script exactly for this purpose. I've got a few remarks:
|
Another comment regarding the other script:
|
I don't have the time to review ATM, but this...
... is a really good point that I had not thought of. Thank you for bringing this up, @Rick-Jongbloed! I've opened #218 for this. 000_StartupDelay.py should not be an issue, as it is a core file that nobody should be modifying. The configuration.py is distributed as configuration.py.example to avoid overwrites on upgrade. |
OK, i'm testing the jython script out and will post my findings in this post (I don't know how to do these fancy code reviews yet haha)
|
A suggestion from my work on an install script: Also look at #213 for a much simpler way to install Jython (and Groovy) |
This will only work if the Jython jar file is the only jar file that needs to be loaded into OpenHAB this way (Which it usually is, but still, we should leave the option open that other jar files could be at that location. I like the way you used the registry file though :-) |
Credit should go to Scott for that, I just implemented it and and wrote the docs. You can have as many jars in that folder as you like, I see no issue |
Like Scott mentioned, the The The following files/directories are all kept outside of the helper libraries directory:
So you can safely delete For the community libraries that you want to make use of you basically have two options:
Thanks for all the feedback so far! I'll try to incorporate it all tomorrow. To make inline comments go to the top of this page and just below the PR title you see the 'Files changed' "tab". If you click on it then you'll get to see the files. When you hover your mouse over the code a blue + icon will appear. Press it and you can type in your comment. |
I'm currently deleting the installer file after Jython has been installed. I don't think it is worth keeping it. Downloading it takes only a few seconds. I do intend to add Rick's suggestion to specify an alternate URI. If that is used I want to add a check to see if it is a download (URI starts with http) or a local file mounted into the container (in the latter case the script must not remove the file after installing because then it would remove the users original file).
I already saw it but wasn't sure if that was the new way of doing the installation or just a suggestion Now that it is clear that that will be the new way I'll try to incorporate that as well. I appreciate all the feedback! |
Why not download the jar directly? |
Because I use the full installer which comes with pip and other tools to be able to install additional Python packages. |
Ah hah. Interesting. I haven't actually looked at the full install yet. |
How is it clear now? 😉 I haven't responded to the PR, but I have serious reservations about changing the install in this way. I'll get to it today. |
Well I figured it it's your idea and CrazyIvan359 already did the documentation then that must be the way forward :) I'll hold off on that part for now. |
IIRC, I had thrown it out to Michael as a possible simplification for use in an install script, but it wasn't thought out (but works!). The OH update script removes the runtime directory, so you'd need to reinstall Jython after every time. Best to stay with EXTRA_JAVA_OPTS for now. I'm still getting something wrapped up and might have time to take a look at your script later this weekend, if things go well. Until then, is there any possibility to turn your script into something that could be used both with and without Docker? |
I wasn't aware of this before, but in the context of Docker that's the normal lifecycle of a container. For regular installs it's a problem though. |
I'm concerned about the enable_next_generation_rule_engine function. It seems to assume that the user is only using add-ons.cfg to install bindings. I'm not super good at reading sed expressions but I'll assume it appends to the list instead of replacing the misc entry, which is ok. But if a user is only installing add-ons through the REST API, all of their previously installed misc add-ons will be uninstalled. That's not good. The script should support both get types of users and not require the use of addons.cfg. it won't be as clean and simple but you can do rest calls using curl to figure out if adding the misc line to add-ons.cfg will be destructive and add it using a rest call. |
I tested this just now and see what you mean. That is not good no. I find this very strange behavior, but I guess it is intentional and there is a reason why this works in the way it does. Unfortunately, |
ok, addons are not stored in the |
It has always worked this way and it stems from the fact that the text config files always take precedence over what has been done through PaperUI. As soon as you create an entry in addons.cfg, you need to specify all of your addons for that category in addons.cfg. OH will look at the file and make itself match the file (i.e. installing any missing and uninstalling any not listed).
I'm not sure if I'm understanding correctly, but you mean if you have the rules engine listed in addons.cfg and then uninstall it through PaperUI that it will get reinstalled on a restart of OH? If yes then indeed that is what will happen as openHAB tries to make itself match what is in addons.cfg. I personally do use addons.cfg (I never got around to switching, probably will at some point). Have you confirmed that addons.config get's populated even when you are not using addons.cfg? |
Your understanding is correct. I've come to the conclusion that enabling the Rule Engine via script is a no go. There are simply to many possibilities. It will be much clearer for a user to always manually enable the Rule Engine than it will be for him/her to understand under which circumstances the Rule Engine is automatically enabled or needs to be manually enabled.
Yes. If you enable the Rule Engine addon via PaperUI it is added to the |
* Install Jython using the jython-installer.jar if not installed already. * Enable the Next Generation Rule Engine if not enabled already. * Remove *$py.class files upon container startup. * Export EXTRA_JAVA_OPTS environment variable updated for Jython.
* Download an install the helper libraries. * Copy the initial configuration.[groovy|js|py] in place if it does not exist. This will script will download the helper libraries to the /openhab/automation/openhab-helper-libraries directory when this directory does not exist. Before making any changes, the script does a sanity check to make sure that the required changes can be made. In case of a problem the script will abort before making any changes to the system. Updating the helper libraries is as easy as deleting the /openhab/automation/openhab-helper-libraries directory and restarting the container. The script creates the following directory structure (where 'core' is a symlink the the indicated core directory in the openhab-helper-library): /openhab/conf/automation/jsr223/groovy/community /openhab/conf/automation/jsr223/groovy/core -> /openhab/conf/automation/openhab-helper-libraries/Core/automation/jsr223/groovy/core /openhab/conf/automation/jsr223/groovy/personal /openhab/conf/automation/jsr223/javascript/community /openhab/conf/automation/jsr223/javascript/core -> /openhab/conf/automation/openhab-helper-libraries/Core/automation/jsr223/javascript/core /openhab/conf/automation/jsr223/javascript/personal /openhab/conf/automation/jsr223/python/community /openhab/conf/automation/jsr223/python/core -> /openhab/conf/automation/openhab-helper-libraries/Core/automation/jsr223/python/core /openhab/conf/automation/jsr223/python/personal /openhab/conf/automation/lib/groovy/community /openhab/conf/automation/lib/groovy/configuration.groovy /openhab/conf/automation/lib/groovy/core -> /openhab/conf/automation/openhab-helper-libraries/Core/automation/lib/groovy/core /openhab/conf/automation/lib/groovy/personal /openhab/conf/automation/lib/javascript/community /openhab/conf/automation/lib/javascript/configuration.js /openhab/conf/automation/lib/javascript/core -> /openhab/conf/automation/openhab-helper-libraries/Core/automation/lib/javascript/core /openhab/conf/automation/lib/javascript/personal /openhab/conf/automation/lib/python/community /openhab/conf/automation/lib/python/configuration.py /openhab/conf/automation/lib/python/core -> /openhab/conf/automation/openhab-helper-libraries/Core/automation/lib/python/core /openhab/conf/automation/lib/python/personal /openhab/conf/automation/openhab-helper-libraries
… (can also be a file location inside the container). * Added option to install additional python modules using the PYPI_MODULES variable. * No longer automatically enable the Next Generation Rule Engine because this may uninstall other addons installed via the REST API (e.g. PaperUI). * Notify user when Next Generation Rule Engine is not enabled. * Added documentation for the configuration variables.
… container. * Create __init__.py files in the lib/python/community and lib/python/personal directories in case they do not exist. * No longer automatically enable the Next Generation Rule Engine because this may uninstall other addons installed via the REST API (e.g. PaperUI). * Notify user when Next Generation Rule Engine is not enabled.
Based on the feedback received I have updated both scripts. The only thing that I did not change is the check whether or not Jython is installed based on the existence of the $JYTHON_HOME directory. The complexity that this would add is in my opinion not worth it. The current test is unambiguous; when the directory exists then jython does not need to be installed, when it does not exist jython needs te be installed. |
Forgot to mention in the commit. When installing the jython libraries on either an Alpine based image or ARM architecture then pip is not installed (see jythontools/jython#108 as mentioned by Rick). |
@marcelerkel nice one, i've implemented your scripts and these are my findings for the JSR232 script:
I think ensure pips is executed in the standard installation. See https://wiki.python.org/jython/JythonReleaseNotes
|
These are my findings for the upgrade script:
Thanks for updating the scripts! |
@Rick-Jongbloed , the original submission did add to addons.cfg. The problem with that is you can't just blindly add any entry to that file because if the user doesn't use addons.cfg and installs everything through PaperUI, OH will uninstall everything else in the MISC category that the user has installed through PaperUI because addons.cfg always takes precedence. If you use addons.cfg at all, you have to use it for everything. Marc determined that trying to work around this is far too complex and removed that from the script. |
It seems to be convention to not use a filename extension for
The
My reasoning was that this way the script wouldn't need to keep track of how the installer jar ended up on the users system, what its filename was (always
Thanks, fixed.
I'm not sure what you're trying to say here. All the Anyway, I'm not too happy with this myself either. The script should enable the next gen rule engine if possible. There are a number of problems though:
I've redesigned the script so that it will try to enable the next gen rule engine automatically using the following algorithm:
Yes, to exclude pip from being installed on architectures/images on which it fails to install, e.g. 2.7.1 on Raspberry Pi or the Alpine based image on AMD64.
The script is designed so that it will only make changes to the users system when all prerequisites are met and that includes downloading and successfully unzipping the zip file. If any of this fails then the script will exit and no changes to the users system are made. So in case of failure the users system isn't in some halve installed state. It's either installed/updated or not changed.
See my previous answer :)
Not in the way that I use it.
I'll fix this.
I removed the I need to do some more testing and will update the PR (probably early next week). |
And you can't use the .sh extension for scripts you put into /etc/cron.daily (for example). Took me a long time figure out why my cron scripts weren't running because of that one.
Wouldn't you want to check the entries in the cfg file first? Consider this scenario. I've installed a bunch of stuff including something from the misc category though addons.cfg. Later I come along and decide to use this script. Because I've run before, addons.conf will have an entry for misc in addons.conf. Thus only 1 will run and ruleengine will only be added to addons.conf. Then OH starts again and addons.cfg will get loaded and wipe out the change to addons.conf because ruleengine isn't there. At a minimum I would expect you to need to do just if instead of else if for 2 if you can't change the order of the checking. |
I was under the impression that if addons were configured in So I will swap the order of 1 and 2, and also the order of 3 and 4 so that
|
I'm going to remove the enabling and verification of the NGRE. I simply do not fully understand how this works and as such I find it too dangerous to automatically enable the NRGE by simply modifying either While testing I ran into a failure situation:
(left bindings/misc is addons.config, right bindings/misc is addons.cfg) In case of the failed test case, the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@marcelerkel, do you think this will still be necessary after this PR is merged? If so, I'm ready to review.
Here are some minor things I'd spotted.
#!/bin/bash | ||
|
||
## Jython Version | ||
# The Jython verion to install when using the default download location. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
verion
version
# For example: | ||
# PYPI_MODULES="requests python-dateutil" | ||
# | ||
# Leave empty when no additional python module need to be installed. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
python module
Python modules
|
||
|
||
## Jython Installer Options | ||
# Specifiy the command line options that will be used when running the jython-installer.jar. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Specifiy
Specify
# Or file location inside the container: | ||
# JYTHON_INSTALLER_URI=/openhab/jython-installer/jython-installer-2.7.0.jar | ||
# | ||
# Leave empty to use the default location. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What is the default location? Maybe move the definition to here?
# For example: | ||
# JYTHON_INSTALLER_OPTIONS="-t standard -e demo doc src ensurepip" | ||
# | ||
# Leave empty to use the default options. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What are the default options? Maybe move the definition to here?
JYTHON_DEFAULT_INSTALLER_OPTIONS="-t standard -e demo doc src" | ||
|
||
JYTHON_HOME="${OPENHAB_CONF}/automation/jython" | ||
export EXTRA_JAVA_OPTS="${EXTRA_JAVA_OPTS} -Xbootclasspath/a:${JYTHON_HOME}/jython.jar -Dpython.home=${JYTHON_HOME} -Dpython.path=${JYTHON_HOME}/Lib:${OPENHAB_CONF}/automation/lib/python" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
${JYTHON_HOME}/Lib
I don't believe this is necessary, since this directory is added to sys.path... https://jython.readthedocs.io/en/latest/chapter7/#module-search-path-and-loading
@openhab-5iver I need to check if this is still necessary, but most likely will not get to it before xmas. Next to my full time job I'm studying to obtain my Software Engineering BSc degree in the evenings/weekends, hence my absence here and on the forums lately (I kind of forgot about that I still needed to update this PR). I'm in my final year and hope to get my diploma at the end of June in 2020 so that I can get my life back :) |
No worries from my side! I have a pile of PRs to review and this was first on the list. I think this PR will be obsolete, except for people who want a full install of Jython. I have an idea that might help with that... a configuration option in the addon to manually set python.home. Get back to studying and quit playing with OH! |
Is there any progress on this? |
@wertzui, this script will be pretty much obsolete when the Jython bundle gets merged... |
Two scripts cont.init.d scripts for the official openHAB Docker image.
The
10-jsr-jython
script performs the following actions:jython-installer.jar
if not installed already.*$py.class
files upon container startup.EXTRA_JAVA_OPTS
environment variable updated for Jython.You no longer need to build your own docker container based on the official openHAB image.
Minor issue: The
jython-installer.jar
seems to have a problem when using the Alpine image when it installs PIP.When using the Debian image all works ok. I need to further investigate what is going wrong and may simply work around it by using the
jython-standalone.jar
when the script is run on the Alpine image.The
10-openhab-helper-libraries
script performs the following actions:configuration.[groovy|js|py]
in place if it does not exist.This will script will download the helper libraries to the
/openhab/automation/openhab-helper-libraries
directory when this directory does not exist.Before making any changes, the script does a sanity check to make sure that the required changes can be made. In case of a problem the script will abort before making any changes to the system.
Updating the helper libraries is as easy as deleting the
/openhab/automation/openhab-helper-libraries
directory and restarting the container.The script creates the following directory structure (where 'core' is a symlink the the indicated core directory in the openhab-helper-library):
/openhab/conf/automation/jsr223/groovy/community
/openhab/conf/automation/jsr223/groovy/core -> /openhab/conf/automation/openhab-helper-libraries/Core/automation/jsr223/groovy/core
/openhab/conf/automation/jsr223/groovy/personal
/openhab/conf/automation/jsr223/javascript/community
/openhab/conf/automation/jsr223/javascript/core -> /openhab/conf/automation/openhab-helper-libraries/Core/automation/jsr223/javascript/core
/openhab/conf/automation/jsr223/javascript/personal
/openhab/conf/automation/jsr223/python/community
/openhab/conf/automation/jsr223/python/core -> /openhab/conf/automation/openhab-helper-libraries/Core/automation/jsr223/python/core
/openhab/conf/automation/jsr223/python/personal
/openhab/conf/automation/lib/groovy/community
/openhab/conf/automation/lib/groovy/configuration.groovy
/openhab/conf/automation/lib/groovy/core -> /openhab/conf/automation/openhab-helper-libraries/Core/automation/lib/groovy/core
/openhab/conf/automation/lib/groovy/personal`/openhab/conf/automation/lib/javascript/community
/openhab/conf/automation/lib/javascript/configuration.js
/openhab/conf/automation/lib/javascript/core -> /openhab/conf/automation/openhab-helper-libraries/Core/automation/lib/javascript/core
/openhab/conf/automation/lib/javascript/personal
/openhab/conf/automation/lib/python/community
/openhab/conf/automation/lib/python/configuration.py
/openhab/conf/automation/lib/python/core -> /openhab/conf/automation/openhab-helper-libraries/Core/automation/lib/python/core
/openhab/conf/automation/lib/python/personal
/openhab/conf/automation/openhab-helper-libraries
To use the community scripts you either create a symlink or copy the files into the
/openhab/conf/automation/[jsr223|lib]/python/community
directory. Both thecommunity
andpersonal
directories have group write permissions so that if you add your own user to theopenhab
group on the Docker host you have write permissions in these directories.When using a different directory structure on the Docker host compared to the one inside the container then the symlink to the
core
directory in the openhab-helper-libraries will be broken. However, I think that is a small price to pay for the ease of updating the libraries. Just delete theopenhab-helper-libraries
directory, restart the container and the latestopenhab-helper-libraries-master.zip
file will automatically be downloaded and installed.I have created two scripts so that those who don't want to use Python don't need to install the Jython libraries and those you want to use Python and not the openHAB Helper Libraries can do so as well. Each script can be used without using the other. There are not dependencies.
Build and tested on 2.5.0-SNAPSHOT Build #1654
When excepted as the default method for installing the helper libraries on Docker then I will of course update the documentation.