Update beaker setup dependencies with linchpin #1346

samvarankashyap · 2019-09-25T17:42:35Z

Install beaker dependencies from beaker repo

adl-bot · 2019-09-25T17:53:58Z

ci-linchpin-messageBus-trigger build #235: SUCCESS: https://jenkins-linchpin.cloud.privileged.psi.redhat.com/job/ci-linchpin-messageBus-trigger/235/

samvarankashyap · 2019-09-25T18:35:20Z

[test]

adl-bot · 2019-09-25T18:46:06Z

ci-linchpin-messageBus-trigger build #236: SUCCESS: https://jenkins-linchpin.cloud.privileged.psi.redhat.com/job/ci-linchpin-messageBus-trigger/236/

StykMartin · 2019-09-26T08:00:24Z

linchpin/provision/roles/dependencies/tasks/main.yml

  include_tasks: "setup_dnf_deps.yml"
  when: setup_providers | length == 0  and ansible_distribution == 'Fedora'

- name: Install os-specific depdendencies:: CentOS/RHEL
+- name: "Install os-specific depdendencies CentOS/RHEL"


Are you sure about the quotes you added?
Another thing you have typo there depdendencies -> dependencies

Adding special characters like "::" causes failures in ansible.
The is the reason I have added quotes to name. Else it is not necessary.
I will fix typo

Sure, I understand that.
But you removed "::" from the name, therefore you don't need quotes.

Remove "::" and you don't need quotes.

Keep "::" and add quotes.

And typo is not fixed yet.

Why even have the double colon?

StykMartin · 2019-09-26T08:00:29Z

linchpin/provision/roles/dependencies/tasks/main.yml

@@ -10,11 +10,11 @@
  loop_control:
    loop_var: indv_prov

- name: Install os-specific depdendencies:: Fedora
+- name: "Install os-specific depdendencies Fedora"


Are you sure about the quotes you added?
Another thing you have typo there depdendencies -> dependencies

StykMartin · 2019-09-26T08:01:57Z

linchpin/provision/roles/dependencies/tasks/beaker_deps.yml

-    - 'bottle'
-  loop_control:
-    loop_var: bkr_pkg
+    name: ['git+https://github.com/beaker-project/beaker.git@develop#subdirectory=Client/', 'python-krbV', 'bottle']


I can see that you are trying to get beaker from Develop branch. Develop branch contains Beaker 27.
I replaced python-krbV with python-gssapi

I could not find a tag specific to beaker client that is the reason I have used to develop branch code.
Further, does the above mean python-krbV is no longer needed?

Beaker client doesn't require python-krbV anymore. python-gssapi is sufficient for beaker client / common.

I think you should consume our tags.. Not blindly install develop

I completely agree with you.
However, could share the link/tag which contains python3 patch
the GitHub repo seems confusing.
Thank you.

Hello @samvarankashyap.
Python3 patch is not tagged (yet). I'm working on it, we have to finish Beaker 27 changes. Then I can provide you tag. I'm about to release 26.6. And we are close to 27 - We are performing tests in stage.

adl-bot · 2019-09-26T15:18:03Z

ci-linchpin-messageBus-trigger build #243: FAILURE: https://jenkins-linchpin.cloud.privileged.psi.redhat.com/job/ci-linchpin-messageBus-trigger/243/

adl-bot · 2019-09-26T15:25:34Z

ci-linchpin-messageBus-trigger build #244: FAILURE: https://jenkins-linchpin.cloud.privileged.psi.redhat.com/job/ci-linchpin-messageBus-trigger/244/

adl-bot · 2019-09-26T15:42:52Z

ci-linchpin-messageBus-trigger build #247: SUCCESS: https://jenkins-linchpin.cloud.privileged.psi.redhat.com/job/ci-linchpin-messageBus-trigger/247/

samvarankashyap · 2019-09-26T15:43:56Z

@14rcole Needs review

Dannyb48 · 2019-09-26T20:03:24Z

Can we push this out to 1.9.0 release? I ran into an issue manually testing the beaker-client on github locally in a virtualenv. The gssapi caused issues with the paramiko version I was using. I just want to spend time from the carbon side make sure this works.

StykMartin · 2019-09-29T10:38:23Z

Are you in hurry with this patch?
When Beaker 27.0 will be released I will also push changes to currently unmaintained PyPI.

I think I can help you guys to make this clean and don't install mess into client machines

samvarankashyap · 2019-09-30T13:49:19Z

@StykMartin :
We are planning to add the patch for the next release which is due October 28th.
quick question is there any tentative date for beaker 27 release?
Thank you

StykMartin · 2019-09-30T13:56:27Z

@samvarankashyap :
Our release is blocked right now by testing performed in a staging environment.
I hope we can manage it within a month. But hard to say.
Right now one team is doing testing and they didn't provide a new status. I will try to reach them if they can provide a date.

Dannyb48 · 2019-10-01T16:27:26Z

I think we should at least test using develop branch or the internal dev rpm to test out the Linchpin Beaker provisioning. To at least make sure the Linchpin python 3 compatible side works consuming the python beaker 3 client apis? That way we can catch things early before beaker 27 is tagged and released?

StykMartin · 2019-10-02T05:09:02Z

For your testing purposes definitely yes! If you want to use internal dev RPM feel free to reach me out. I'm creating new RPM for every new commit merged to develop branch

samvarankashyap · 2019-10-17T14:41:40Z

@StykMartin :
Quick question ,
does beaker 26.5 include python3 support?

StykMartin · 2019-10-17T14:43:37Z

Successfully it does not

samvarankashyap · 2019-10-17T14:45:46Z

@StykMartin Any update on the release date of beaker 27?

Dannyb48 · 2019-10-17T22:05:28Z

so far my testing with the internal dev beaker rpm on RHEL 8 (Red Hat Enterprise Linux release 8.0 (Ootpa)) has been successful using linchpin 1.8.2 and carbon 1.4.1.

The minimal bkr parameters I passed in my scenario
`

      arch: x86_64
      distro: RHEL-7.6
      variant: Server
      whiteboard: {{ BKR_WHITEBOARD }}
      jobgroup: {{ BKR_JOBGROUP }}
      username: {{ BKR_USERNAME }}
      password: {{ BKR_PASSWORD }}
      ssh_key: keys/{{ BKR_KEYPAIR }}
      host_requires_options: 
        - "force = {{ BKR_HOST_FQDN }}"

The only issue I ran into was that the carbon framework had an issue with GSSAPI but that was because of we were using an older version of paramiko, 2.4.2, had to upgrade paramiko 2.6.0 which has full support for GSSAPI.

I will test with the develop branch of linchpin next. Not I have been driving this from carbon not Linchpin CLI.

@StykMartin I think you've mentioned in the beaker-user list that if I have a RHEL 7 w/python3 and I try to install the beaker27 client rpm, this isn't supported? or won't work?

StykMartin · 2019-10-21T07:49:35Z

@Dannyb48 at this moment RHEL 7 client is built for Python 2 only.
We are not providing python 3 support (yet) in RHEL 7.

Dannyb48 · 2019-10-21T15:38:22Z

@StykMartin Sounds good thanks!

Dannyb48 · 2019-10-21T20:42:44Z

@samvarankashyap
My intial testing of both linchpin 1.8.2 and linchpin develop on RHEL8 and Fedora30 with the beaker common/clients 27 build 107 rpm from the internal beakerrepo download site have proven successful to provision and destroy a single beaker resource through carbon using the simple pinfile generated by carbon below.

The only real gotcha was from carbon side, where we had to update paramiko 2.4.2 to paramiko 2.6.0 to support the new beaker GSSAPI requirement.

I did have issues trying to install beaker from the github Client sub-directory issues with finding python dependencies.

Although I do question, based on Martin's answer, if we need to add another task to this playbook? One that installs the bkr client 27 install on systems that support it, i.e. RHEL8/Fedora30 and leave this existing task for systems <RHEL8/Fedora30?

carbon:
  layout:
    inventory_layout:
      hosts:
        node:
          count: 1
          host_groups:
          - hosts
          - client
      vars:
        ansible_ssh_private_key_file: keys/demo
        ansible_user: root
        hostname: db2-carbon-beaker-node
  topology:
    resource_groups:
    - resource_definitions:
      - job_group: ci-ops-central
        recipesets:
        - arch: x86_64
          count: 1
          distro: RHEL-7.6
          hostrequires:
          - force: dell-per220-02.cci.lab.eng.bos.redhat.com
          name: db2-carbon-beaker-node
          ssh_key_file:
          - demo.pub
          variant: Server
        role: bkr_server
        ssh_keys_path: /home/fedora/cbn/e2e-acceptance-tests/keys
        whiteboard: py3-beaker-testing
      resource_group_name: carbon
      resource_group_type: beaker
    topology_name: carbon

Dannyb48

Based on Martin's comments, it looks like right now the initial support for Beaker 27 client python 3 is Fedora 30 systems and RHEL 8. Support for RHEL 7 python 3 is coming later on.

Should we keep the original tasks for systems <RHEL8/Fedora 30 to install beaker client < 27 and add a new task for RHEL8/Fedora30 to install the beaker client 27? That way the correct Beaker version is installed appropriately until Beaker 27 for python 3 is supported on RHEL 7 systems?

samvarankashyap · 2019-10-22T13:35:50Z

@Dannyb48
I agree with you.
We can maintain multiple tasks based on operating systems.
Currently, we do test on fedora30, centos7 and fedora29.
I will update the task accordingly

samvarankashyap · 2019-10-22T13:38:25Z

@StykMartin
It would be great if we can get a tentative release date for the beaker 27 release.

StykMartin · 2019-10-23T14:39:14Z

I think in 3 weeks we have maintenance window to push updates.
So I'm expecting around this time we will push 27.

Dannyb48 · 2019-10-29T17:40:07Z

FYI on RHEL 8 - when using the install from github method, we need to make sure that the python3-gssapi rpm bindings were installed before the python gssapi package would install.

Did not have this problem when the beaker-client was installed using the rpm method.

samvarankashyap · 2019-10-29T17:43:42Z

@Dannyb48
I am planning to add rhel7 rhel8 support in upcoming release of linchpin
ref: #1197 #1445
I will keep that in mind before I do that.
Thanks.

StykMartin · 2019-11-25T12:20:27Z

Working on release notes. This week we will tag RC

samvarankashyap · 2019-12-04T16:43:30Z

@StykMartin
I see there is a beaker-client rpm on fedora 30 repos.
Are there any plans pushing beaker-client to PyPI anytime soon?

StykMartin · 2019-12-04T17:29:36Z

@samvarankashyap beaker-client 27.0

samvarankashyap · 2019-12-04T17:30:46Z

@StykMartin Awesome news.
Thank you very much for the update 👍

StykMartin · 2019-12-04T19:13:04Z

🎉 🎉 🎉
Sorry for prolonging this release. We had many problems to solve.

StykMartin · 2019-12-04T19:14:08Z

linchpin/provision/roles/dependencies/tasks/beaker_deps.yml

-    - 'bottle'
-  loop_control:
-    loop_var: bkr_pkg
+    name: ['git+https://github.com/beaker-project/beaker.git@develop#subdirectory=Client/', 'python-gssapi', 'bottle']


This can be replaced with the latest 27.0 Beaker release.

StykMartin · 2019-12-09T15:26:33Z

setup.py

@@ -46,7 +46,7 @@
    tests_require=["pytest<=4.4.0", "nose", "mock", "coverage", "flake8"],
    extras_require={
        'krbV': ["python-krbV"],
-        'beaker': ['beaker-client>=23.3', 'python-krbV'],
+        'beaker': ['beaker-client>=27.0', 'python-gssapi', 'bottle'],


Why do you need bottle?
It is definitely not part of Beaker requirements.

samvarankashyap · 2019-12-09T16:52:11Z

[merge]

samvarankashyap requested a review from 14rcole September 25, 2019 17:42

samvarankashyap added this to the v1.8.2 milestone Sep 25, 2019

samvarankashyap self-assigned this Sep 25, 2019

samvarankashyap added 1 ansible galaxy code improvement dependencies dependency check developer tools labels Sep 25, 2019

StykMartin reviewed Sep 26, 2019

View reviewed changes

samvarankashyap force-pushed the update_beaker_dep branch 2 times, most recently from e64d4dc to 2ec1075 Compare September 26, 2019 15:16

samvarankashyap requested a review from Dannyb48 September 26, 2019 17:40

samvarankashyap modified the milestones: v1.8.2, v1.9.0 Sep 26, 2019

14rcole approved these changes Oct 14, 2019

View reviewed changes

Dannyb48 reviewed Oct 22, 2019

View reviewed changes

samvarankashyap modified the milestones: v1.9.0, v1.9.1 Oct 28, 2019

samvarankashyap removed this from the v1.9.1 milestone Dec 2, 2019

StykMartin suggested changes Dec 4, 2019

View reviewed changes

samvarankashyap added 2 commits December 9, 2019 10:23

Update beaker setup dependencies with linchpin

b4bbca5

update beaker dependencies for linchpin

f501ccd

samvarankashyap force-pushed the update_beaker_dep branch from 2ec1075 to f501ccd Compare December 9, 2019 15:23

StykMartin reviewed Dec 9, 2019

View reviewed changes

adl-bot merged commit f330110 into CentOS-PaaS-SIG:develop Dec 9, 2019

Update beaker setup dependencies with linchpin #1346

Update beaker setup dependencies with linchpin #1346

Conversation

samvarankashyap commented Sep 25, 2019

adl-bot commented Sep 25, 2019

samvarankashyap commented Sep 25, 2019

adl-bot commented Sep 25, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adl-bot commented Sep 26, 2019

adl-bot commented Sep 26, 2019

adl-bot commented Sep 26, 2019

samvarankashyap commented Sep 26, 2019

Dannyb48 commented Sep 26, 2019

StykMartin commented Sep 29, 2019

samvarankashyap commented Sep 30, 2019

StykMartin commented Sep 30, 2019

Dannyb48 commented Oct 1, 2019

StykMartin commented Oct 2, 2019

samvarankashyap commented Oct 17, 2019

StykMartin commented Oct 17, 2019

samvarankashyap commented Oct 17, 2019

Dannyb48 commented Oct 17, 2019

StykMartin commented Oct 21, 2019

Dannyb48 commented Oct 21, 2019

Dannyb48 commented Oct 21, 2019

Dannyb48 left a comment

Choose a reason for hiding this comment

samvarankashyap commented Oct 22, 2019 • edited Loading

samvarankashyap commented Oct 22, 2019

StykMartin commented Oct 23, 2019

Dannyb48 commented Oct 29, 2019

samvarankashyap commented Oct 29, 2019

StykMartin commented Nov 25, 2019

samvarankashyap commented Dec 4, 2019

StykMartin commented Dec 4, 2019 • edited Loading

samvarankashyap commented Dec 4, 2019

StykMartin commented Dec 4, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

samvarankashyap commented Dec 9, 2019

samvarankashyap commented Oct 22, 2019 •

edited

Loading

StykMartin commented Dec 4, 2019 •

edited

Loading