Use OpenSSL 1.0.2 for Connext on MacOS #436

mikaelarguedas · 2020-04-18T16:07:05Z

Without this, Connext cannot use the security plugins: ros2/build_farmer#269

Mini2:
Lore:

Mini3:
Not sure what's going on on mini3, It shows the exact same error as the other ones without this patch applied (e.g. nightly on lore https://ci.ros2.org/view/nightly/job/nightly_osx_release/1620/).
it doesn't have the same OpenSSL version as the other ones (1.0.2s vs 1.0.2t), maybe that's part of the reason ?

Tested with #421

nuclearsandwich · 2020-04-23T17:09:17Z

@mikaelarguedas am I correct that this modified version of my branch is using OpenSSL 1.0.2 from Homebrew rather than the one distributed by Connext? I have two concerns if that's the case. First I'm not sure when homebrew will drop that OpenSSL version and second it's not guaranteed that Connext will continue to work with that version rather than the one they distribute.

mikaelarguedas · 2020-04-23T21:09:16Z

am I correct that this modified version of my branch is using OpenSSL 1.0.2 from Homebrew rather than the one distributed by Connext?

Yes

First I'm not sure when homebrew will drop that OpenSSL version

That's a valid concern

second it's not guaranteed that Connext will continue to work with that version rather than the one they distribute.

I'm not sure I got this concern. the deb of connext is not changing so it will keep supporting it. If Connext version finally gets updated it means we will be able to use 1.1.1 and drop the use of the EOL one.

I'm fine using the version provided by RTI as well. I chose to use the homebrew one as this is the only thing that's available on all workers and easily testable for us not managing these machines. I'm 👍 for any approach that gets us to the point where we can test security changes for Foxy

sloretz · 2020-04-30T23:43:18Z

am I correct that this modified version of my branch is using OpenSSL 1.0.2 from Homebrew rather than the one distributed by Connext

First I'm not sure when homebrew will drop that OpenSSL version

That's a valid concern

Looks like Homebrew already dropped [email protected]: Homebrew/homebrew-core#46876

sloretz · 2020-05-01T18:32:06Z

Testing this with 1 minor tweak to the path and with osrf/homebrew-ros2#8 which has been installed onto mini1. Just test_security since those are the tests that have been failing.

-> ros2/ros2_documentation#641

sloretz · 2020-05-01T19:32:54Z

Test results look ok using osrf/homebrew-ros2#8. @clalancette was asking why roll a homebrew formula instead of use the RTI provided openssl. It seems like there should be some file called openssl-1.0.2n-host-x64Darwin17clang9.rtipkg, but I've completely failed to find it.

@mikaelarguedas @nuclearsandwich @dirk-thomas do you know how to get access to the connext build of OpenSSL 1.0.2? If not, I'm inclined to move forward with osrf/homebrew-ros2#8.

mikaelarguedas · 2020-05-01T20:00:39Z

clalancette was asking why roll a homebrew formula instead of use the RTI provided openssl. It seems like there should be some file called openssl-1.0.2n-host-x64Darwin17clang9.rtipkg, but I've completely failed to find it.

Yes I believe that's what @nuclearsandwich tried with the first commit of this PR (ros2/system_tests#409 (comment)). But there were still the security failures. As I didnt know which machines had the connext openssl installed, I used the homebrew one because I could see from the job logs that is was installed on all nodes.

I do not know where @nuclearsandwich found the rtipkgs for MacOS

sloretz · 2020-05-01T20:30:41Z

I do not know where @nuclearsandwich found the rtipkgs for MacOS

Since ros2/system_tests#409 has been merged I'll install openssl using osrf/homebrew-ros2#8 and set RTI_OPENSSL_BIN and RTI_OPENSSL_LIBS in ~/.bashrc. That way the tests will be useful over the weekend until we can get and use the RTI provided openssl.

sloretz · 2020-05-05T00:14:06Z

Since ros2/system_tests#409 has been merged I'll install openssl using osrf/homebrew-ros2#8 and set RTI_OPENSSL_BIN and RTI_OPENSSL_LIBS in ~/.bashrc. That way the tests will be useful over the weekend until we can get and use the RTI provided openssl.

Seems the variables in ~/.bashrc didn't have an effect, but I know this PR works. Merged this branch with master and running a full OSX job

ruffsl · 2020-05-05T09:16:42Z

It seems like there should be some file called openssl-1.0.2n-host-x64Darwin17clang9.rtipkg, but I've completely failed to find it.

I'm guessing that OSRF keeps a local archive of such files, but if you've lost them then you could use the support account that RTI presumably gave to OSRF to download the openssl rtipkgs from: https://support.rti.com/s/downloads?Release=5.3.1

Two days ago they updated the support portal to rename accounts via [email protected], so you may also want to check the respective email address with the account for reset credentials.

sloretz · 2020-05-06T20:30:56Z

@ruffsl @mikaelarguedas what generates the permissions.p7s file? Mini1 is set up to use the RTI provided openssl version, and RTI_OPENSSL_BIN/RTI_OPENSSL_LIBS appear to be set up correctly, but I'm still seeing the security tests fail. It looks like complaint is the current time is earlier than the not_before time in the permissions.p7s file

7: RTI Security Plugins Evaluation License issued to OSRF (OSRF01) ... For non-production use only.
7: Expires on 22-oct-2020 See www.rti.com for more information.
7: [CREATE Participant]RTI_Security_PermissionsGrant_isValidTime:now is before not_before of permissions file
7: [CREATE Participant]RTI_Security_AccessControl_validate_permissions:grant has invalid time
7: [CREATE Participant]RTI_Security_AccessControl_validate_local_permissions:failed to validate local permissions. XML file:
7: [CREATE Participant]RTI_Security_AccessControl_validate_local_permissions:/Users/osrf/jenkins-agent/workspace/test_ci_osx/ws/build/test_security/test/test_security_files/enclaves/subscriber/permissions.p7s
7: [CREATE Participant]DDS_DomainParticipantTrustPlugins_getLocalParticipantSecurityState:!security function validate_local_permissions returned NULL
7: [CREATE Participant]DDS_DomainParticipant_createI:!get local participant security state
7: [CREATE Participant]DDS_DomainParticipantFactory_create_participant_disabledI:!create participant
7: DDSDomainParticipant_impl::create_disabledI:!create participant
7: DDSDomainParticipant_impl::createI:!create participant
7: DomainParticipantFactory_impl::create_participant():!create failure creating participant

permissions.p7s validity section

      <validity>^M
        <not_before>2020-05-06T19:33:29</not_before>^M
        <not_after>2030-05-04T19:33:29</not_after>^M
      </validity>^M

Current date on mini1

$ date
Wed May  6 13:30:01 PDT 2020

Is the permissions.p7s file being generated in the right timezone? The timestamp in permissions.p7s looks like UTC.

mikaelarguedas · 2020-05-06T20:47:17Z

oh this is a new one then.
Dates are generated in UTC time because Connext was complaining about it in the certificates before ros2/sros2#148.

These files are generated by sros2.
This issue must have appeared in ros2/sros2#205.
Can you can try with ros2/sros2#209 to see if it works around your issue ?

As the tests were failing before that was merged, I would imagine you'll uncover another failure mode after this one.

ruffsl · 2020-05-06T20:51:00Z

Is there some logic in checking for daylights savings time or leap seconds that is tripping up MacOS?

https://www.timeanddate.com/worldclock/converter.html?iso=20200506T193329&p1=tz_pt&p2=1440

~~It seems that mini1 is 3 minutes behind whatever python3 returns here:~~

https://github.com/ros2/sros2/blob/d4b32b4827b1a45529b8183c82b0841c8132e9a1/sros2/sros2/api/_utilities.py#L83

The mini1's local clock should read after 12:33pm PDT when the cert is created.

sloretz · 2020-05-06T22:02:03Z

Is there some logic in checking for daylights savings time or leap seconds that is tripping up MacOS?

The mini1's local clock should read after 12:33pm PDT when the cert is created.

It probably did read 12:33pm. I would interpret the difference between 13:30 and 12:33 as from the time I ran the build it took 57 minutes until I was ready to write a comment on github.

Can you can try with ros2/sros2#209 to see if it works around your issue ?

It works 🎉 All test_security tests passed using that PR

mikaelarguedas · 2020-05-13T21:21:24Z

It works 🎉 All test_security tests passed using that PR

@sloretz is this ready for merge then ? or are there some failing tests that arose since?

sloretz · 2020-05-13T21:27:16Z

@sloretz is this ready for merge then ? or are there some failing tests that arose since?

I think we could close this one. The OSX machines are setup with the RTI_OPENSSL_BIN and RTI_OPENSSL_LIBS variables in the ~/.bashrc of the user running the jenkins agent. They're all configured to use the RTI provided openssl, which is a tar archive extracted to /User/<user>/openssl..... It may not make sense to use the old homebrew path as the default since that's no longer available.

mikaelarguedas · 2020-05-13T21:41:05Z

The OSX machines are setup with the RTI_OPENSSL_BIN and RTI_OPENSSL_LIBS variables in the ~/.bashrc of the user running the jenkins agent.

That's great news! Do you know when this change is expected to take effect?

Apparently the nightlies test_security tests still fail the same way and don't seem to have any /User/<user>/openssl on their PATH/DYLD_LIBRARY_PATH (suggesting the RTI_OPENSSL_* vars are not set)

sloretz · 2020-05-13T21:50:46Z

Apparently the nightlies test_security tests still fail the same way and don't seem to have any /User//openssl on their PATH/DYLD_LIBRARY_PATH (suggesting the RTI_OPENSSL_* vars are not set)

Which nightly has the failing tests? AFAIK all machines are set up this way already.

mikaelarguedas · 2020-05-13T21:56:15Z

All the ones using connext afaict:
https://ci.ros2.org/view/nightly/job/nightly_osx_debug/1623/#showFailuresLink
https://ci.ros2.org/view/nightly/job/nightly_osx_release/1646/#showFailuresLink
https://ci.ros2.org/view/nightly/job/nightly_osx_repeated/1952/#showFailuresLink

sloretz · 2020-05-13T22:21:36Z

https://ci.ros2.org/view/nightly/job/nightly_osx_debug/1623/#showFailuresLink

Strange, this one happened on mini3, and it looks like it couldn't open libssl...dylib

7: [CREATE Participant]RTIOsapiLibrary_open:error opening library libnddssecurityd.dylib: dlopen(libnddssecurityd.dylib, 2): Library not loaded: libssl.1.0.0.dylib
7:   Referenced from: /Applications/rti_connext_dds-5.3.1/lib/x64Darwin16clang8.0/libnddssecurityd.dylib
7:   Reason: image not found
7: [CREATE Participant]DDS_DomainParticipantTrustPlugins_initialize:!failed to load library
7: [CREATE Participant]DDS_DomainParticipant_createI:!create builtin trust plugins support
7: [CREATE Participant]DDS_DomainParticipantFactory_create_participant_disabledI:!create participant
7: DDSDomainParticipant_impl::create_disabledI:!create participant
7: DDSDomainParticipant_impl::createI:!create participant
7: DomainParticipantFactory_impl::create_participant():!create failure creating participant

but the RTI_OPENSSL_* environment variables look ok when I ssh in. It looks like the agent process is a child ofPID 1, so maybe it's not being started in a bash terminal, and no environment variables from ~/.bashrc are set? This is strange because I would have sworn it worked on mini1 using the test_ci_osx job.

mikaelarguedas · 2020-05-14T06:46:28Z

A while back .bashrc and bash_profile didn't take effect in Jenkins, this is why defining variables in here was needed. Maybe this is still the case.

mikaelarguedas · 2020-05-20T07:30:28Z

@sloretz friendly ping: is there anything we can do to move this forward ? Maybe by installing the openssl 1.0 bottle you built? or by modifying this PR to point the variables to the installation location of RTI's OpenSSL ?

sloretz · 2020-05-29T03:24:26Z

@sloretz friendly ping: is there anything we can do to move this forward ?

@jacobperron will pick it up from here

mikaelarguedas · 2020-06-15T20:53:00Z

@jacobperron any update on this ? This seems to still be the source of failing tests. If provided the path to the connext security plugins install location on the macOS machines, I can update this PR to point to it and it should hopefully fix all the failing test_security tests

cc @kyrofa

jacobperron · 2020-06-16T16:56:21Z

@mikaelarguedas Thanks for the bump. It dropped from my radar. I'll take a look today.

jacobperron · 2020-06-16T21:56:16Z

@mikaelarguedas I believe the location of the missing dylib (libssl.1.0.0.dylib) is the same on all of our CI machines: "/Users/osrf/openssl-1.0.2n/x64Darwin17clang9.0/release/lib"

RTI_OPENSSL_LIBS is pointing to the correct place when I login and run the tests manually, but I still see the same test failures. If I manually set DYLD_LIBRARY_PATH in a shell, then the tests pass, (e.g. DYLD_LIBRARY_PATH=${DYLD_LIBRARY_PATH}:${RTI_OPENSSL_LIBS}).

mikaelarguedas · 2020-06-17T19:16:35Z

I believe the location of the missing dylib (libssl.1.0.0.dylib) is the same on all of our CI machines: "/Users/osrf/openssl-1.0.2n/x64Darwin17clang9.0/release/lib"

I gave that a try with f095669.
Looks to be working on all machines but lore:

agent	Release	Debug
lore	https://ci.ros2.org/job/ci_osx/9126/testReport/	https://ci.ros2.org/job/ci_osx/9128/testReport/
mini1	https://ci.ros2.org/job/ci_osx/9125/testReport/	https://ci.ros2.org/job/ci_osx/9145/testReport/
mini2	https://ci.ros2.org/job/ci_osx/9127/testReport/	https://ci.ros2.org/job/ci_osx/9137/testReport/
mini3	https://ci.ros2.org/job/ci_osx/9124/testReport/	https://ci.ros2.org/job/ci_osx/9138/testReport/

jacobperron · 2020-06-17T19:40:24Z

@mikaelarguedas I've updated the path to the RTI OpenSSL libraries on lore. Please try again.

Connext 5.3.1 requires an older version of openssl which is no longer available from the system macOS installation. Connext supplies a compatible openssl distribution and ros2/system_tests#409 sets up a convention for using it only for the security tests which require it when running with Connext. Signed-off-by: Steven! Ragnarök <[email protected]>

Signed-off-by: Mikael Arguedas <[email protected]>

mikaelarguedas · 2020-06-17T20:19:51Z

Lore Release:
Lore Debug:

Looks like this can finally be merged

kyrofa · 2020-06-17T20:23:44Z

I've updated the path to the RTI OpenSSL libraries on lore.

Thank you @jacobperron! @mikaelarguedas, thank you for your continued efforts.

mikaelarguedas requested a review from nuclearsandwich April 18, 2020 16:07

This was referenced Apr 18, 2020

Make Connext use Connext's openssl on all platforms ros2/system_tests#409

Merged

Foxy support ros2/sros2#180

Closed

sloretz mentioned this pull request May 1, 2020

Openssl 1.0 osrf/homebrew-ros2#8

Closed

sloretz mentioned this pull request May 5, 2020

Platform on OSX may start with macos #444

Merged

mikaelarguedas mentioned this pull request May 6, 2020

start valid date a day before to account for timezone mismatch ros2/sros2#209

Merged

mikaelarguedas mentioned this pull request May 7, 2020

Start date explicit timezone to UTC for all files ros2/sros2#210

Closed

sloretz mentioned this pull request May 7, 2020

Install RTI version of openssl on OSX ros2/ros2_documentation#662

Merged

kyrofa mentioned this pull request May 20, 2020

test_security tests failing with Connext on OS X ros2/build_farmer#269

Closed

sloretz assigned jacobperron May 29, 2020

nuclearsandwich and others added 5 commits June 17, 2020 22:09

use homebrew openssl 1.0

d5bc7d3

Signed-off-by: Mikael Arguedas <[email protected]>

clarify comment

4741285

Signed-off-by: Mikael Arguedas <[email protected]>

update path to rti openssl

0cf36b0

Signed-off-by: Mikael Arguedas <[email protected]>

typo

2a0eccd

Signed-off-by: Mikael Arguedas <[email protected]>

mikaelarguedas force-pushed the mikael/rti-openssl branch from 9801505 to 2a0eccd Compare June 17, 2020 20:19

jacobperron approved these changes Jun 17, 2020

View reviewed changes

jacobperron merged commit 8c758cf into master Jun 17, 2020

delete-merged-branch bot deleted the mikael/rti-openssl branch June 17, 2020 20:33

jacobperron mentioned this pull request Jun 25, 2020

Connext security tests failing on Windows #488

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use OpenSSL 1.0.2 for Connext on MacOS #436

Use OpenSSL 1.0.2 for Connext on MacOS #436

mikaelarguedas commented Apr 18, 2020

nuclearsandwich commented Apr 23, 2020

mikaelarguedas commented Apr 23, 2020

sloretz commented Apr 30, 2020

sloretz commented May 1, 2020 •

edited

Loading

sloretz commented May 1, 2020

mikaelarguedas commented May 1, 2020

sloretz commented May 1, 2020

sloretz commented May 5, 2020

ruffsl commented May 5, 2020

sloretz commented May 6, 2020

mikaelarguedas commented May 6, 2020

ruffsl commented May 6, 2020 •

edited

Loading

sloretz commented May 6, 2020 •

edited

Loading

mikaelarguedas commented May 13, 2020

sloretz commented May 13, 2020

mikaelarguedas commented May 13, 2020

sloretz commented May 13, 2020

mikaelarguedas commented May 13, 2020

sloretz commented May 13, 2020

mikaelarguedas commented May 14, 2020

mikaelarguedas commented May 20, 2020

sloretz commented May 29, 2020

mikaelarguedas commented Jun 15, 2020

jacobperron commented Jun 16, 2020

jacobperron commented Jun 16, 2020

mikaelarguedas commented Jun 17, 2020 •

edited

Loading

jacobperron commented Jun 17, 2020

mikaelarguedas commented Jun 17, 2020

kyrofa commented Jun 17, 2020

Use OpenSSL 1.0.2 for Connext on MacOS #436

Use OpenSSL 1.0.2 for Connext on MacOS #436

Conversation

mikaelarguedas commented Apr 18, 2020

nuclearsandwich commented Apr 23, 2020

mikaelarguedas commented Apr 23, 2020

sloretz commented Apr 30, 2020

sloretz commented May 1, 2020 • edited Loading

sloretz commented May 1, 2020

mikaelarguedas commented May 1, 2020

sloretz commented May 1, 2020

sloretz commented May 5, 2020

ruffsl commented May 5, 2020

sloretz commented May 6, 2020

mikaelarguedas commented May 6, 2020

ruffsl commented May 6, 2020 • edited Loading

sloretz commented May 6, 2020 • edited Loading

mikaelarguedas commented May 13, 2020

sloretz commented May 13, 2020

mikaelarguedas commented May 13, 2020

sloretz commented May 13, 2020

mikaelarguedas commented May 13, 2020

sloretz commented May 13, 2020

mikaelarguedas commented May 14, 2020

mikaelarguedas commented May 20, 2020

sloretz commented May 29, 2020

mikaelarguedas commented Jun 15, 2020

jacobperron commented Jun 16, 2020

jacobperron commented Jun 16, 2020

mikaelarguedas commented Jun 17, 2020 • edited Loading

jacobperron commented Jun 17, 2020

mikaelarguedas commented Jun 17, 2020

kyrofa commented Jun 17, 2020

sloretz commented May 1, 2020 •

edited

Loading

ruffsl commented May 6, 2020 •

edited

Loading

sloretz commented May 6, 2020 •

edited

Loading

mikaelarguedas commented Jun 17, 2020 •

edited

Loading