Rexster SSL Support

[alszeb]

Summary - Rexster HTTP SSL

This pull request provides SSL security to Rexster’s HTTP based services. There are no API changes, only one line of code modified, and behavior is not affected unless SSL is enabled. Documentation is provided via changes to the Rexster Wiki. SSL is configured via rexster.xml. Designed with the intent to extend SSL support to RexPro.

Wiki Changes

Includes a new page for Rexster-SSL and minor changes to Rexster-Configuration.

Rexster-SSL.textile

This document includes basic info on where to find SSL configuration for Rexster and how to enable it. It does not provide documentation on configuring SSL in general.

Rexster-Configuration.textile

Added a line that that says Rexster supports SSL and links to Rexster-SSL.textile. Added property to configuration layout section: ‘rexster.http.enable-ssl’ with a default value of false. Added default SSL configuration to configuration layout section.

rexster.xml

Added a section for SSL configuration between 'security' and 'metrics' sections.
Added a property ‘http.enable-ssl’ which takes a boolean value and turns SSL on and off for HttpRexsterServer

Code

HttpRexsterServer.java

In configureNetworkListener() a check is performed to see if the property http.enable-ssl is true. When SSL is enabled a private method is called that configures SSL on the NetworkListener and calls setSecure(true) on it.

RexsterSslHelper.java

Contains most of the SSL setup code. Is constructed with a configuration (from rexster.xml) and provides helper methods to access SSL properties. Has a helper method for creating the SSLContext, which can be used in the future for securing RexPro with SSL.

Tests

Integration tests are included.

Limitations

Generally none, these changes should only provide new features. There is one gotcha where the http.base-uri property needs to be changed (http -> https) in order to get Dog House to work correctly; this is documented in the new Rexster-SSL wiki page.

Summary - Rexster RexPro SSL

This MR provides TLS/SSL security to Rexster’s binary protocol based interfaces. There are small additions to the API to permit SSL support with Rexster Clients (new methods and constructors outlined below). There are no behavior changes unless SSL is enabled. Documentation is provided via changes to the Rexster Wiki. SSL is configured via the currently existing SSL settings in rexster.xml. RexPro has one new property with which SSL support can be enabled/disabled (separate from http.enable-ssl).

Wiki Changes

Rexster-SSL.textile

New sections for RexPro Server, Rexster Console and RexPro-Java.

RexPro-Java.textile

Added new SSL properties to configuration table.

Rexster-Console.textile

Added ‘-s’ and ‘-sc’ parameter and description to ‘Command Line’ table. These are used to enable SSL and pass the location of the SSL config for Rexster-Console, respectively.

For example the following turns SSL on and specifies a file to look in for SSL configuration:

rexster-console.sh -s true --sc path_to_config.xml

Rexster-Configuration.textile

Added property to configuration layout section: ‘rexster.rexpro.ssl-enabled’ with a default value of false.

rexster.xml

‘rexpro.enable-ssl’ property added with default value of false.

Code

ConsoleSettings.java

Added additional command line parameter ‘-sc’ for SSL configuration. This takes an argument of the path to an SSL config file. The config file should be in the same format as the SSL settings for Rexster Server. (Only the SSL section is needed). Passing a value for this parameter automatically configures SSL to be on for the Console’s client.

RexsterConsole.java

Checks to see if SSL has been enabled by looking at its instance of ConsoleSettings. If so it creates a new RemoteRexsterSession with any passed in settings (see RemoteRexsterSession below). Else the original RemoteRexsterSession path is followed.

RemoteRexsterSession.java

New constructor which takes a String ‘sslConfigFile’ on top of the original constructor’s parameters. Creates a new RexProClientConnection which takes a String ‘sslConfigFile’ (see below).

RexProClientConnection.java

New Constructor which takes a String ‘sslConfig’. New method called by the new constructor “getSecureTransport(...)” which takes the sslConfig param and passes it to the RexsterClientSslFilterHelper (see below) in order to add SSL security via an SSL filter on the filter chain. Method “startTransport(...)” extracted from original constructor as used by both ‘secure’ and ‘unsecured’ constructors. “connection” member no longer final as it is initialized in a method rather than the constructor.

RexsterClientSslFilterHelper.java

Helper class to build SSLFilters appropriate for SSLClients. Provides client specific logging and checks for client SSL configuration file location based on passed in parameters. Allows for overrides of all ‘default’ SSL properties (for use with RexsterClientFactory - see changes below). Used to provide filters for both Console (via configuration file) and custom java based RexPro connections (via RexsterClientFactory).

RexsterClientTokens.java

Added two new tokens ‘ssl-enabled’ and ‘ssl-client-config’. Used in RexsterClientFactory in the same way the other tokens from this class are.

RexsterClientFactory.java

Adds ability to create SSL enabled clients. Added new tokens from RexsterClientTokens.java to base configuration (‘ssl-enabled’ and ‘ssl-client-config’).

If ‘ssl-enabled’ is set to true, a new method getSecureTransport(...) is used instead of the getTransport() method. Unlike the getTransport() method (which maintains a static instance of a transport), getSecureTransport(...) produces a new transport every time a client is initialized as SSL settings may have changed . getSecureTransport(...) results in an SSLFilter being added on to the transport’s filter chain.

Method getRexsterProcessor(...) was extracted from getTransport() as it is used by both getTransport() and getSecureTransport(...). it was modified to take a boolean ssl-enabled that determines if it adds an SSL filter to the filter chain.

RexProRexsterServer.java

Now checks if property ‘rexpro.enable-ssl’ is set to true in rexster.xml. If so, creates and adds an SSLFilter and adds it on to the filter chain on the RexPro Server.

Limitations

• If the client is rejected via failure to authenticate with SSL, the RexPro client will still make retry attempts; this can be changed via configuration, although this shouldn't be an issue with SSL properly configured.
• Adding additional integration tests seems to make some pre-existing memory management issues (specific to the integration tests) worse - Grizzly might not be good at cleaning up after itself?
• Grizzly's client SSL implementation, the "SSLFilter", doesn't seem to support hostname verification at this version. If this is something users of RexPro SSL need, they will have to figure out how to implement it themselves and/or try to add that functionality to the Grizzly's client SSL support.

[spmallette]

thanks for this - i'll not have time to review it in detail right away, but will get to it soon enough.

[alszeb]

Sounds good, thanks for letting me know.

[spmallette]

sorry, but i'd seriously lost track of reviewing this PR given latest work on Gremlin Server in TP3. not sure if you're still interested in this being merged or not, but i was wondering if there was a way to make this work such that by default rexster could start with ssl support via self-signed certificate. it's a nice way to get users started (in a non-production sense) if they can just "enable" ssl with one setting in the rexster.xml and be ssl-ready.

[alszeb]

That sounds good to me - I'll try to take a look soon. I've also got a branch that adds ssl support for RexPro that I can make a PR for once I make any changes you are interested in. Here are a few options I was thinking of for this change - let me know if you like any of them:

1. Add a self-signed cert into the repo
2. Automate generation of the cert when starting up Rexster
3. Add docs to make it easy to generate a self signed cert like http://docs.jboss.org/jbossweb/3.0.x/ssl-howto.html or https://wiki.eclipse.org/Jetty/Howto/Configure_SSL

[spmallette]

Yes - that all sounds good. SSL support for RexPro would be a nice symmetric feature to HTTP (i was going to ask about that actually).

Regarding the self-signed certificate i think it would be nice if it worked as we have it in Gremlin Server from TinkerPop3:

https://github.com/apache/incubator-tinkerpop/blob/master/gremlin-server/src/main/java/org/apache/tinkerpop/gremlin/server/AbstractChannelizer.java#L186

of course that uses a Netty construct (as Gremlin Server is based on Netty not Grizzly):

https://github.com/netty/netty/blob/master/handler/src/main/java/io/netty/handler/ssl/util/SelfSignedCertificate.java

In this way the cert is dynamically generated and thrown away. Perhaps that's what you were thinking in your description.

any additional docs you can provide would be helpful. i can give you access to the wiki to make ssl edits directy - editing docs/ isn't quite the right way - those are meant to be "static" and for "release" only.

[alszeb]

Great, I'll try to implement self-signed cert generation in a way similar to Gremlin Server and pull the doc changes out of the PR.

[alszeb]

It seems like there are several options for generating a self signed cert dynamically that we could use although I haven’t found any that I am really happy about. Here are some of the solutions I’ve seen so far:

• Use Bouncy Castle (requires the Bouncy Castle dependency)
• A lot of people are using the unsupported sun.* packages to do it like: http://bfo.com/blog/2011/03/08/odds_and_ends_creating_a_new_x_509_certificate.html although that seems like it could have risks
• Use Runtime.exec(…)/ ProcessBuilder to run keytool directly

Let me know if you have a preference on using any of these (or other) options.

[spmallette]

At this stage, given release of Gremlin Server in a few days, I'd prefer not adding new dependencies. The second approach sounds best using sun.* packages, but I don't understand the "risks" you referred to. Can you elaborate?

[alszeb]

Oracle gives no guarantees that the API in the sun.* packages will stay the same and these packages may not be available to alternative JVMs. I'm not too worried about them changing any time soon for our use case, but I am not sure if we'll start getting NoClassDefFoundError if we are running Rexster in a non-Oracle JVM (is that something that needs to be supported?). Oracle sums it up here: http://www.oracle.com/technetwork/java/faq-sun-packages-142232.html. If those won't be a problem I also like that option the most.

[spmallette]

Hmmm - I guess you could detect a non-Oracle jvm and act accordingly. Of course, it seems like a limitation I wouldn't impose on Rexster, if Rexster were not being replaced by Gremlin Server (I'd probably just use bouncy castle as netty did). That leads me to this issue...i feel like i'm asking you to do a bunch of extra work on this PR to get it merged, when Rexster is headed to "long term support". I guess I should ask the question as to whether or not its worth the effort involved on your end to do all this and if enough people would benefit from it to make it worthwhile. At this point I'd say that people who are running Rexster in production have figured out how to secure it with SSL (without Rexster directly supporting it) and likely wouldn't use this feature. wdyt?

[alszeb]

That sounds doable and I am happy to add it in. I think built in SSL support for Rexpro is a lot more useful than it is for REST as enabling SSL for Rexpro would significantly lower the barrier to entry for using Rexpro when SSL is a requirement (that was part of the motivation for these changes). If we are adding in SSL support for Rexpro it seems like it makes sense to add it in for http also. It would definitely be great to get these changes in with any code changes you require. Since additional features at this point might not be heavily used, it could make sense just to add a quick guide for setting up certs for Rexster rather than spending extra time making and testing additions to the code.

[spmallette]

ok - so just so i'm clear on what you're proposing. We skip the "self-siigned cert" thing and avoid bouncy castle dependency (or the other less than perfect approaches) and you provide a guide in the wiki to get users up to speed quickly with SSL (like how to create a self-signed cert themselves to add to rexster). Is that right?

[alszeb]

Exactly - thats what I was thinking.

[alszeb]

I pushed up a few integration tests, removed docs from the commits, and have a draft of the docs here: https://github.com/alszeb/rexster/wiki/Rexster-SSL. There are other document changes in the wiki as well, including changes to the sample configuration. Also in that wiki are sections related to the RexPro changes, which, of course, don't apply to this PR. Let me know what you think/ if there are any changes you want me to make. Thanks for taking the time to look at this!

[spmallette]

I'll look to review next week. Also, if you don't mind I'd like to just give you temporary rights to edit the wiki directly so that you can get all the changes in as needed.

[alszeb]

That works for me, thanks.

[spmallette]

The docs look pretty good. I'll give you access to the wiki once we square up the pull request and get it merged. You noted that the RexPro SSL feature isn't included in this PR. How do you want to do that? do you intend to submit a separate PR for RexPro? I think I'd prefer to just change the context of this PR to be "Rexster SSL Support" so that I can just evaluate/test it all at once, but I'm fine with two separate PRs if that's how you want to go. wdyt?

[alszeb]

Whatever is easiest is good - I appended the RexPro changes onto this merge request. A summary of those changes has been added to the description of the PR in the initial comment. Let me know if you have any questions.

[spmallette]

when i run the integration tests:

rexster-server$ mvn clean install -Pintegration-test

i get a failure:

-------------------------------------------------------------------------------
Test set: com.tinkerpop.rexster.ssl.RexsterHttpSslTest
-------------------------------------------------------------------------------
Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 12.986 sec <<< FAILURE!
testRequireClientAuthRequestThrowsSocketExceptionIfServerDoesntTrustClient(com.tinkerpop.rexster.ssl.RexsterHttpSslTest)  Time elapsed: 2.708 sec  <<< FAILURE!
java.lang.AssertionError: Unexpected exception.
    at org.junit.Assert.fail(Assert.java:88)
    at com.tinkerpop.rexster.ssl.RexsterHttpSslTest.testRequireClientAuthRequestThrowsSocketExceptionIfServerDoesntTrustClient(RexsterHttpSslTest.java:150)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
    at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
    at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:234)
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:133)
    at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:114)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:188)
    at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:166)
    at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:86)
    at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:101)
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)

[alszeb]

I pushed a commit to actually log what exception actually occurred during the test. My current thinking is that the test is too specific for SSL implementations across different systems. This test was written using osx yosemite with java 1.7. Could you possibly rerun the tests using mvn clean install -Pintegration-test -Djavax.net.debug=ssl and report what exception is actually being thrown? With the additional SSL logging, I'd expect to see an SSLHandshakeException server side (which, in my case, resulted in a socket exception client side). If the server is rejecting the client based on failure of the client to authenticate, then I think the best solution is to either broaden the expected exception (to IOException maybe...) or OR the expected exception with the one that youre getting.

[spmallette]

i'm on ubuntu fwiw. i pulled your changes, but i'm not sure there's too much different in the output of the test:

-------------------------------------------------------------------------------
Test set: com.tinkerpop.rexster.ssl.RexsterHttpSslTest
-------------------------------------------------------------------------------
Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 15.236 sec <<< FAILURE!
testRequireClientAuthRequestThrowsSocketExceptionIfServerDoesntTrustClient(com.tinkerpop.rexster.ssl.RexsterHttpSslTest)  Time elapsed: 3.169 sec  <<< FAILURE!
java.lang.AssertionError: Unexpected exception.
    at org.junit.Assert.fail(Assert.java:88)
    at com.tinkerpop.rexster.ssl.RexsterHttpSslTest.testRequireClientAuthRequestThrowsSocketExceptionIfServerDoesntTrustClient(RexsterHttpSslTest.java:152)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
    at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
    at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:234)
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:133)
    at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:114)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:188)
    at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:166)
    at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:86)
    at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:101)
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)

I'm also now seeing other test failures that weren't there before the pull....all together i'm looking at:

Failed tests:   testRequireClientAuthRequestThrowsSocketExceptionIfServerDoesntTrustClient(com.tinkerpop.rexster.ssl.RexsterHttpSslTest): Unexpected exception.

Tests in error: 
  testSessionRequestAndResponse(com.tinkerpop.rexster.rexpro.json.SessionRequestIntegrationTest): Java heap space
  testSessionRequestAndResponse(com.tinkerpop.rexster.rexpro.json.SessionRequestIntegrationTest)
  testSessionGraphDefinition(com.tinkerpop.rexster.rexpro.json.SessionRequestIntegrationTest): Java heap space
  testSessionGraphDefinition(com.tinkerpop.rexster.rexpro.json.SessionRequestIntegrationTest)

here's the stack trace:

-------------------------------------------------------------------------------
Test set: com.tinkerpop.rexster.rexpro.json.SessionRequestIntegrationTest
-------------------------------------------------------------------------------
Tests run: 5, Failures: 0, Errors: 4, Skipped: 0, Time elapsed: 9.07 sec <<< FAILURE!
testSessionRequestAndResponse(com.tinkerpop.rexster.rexpro.json.SessionRequestIntegrationTest)  Time elapsed: 1.524 sec  <<< ERROR!
java.lang.OutOfMemoryError: Java heap space
    at java.util.concurrent.atomic.AtomicReferenceArray.<init>(AtomicReferenceArray.java:94)
    at org.neo4j.kernel.impl.cache.GCResistantCache.<init>(GCResistantCache.java:96)
    at org.neo4j.kernel.impl.cache.GCResistantCacheProvider.newNodeCache(GCResistantCacheProvider.java:55)
    at org.neo4j.kernel.impl.core.DefaultCaches.node(DefaultCaches.java:51)
    at org.neo4j.kernel.InternalAbstractGraphDatabase.create(InternalAbstractGraphDatabase.java:467)
    at org.neo4j.kernel.InternalAbstractGraphDatabase.run(InternalAbstractGraphDatabase.java:259)
    at org.neo4j.kernel.EmbeddedGraphDatabase.<init>(EmbeddedGraphDatabase.java:106)
    at org.neo4j.kernel.EmbeddedGraphDatabase.<init>(EmbeddedGraphDatabase.java:81)
    at com.tinkerpop.blueprints.impls.neo4j.Neo4jGraph.<init>(Neo4jGraph.java:154)
    at com.tinkerpop.rexster.config.Neo4jGraphConfiguration.configureGraphInstance(Neo4jGraphConfiguration.java:70)
    at com.tinkerpop.rexster.config.GraphConfigurationContainer.getGraphFromConfiguration(GraphConfigurationContainer.java:124)
    at com.tinkerpop.rexster.config.GraphConfigurationContainer.<init>(GraphConfigurationContainer.java:54)
    at com.tinkerpop.rexster.server.XmlRexsterApplication.reconfigure(XmlRexsterApplication.java:99)
    at com.tinkerpop.rexster.server.XmlRexsterApplication.<init>(XmlRexsterApplication.java:34)
    at com.tinkerpop.rexster.rexpro.AbstractRexProIntegrationTest.setUp(AbstractRexProIntegrationTest.java:47)
    at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
    at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
testSessionRequestAndResponse(com.tinkerpop.rexster.rexpro.json.SessionRequestIntegrationTest)  Time elapsed: 1.524 sec  <<< ERROR!
java.lang.NullPointerException
    at com.tinkerpop.rexster.rexpro.AbstractRexProIntegrationTest.tearDown(AbstractRexProIntegrationTest.java:54)
    at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
    at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
    at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:234)
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:133)
    at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:114)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:188)
    at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:166)
    at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:86)
    at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:101)
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)
testSessionGraphDefinition(com.tinkerpop.rexster.rexpro.json.SessionRequestIntegrationTest)  Time elapsed: 0.878 sec  <<< ERROR!
java.lang.OutOfMemoryError: Java heap space
    at java.util.concurrent.atomic.AtomicReferenceArray.<init>(AtomicReferenceArray.java:94)
    at org.neo4j.kernel.impl.cache.GCResistantCache.<init>(GCResistantCache.java:96)
    at org.neo4j.kernel.impl.cache.GCResistantCacheProvider.newNodeCache(GCResistantCacheProvider.java:55)
    at org.neo4j.kernel.impl.core.DefaultCaches.node(DefaultCaches.java:51)
    at org.neo4j.kernel.InternalAbstractGraphDatabase.create(InternalAbstractGraphDatabase.java:467)
    at org.neo4j.kernel.InternalAbstractGraphDatabase.run(InternalAbstractGraphDatabase.java:259)
    at org.neo4j.kernel.EmbeddedGraphDatabase.<init>(EmbeddedGraphDatabase.java:106)
    at org.neo4j.kernel.EmbeddedGraphDatabase.<init>(EmbeddedGraphDatabase.java:81)
    at com.tinkerpop.blueprints.impls.neo4j.Neo4jGraph.<init>(Neo4jGraph.java:154)
    at com.tinkerpop.rexster.config.Neo4jGraphConfiguration.configureGraphInstance(Neo4jGraphConfiguration.java:70)
    at com.tinkerpop.rexster.config.GraphConfigurationContainer.getGraphFromConfiguration(GraphConfigurationContainer.java:124)
    at com.tinkerpop.rexster.config.GraphConfigurationContainer.<init>(GraphConfigurationContainer.java:54)
    at com.tinkerpop.rexster.server.XmlRexsterApplication.reconfigure(XmlRexsterApplication.java:99)
    at com.tinkerpop.rexster.server.XmlRexsterApplication.<init>(XmlRexsterApplication.java:34)
    at com.tinkerpop.rexster.rexpro.AbstractRexProIntegrationTest.setUp(AbstractRexProIntegrationTest.java:47)
    at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
    at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
testSessionGraphDefinition(com.tinkerpop.rexster.rexpro.json.SessionRequestIntegrationTest)  Time elapsed: 0.878 sec  <<< ERROR!
java.lang.NullPointerException
    at com.tinkerpop.rexster.rexpro.AbstractRexProIntegrationTest.tearDown(AbstractRexProIntegrationTest.java:54)
    at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
    at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
    at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:234)
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:133)
    at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:114)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:188)
    at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:166)
    at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:86)
    at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:101)
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)

"heap space" errors - weird.

[alszeb]

I noticed this sometimes when I was running the int tests. It seems like there is some kind of memory leak between integration tests (I think it might be grizzly or something to do with the TCPNIO connection not closing properly?). When I used top/ activity monitor I saw memory consumption increases a significant amount after some tests complete and never ends up dropping down until maven terminates; I was trying to get it to behave a little better by closing some of the clients/ graphs but while that reduced the number of running threads, it didn't help much memory wise. Without the SSL changes I think the integration tests were using > 4gigs of ram by the end of the tests with occasional heap space errors. Adding on new integration tests seemed to result in more occurring.

[alszeb]

Just wanted to check in - is there anything you want me to work on or take a look at? Thanks!

[spmallette]

I'm still broken on the tests (given my last comment) and haven't had time to figure out why things are failing....... :(

[alszeb]

no worries and no rush, it seemed like it could be a pretty tricky thing to solve!

[alszeb]

I had a chance to poke around a bit - it seems like neo4j might be the culprit. I took it out of the rexster settings.xml for the integration tests and it seems like memory consumption never goes above 1.5GB rather than >4GB. Haven't really had a chance to look any further into it, maybe something with Maven not handling shutdown hooks well?

[spmallette]

That's odd. I guess none of the integration tests ever touched neo4j - that's a different problem i suppose. I'll give your change a try later. thanks for digging deeper.

[spmallette]

Looks like those side errors are cleared up now that you got rid of Neo4j from the config, but I still have a failing test related to SSL - looks like the same one as the initial one I mentioned - any ideas on what this is?

-------------------------------------------------------------------------------
Test set: com.tinkerpop.rexster.ssl.RexsterHttpSslTest
-------------------------------------------------------------------------------
Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 11.934 sec <<< FAILURE!
testRequireClientAuthRequestThrowsSocketExceptionIfServerDoesntTrustClient(com.tinkerpop.rexster.ssl.RexsterHttpSslTest)  Time elapsed: 2.651 sec  <<< FAILURE!
java.lang.AssertionError: Unexpected exception.
    at org.junit.Assert.fail(Assert.java:88)
    at com.tinkerpop.rexster.ssl.RexsterHttpSslTest.testRequireClientAuthRequestThrowsSocketExceptionIfServerDoesntTrustClient(RexsterHttpSslTest.java:152)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
    at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
    at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:234)
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:133)
    at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:114)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:188)
    at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:166)
    at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:86)
    at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:101)
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)

[alszeb]

The exception was occurring because I expected the client to handle rejection by the server in the same way across different JDKs/SSL implementations. I'm ok with this because the test was making sure that Rexster server knows when its not getting a valid cert from the client and rejects the client accordingly (based on SSL debug logs the server's behavior appears to be the same across different JDKs). Also tested with openJDK8, oracle JDK8 (on fedora), and oracle JDK7 (os x), tests passed in these environments. On ubuntu with openjdk 6 or 7 that test failed. I researched the standards and looked at bug reports to try to see if one was right and one was wrong but I wasn't able to find anything definitive, so I just assumed both were right and am expecting an additional exception on the client side (I pushed this fix). Tests all passed when I ran them on Ubuntu after this fix.

Thanks and let me know if you run into any more issues, or if you want me to rebase any of these commits/ change anything.

[WolfgangFahl]

what happened to this pull request?

[spmallette]

iirc, development on Rexster was stopped as we were close to release Gremlin Server on 3.x and no further releases along the 2.x line were being considered.

[alszeb]

That sounds right to me.