Exception - "javax.crypto.BadPaddingException: bad record MAC" when trying to submit a job

[thejkane]

I am experiencing exception in [1] when trying to submit a job. So far I am experiencing this issue with following providers;

1. stampede
2. lonestar
3. trestles.

I am using JGlobus 2.0.6-RC2.

main] INFO gram.GramJobSubmission - RSL = &( hostCount = "1" )( queue = "normal" )( project = "TG-STA110014S" )( jobtype = "single" )( arguments = "30" )( proxy_timeout = "1" )( count = "1" )( executable = "/bin/hostname" )( maxwalltime = "1" )
[main] INFO gram.GramJobSubmission - Request to contact:trestles-login2.sdsc.edu:2119/jobmanager-pbstest2
Data transfer to the server failed. Caused by unwrap failed. Caused by Token had invalid integrity check. Caused by javax.net.ssl.SSLException: bad record MAC
at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:190)
at com.sun.net.ssl.internal.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1508)
at com.sun.net.ssl.internal.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:921)
at com.sun.net.ssl.internal.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:845)
at com.sun.net.ssl.internal.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:721)
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:607)
at org.globus.gsi.gssapi.GlobusGSSContextImpl.sslDataUnwrap(GlobusGSSContextImpl.java:813)
at org.globus.gsi.gssapi.GlobusGSSContextImpl.unwrap(GlobusGSSContextImpl.java:1694)
at org.globus.gsi.gssapi.GlobusGSSContextImpl.unwrap(GlobusGSSContextImpl.java:1631)
at org.globus.gsi.gssapi.net.GssInputStream.unwrap(GssInputStream.java:43)
at org.globus.gsi.gssapi.net.impl.GSIGssInputStream.readMsg(GSIGssInputStream.java:47)
at org.globus.gsi.gssapi.net.GssInputStream.hasData(GssInputStream.java:85)
at org.globus.gsi.gssapi.net.GssInputStream.read(GssInputStream.java:72)
at org.globus.util.http.HttpResponse.readLine(HttpResponse.java:55)
at org.globus.util.http.HttpResponse.parse(HttpResponse.java:106)
at org.globus.util.http.HttpResponse.(HttpResponse.java:43)
at org.globus.gram.internal.GatekeeperReply.(GatekeeperReply.java:43)
at org.globus.gram.Gram.request(Gram.java:335)
at org.globus.gram.Gram.request(Gram.java:272)
at org.globus.gram.GramJob.request(GramJob.java:253)
at org.apache.airavata.jobsubmission.gram.GramJobSubmission.executeJob(GramJobSubmission.java:64)
at org.apache.airavata.jobsubmission.gram.GramJobSubmissionTest.testExecuteJob(GramJobSubmissionTest.java:47)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at junit.framework.TestCase.runTest(TestCase.java:168)
at junit.framework.TestCase.runBare(TestCase.java:134)
at junit.framework.TestResult$1.protect(TestResult.java:110)
at junit.framework.TestResult.runProtected(TestResult.java:128)
at junit.framework.TestResult.run(TestResult.java:113)
at junit.framework.TestCase.run(TestCase.java:124)
at junit.framework.TestSuite.runTest(TestSuite.java:232)
at junit.framework.TestSuite.run(TestSuite.java:227)
at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)
Caused by: javax.crypto.BadPaddingException: bad record MAC
at com.sun.net.ssl.internal.ssl.EngineInputRecord.decrypt(EngineInputRecord.java:226)
at com.sun.net.ssl.internal.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:914)
... 44 more
[main] ERROR gram.GramJobSubmission - org.globus.gram.GramException: Data transfer to the server failed [Caused by: unwrap failed [Caused by: Token had invalid integrity check [Caused by: bad record MAC]]]
1371093405861#id#FAILED#Data transfer to the server failed [Caused by: unwrap failed [Caused by: Token had invalid integrity check [Caused by: bad record MAC]]]

[thejkane]

It seems like under mention exception occurs when both "TLSv1", "SSLv3" are put as enabled protocols in SSLEngine. If we set only "SSLv3" protocol and appropriate cipher suites we will not get following exception and job will get submitted.

According to org.globus.gsi.gssapi.GlobusGSSContextImpl, the easy way of doing this is to set "forceSSLv3AndConstrainCipherSuitesForGram" to true.

But I am not sure where we should make the change

[itfanjun]

Hi, you can check if your jar is normal, a colleague and I call each other, he used version 1.6 jar, I used 1.5, he kept reporting errors, in fact ECC support 1.5, then he changed to 1.5, it is normal