Skip to content
This repository has been archived by the owner on Mar 31, 2022. It is now read-only.

Executing SegmentRunner failed: null #130

Closed
mlowicki opened this issue Dec 3, 2015 · 3 comments
Closed

Executing SegmentRunner failed: null #130

mlowicki opened this issue Dec 3, 2015 · 3 comments

Comments

@mlowicki
Copy link

mlowicki commented Dec 3, 2015

Just trying to launch this tool for the very first time:

./spreaper add-cluster 10.210.3.221
./spreaper repair productioncluster sync --tables user_quota

and I'm getting only:

INFO   [2015-12-03 13:47:03,600] [productioncluster:1:3184] c.s.r.s.SegmentRunner - It is ok to repair segment '3184' on repair run with id '1' 
INFO   [2015-12-03 13:47:03,727] [productioncluster:1:3184] c.s.r.c.JmxProxy - Triggering repair of range (5191522389173550088,5192411715538110057] for keyspace "sync" on host 10.195.15.167, with repair parallelism DATACENTER_AWARE, in cluster with Cassandra version '2.1.11' (can use DATACENTER_AWARE 'true'), for column families: [user_quota] 
ERROR  [2015-12-03 13:47:04,097] [productioncluster:1:3184] c.s.r.s.RepairRunner - Executing SegmentRunner failed: null 
INFO   [2015-12-03 13:47:04,897] [productioncluster:1:3202] c.s.r.s.SegmentRunner - It is ok to repair segment '3202' on repair run with id '1' 
INFO   [2015-12-03 13:47:05,138] [productioncluster:1:3202] c.s.r.c.JmxProxy - Triggering repair of range (5287223852807787957,5288653625584332214] for keyspace "sync" on host 10.195.15.176, with repair parallelism DATACENTER_AWARE, in cluster with Cassandra version '2.1.11' (can use DATACENTER_AWARE 'true'), for column families: [user_quota] 
ERROR  [2015-12-03 13:47:05,502] [productioncluster:1:3202] c.s.r.s.RepairRunner - Executing SegmentRunner failed: null 
INFO   [2015-12-03 13:47:05,930] [productioncluster:1:3232] c.s.r.s.SegmentRunner - It is ok to repair segment '3232' on repair run with id '1' 
INFO   [2015-12-03 13:47:06,002] [productioncluster:1:3232] c.s.r.c.JmxProxy - Triggering repair of range (5392661295720643869,5396401862543039721] for keyspace "sync" on host 10.210.3.221, with repair parallelism DATACENTER_AWARE, in cluster with Cassandra version '2.1.11' (can use DATACENTER_AWARE 'true'), for column families: [user_quota] 
ERROR  [2015-12-03 13:47:06,103] [productioncluster:1:3232] c.s.r.s.RepairRunner - Executing SegmentRunner failed: null 
INFO   [2015-12-03 13:47:06,464] [productioncluster:1:3238] c.s.r.s.SegmentRunner - It is ok to repair segment '3238' on repair run with id '1' 
INFO   [2015-12-03 13:47:06,518] [productioncluster:1:3238] c.s.r.c.JmxProxy - Triggering repair of range (5428508491334977628,5429554845014064640] for keyspace "sync" on host 10.210.3.117, with repair parallelism DATACENTER_AWARE, in cluster with Cassandra version '2.1.11' (can use DATACENTER_AWARE 'true'), for column families: [user_quota] 
ERROR  [2015-12-03 13:47:06,601] [productioncluster:1:3238] c.s.r.s.RepairRunner - Executing SegmentRunner failed: null 
INFO   [2015-12-03 13:47:06,665] [productioncluster:1:3244] c.s.r.s.SegmentRunner - It is ok to repair segment '3244' on repair run with id '1' 
INFO   [2015-12-03 13:47:06,748] [productioncluster:1:3244] c.s.r.c.JmxProxy - Triggering repair of range (5447912260606081125,5452254409804456671] for keyspace "sync" on host 10.210.3.162, with repair parallelism DATACENTER_AWARE, in cluster with Cassandra version '2.1.11' (can use DATACENTER_AWARE 'true'), for column families: [user_quota] 
ERROR  [2015-12-03 13:47:06,836] [productioncluster:1:3244] c.s.r.s.RepairRunner - Executing SegmentRunner failed: null

I'm using C* 2.1.11 in two datacenter 8 nodes each and cassandra-reaper 0.2.4.

ping works fine:

mlowicki:bin mlowicki$ ./spreaper ping
# Report improvements/bugs at https://github.com/spotify/cassandra-reaper/issues
# ------------------------------------------------------------------------------
# Sending PING to Reaper...
# [Reply] Cassandra Reaper ping resource: PONG
# Cassandra Reaper is answering in: localhost:8080

cluster seems to be registered correctly:

mlowicki:bin mlowicki$ ./spreaper list-clusters
# Report improvements/bugs at https://github.com/spotify/cassandra-reaper/issues
# ------------------------------------------------------------------------------
# Listing all registered Cassandra clusters
# Found 1 clusters:
productioncluster

configuration file:

segmentCount: 200 
repairParallelism: DATACENTER_AWARE
repairIntensity: 0.9 
scheduleDaysBetween: 7
#daysToExpireAfterDone: 2
repairRunThreadCount: 15
hangingRepairTimeoutMins: 30
storageType: memory
enableCrossOrigin: true

logging:
  level: INFO
  loggers:
    io.dropwizard: INFO
    org.eclipse.jetty: INFO
  appenders:
    - type: console
      logFormat: "%-6level [%d] [%t] %logger{5} - %msg %n"

server:
  type: default
  applicationConnectors:
    - type: http
      port: 8080
      bindHost: 0.0.0.0
  adminConnectors:
    - type: http
      port: 8081
      bindHost: 0.0.0.0

database:
  driverClass: org.postgresql.Driver
  user: pg-user
  password: pg-pass
  url: jdbc:postgresql://db.example.com/db-prod

jmxAuth:
    username: controlRole
    password: XXX
@mlowicki
Copy link
Author

mlowicki commented Dec 3, 2015

Some debug logs:

INFO   [2015-12-03 14:01:19,396] [productioncluster:1:529] c.s.r.c.JmxProxy - Triggering repair of range (-6671567793146166305,-6666273419541198320] for keyspace "sync" on host 10.210.3.160, with repair parallelism DATACENTER_AWARE, in cluster with Cassandra version '2.1.11' (can use DATACENTER_AWARE 'true'), for column families: [user_quota] 
ERROR  [2015-12-03 14:01:19,426] [productioncluster:1:523] c.s.r.s.RepairRunner - Executing SegmentRunner failed: null 
DEBUG  [2015-12-03 14:01:19,426] [productioncluster:1:601] c.s.r.s.SegmentRunner - Run repair for segment #601 
DEBUG  [2015-12-03 14:01:19,460] [productioncluster:1:529] c.s.r.c.JmxProxy - close JMX connection to '10.210.3.160': service:jmx:rmi:///jndi/rmi://10.210.3.160:7199/jmxrmi 
DEBUG  [2015-12-03 14:01:19,493] [productioncluster:1:505] c.s.r.c.JmxProxy - JMX connection to 10.210.3.118 properly connected: service:jmx:rmi:///jndi/rmi://10.210.3.118:7199/jmxrmi 
DEBUG  [2015-12-03 14:01:19,494] [productioncluster:1:595] c.s.r.c.JmxProxy - JMX connection to 10.210.3.117 properly connected: service:jmx:rmi:///jndi/rmi://10.210.3.117:7199/jmxrmi 
DEBUG  [2015-12-03 14:01:19,519] [productioncluster:1:553] c.s.r.c.JmxProxy - close JMX connection to '10.195.15.183': service:jmx:rmi:///jndi/rmi://10.195.15.183:7199/jmxrmi 
ERROR  [2015-12-03 14:01:19,523] [productioncluster:1:529] c.s.r.s.RepairRunner - Executing SegmentRunner failed: null 
DEBUG  [2015-12-03 14:01:19,523] [productioncluster:1:607] c.s.r.s.SegmentRunner - Run repair for segment #607 
DEBUG  [2015-12-03 14:01:19,586] [productioncluster:1:505] c.s.r.c.JmxProxy - close JMX connection to '10.210.3.118': service:jmx:rmi:///jndi/rmi://10.210.3.118:7199/jmxrmi 
DEBUG  [2015-12-03 14:01:19,591] [productioncluster:1:541] c.s.r.c.JmxProxy - close JMX connection to '10.195.15.162': service:jmx:rmi:///jndi/rmi://10.195.15.162:7199/jmxrmi 
DEBUG  [2015-12-03 14:01:19,592] [productioncluster:1:547] c.s.r.c.JmxProxy - JMX connection to 10.195.15.162 properly connected: service:jmx:rmi:///jndi/rmi://10.195.15.162:7199/jmxrmi 
DEBUG  [2015-12-03 14:01:19,648] [productioncluster:1:505] c.s.r.s.SegmentRunner - checking host '10.210.3.162' for pending compactions and other repairs (can repair?) Run id '1' 
DEBUG  [2015-12-03 14:01:19,714] [productioncluster:1] c.s.r.s.RepairRunner - Did set segment id `1562` to slot 260 
DEBUG  [2015-12-03 14:01:19,714] [productioncluster:1] c.s.r.s.RepairRunner - preparing to repair segment 1562 on run with id 1 
DEBUG  [2015-12-03 14:01:19,762] [productioncluster:1:583] c.s.r.c.JmxProxy - JMX connection to 10.195.15.167 properly connected: service:jmx:rmi:///jndi/rmi://10.195.15.167:7199/jmxrmi 
DEBUG  [2015-12-03 14:01:19,774] [productioncluster:1:553] c.s.r.s.SegmentRunner - checking host '10.195.15.176' for pending compactions and other repairs (can repair?) Run id '1' 
DEBUG  [2015-12-03 14:01:19,805] [productioncluster:1:595] c.s.r.s.SegmentRunner - checking host '10.210.3.117' for pending compactions and other repairs (can repair?) Run id '1' 
DEBUG  [2015-12-03 14:01:19,835] [productioncluster:1:541] c.s.r.s.SegmentRunner - checking host '10.195.15.167' for pending compactions and other repairs (can repair?) Run id '1' 
DEBUG  [2015-12-03 14:01:19,837] [productioncluster:1:547] c.s.r.c.JmxProxy - close JMX connection to '10.195.15.162': service:jmx:rmi:///jndi/rmi://10.195.15.162:7199/jmxrmi 
DEBUG  [2015-12-03 14:01:19,927] [productioncluster:1:535] c.s.r.c.JmxProxy - JMX connection to 10.195.15.182 properly connected: service:jmx:rmi:///jndi/rmi://10.195.15.182:7199/jmxrmi 
DEBUG  [2015-12-03 14:01:20,035] [productioncluster:1:505] c.s.r.c.JmxProxy - JMX connection to 10.210.3.162 properly connected: service:jmx:rmi:///jndi/rmi://10.210.3.162:7199/jmxrmi

@mlowicki
Copy link
Author

mlowicki commented Dec 3, 2015

I've printed stacktrace and this is what I got:

ERROR  [2015-12-03 14:14:03,636] [productioncluster:1:163] c.s.r.s.RepairRunner - DUPA: java.lang.reflect.UndeclaredThrowableException
    at com.sun.proxy.$Proxy51.forceRepairRangeAsync(Unknown Source)
    at com.spotify.reaper.cassandra.JmxProxy.triggerRepair(JmxProxy.java:351)
    at com.spotify.reaper.service.SegmentRunner.runRepair(SegmentRunner.java:182)
    at com.spotify.reaper.service.SegmentRunner.run(SegmentRunner.java:93)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
Caused by: javax.management.ReflectionException: Operation forceRepairRangeAsync exists but not with this signature: (java.lang.String, java.lang.String, java.lang.String, int, java.util.Collection, java.util.Collection, [Ljava.lang.String;)
    at com.sun.jmx.mbeanserver.PerInterface.noSuchMethod(PerInterface.java:170)
    at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:135)
    at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
    at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
    at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
    at com.sun.jmx.remote.security.MBeanServerAccessController.invoke(MBeanServerAccessController.java:468)
    at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1487)
    at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97)
    at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1328)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1427)
    at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:848)
    at sun.reflect.GeneratedMethodAccessor102.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
    at sun.rmi.transport.Transport$2.run(Transport.java:202)
    at sun.rmi.transport.Transport$2.run(Transport.java:199)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.rmi.transport.Transport.serviceCall(Transport.java:198)
    at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:567)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:828)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.access$400(TCPTransport.java:619)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$1.run(TCPTransport.java:684)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$1.run(TCPTransport.java:681)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:681)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
    at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:275)
    at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:252)
    at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:161)
    at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source)
    at javax.management.remote.rmi.RMIConnectionImpl_Stub.invoke(Unknown Source)
    at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.invoke(RMIConnector.java:1029)
    at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:292)
    ... 13 more
Caused by: java.lang.NoSuchMethodException: forceRepairRangeAsync(java.lang.String, java.lang.String, java.lang.String, int, java.util.Collection, java.util.Collection, [Ljava.lang.String;)
    at com.sun.jmx.mbeanserver.PerInterface.noSuchMethod(PerInterface.java:168)
    at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:135)
    at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
    at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
    at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
    at com.sun.jmx.remote.security.MBeanServerAccessController.invoke(MBeanServerAccessController.java:468)
    at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1487)
    at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97)
    at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1328)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1427)
    at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:848)
    at sun.reflect.GeneratedMethodAccessor102.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
    at sun.rmi.transport.Transport$2.run(Transport.java:202)
    at sun.rmi.transport.Transport$2.run(Transport.java:199)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.rmi.transport.Transport.serviceCall(Transport.java:198)
    at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:567)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:828)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.access$400(TCPTransport.java:619)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$1.run(TCPTransport.java:684)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$1.run(TCPTransport.java:681)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:681)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

@Bj0rnen
Copy link
Contributor

Bj0rnen commented Dec 3, 2015

Closed as duplicate of #65.

@Bj0rnen Bj0rnen closed this as completed Dec 3, 2015
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mlowicki @Bj0rnen