Consider moving compactor and scan server path cleaners to a utility

[dlmarion]

The Manager and CompactionCoordinator classes start cleanup threads (here and here) that remove paths in ZooKeeper that are no longer being used. The accumulo-cluster script uses the ZooZap utility to clean ZooKeeper paths when stopping the cluster (here for example).

If we removed the ZK cleaner threads in the Manager and CompactionCoordinator, then it would be relatively easy to determine how many and which compactors and scan servers were started, but not running.

[keith-turner]

then it would be relatively easy to determine how many and which compactors and scan servers were started, but not running.

Is this for the case of determining how many server processes died?

[dlmarion]

then it would be relatively easy to determine how many and which compactors and scan servers were started, but not running.

Is this for the case of determining how many server processes died?

Stopped or died. It would make it easier to detect down servers for the monitor or other utilities.

[dlmarion]

@keith-turner and I discussed this - the idea behind this issue is to use ZooKeeper as a mechanism to understand the intended state of the system. If we know the intended state of the system via paths in ZooKeeper, then we can identify which servers are down by looking for paths that don't have an associated lock. For users using accumulo-cluster, ZooKeeper would be cleaned up automatically when performing a stop operation as accumulo-cluster calls ZooZap. My idea here was that a utility could be created and run periodically by the user to clean up ZooKeeper paths when their intended deployment layout has changed. @keith-turner suggested that it might be too easy for ZooKeeper to get polluted with old paths if, for example, the user doesn't run the utility or in a case where the user is using an orchestration system like Kubernetes that stops and starts pods when needed that end up having different hostnames. @keith-turner suggested allowing the user to specify how many processes by resource group and type that they are intending to run, then we can compare the actual running count against the intended deployment. We might be able to convey this information in a property value (json?).