Feature Proposal: TicTac AAE and NextGenRepl CLI commands

[pjaclark]

Background

One issue we (TI Tokyo) are coming up against with users is the lack of easy runtime management and reporting on TicTac AAE and NextGenRepl. We've written some tools to report on TicTac AAE tree status and told people how to change runtime settings where we can, but it needs to be much easier. There used to be rial-repl for legacy replication and users expect something similar - especially on the reporting side.

This won't provide per-peer object/second or bytes/second counts, which would be very useful to add as well but would require considerably more work. If those were available, they vould be added as extra properties in the "status" outputs without the need for more commands.

Proposal

We propose adding the CLI and HTTP API commands below be added.

Some of these are identical to existing riak attach commands. Some are combinations, or have JSON output instead of an Erlang tuple output. Some require new code to be written or added (such as the aae tree status report). Some require minor tweaks to existing code (looking at rebuild wait and delay specifically).

We'd love for this to be available in the first OpenRiak KV release (3.2.3), but it might well be too late for that. OpenRiak KV 3.4.0 is probably the better target version. We'll start work on a PR for this, but if anyone has any feedback, changes, additions, issues, foreseeable issues, etc then they would be greatly appreciated.

Riak CLI

Tictac runtime management and reporting

• riak tictacaae rebuildwait [<n-in-hours>] [<nodename>]

  • If no <n> then returns runtime tictacaae_rebuildwait, optionally for specific node
  • If <n> then sets runtime tictacaae_rebuildwait, optionally for specific node

• riak tictacaae rebuilddelay [<n-in-minutes>] [<nodename>]

  • If no <n> then returns runtime tictacaae_rebuilddelay, optionally for specific node
  • If <n> then sets runtime tictacaae_rebuilddelay, optionally for specific node

• riak tictacaae rebuildtick [<n-in-milliseconds>] [<nodename>]

  • If no <n> then returns runtime tictacaae_rebuildtick, optionally for specific node
  • If <n> then sets runtime tictacaae_rebuildtick, optionally for specific node

• riak tictacaae exchangetick [<n-in-milliseconds>] [<nodename>]

  • If no <n> then returns runtime tictacaae_exchangetick, optionally for specific node
  • If <n> then sets runtime tictacaae_exchangetick, optionally for specific node

• riak tictacaae maxresults [<n>] [<nodename>]

  • If no <n> then returns runtime tictacaae_maxresults, optionally for specific node
  • If <n> then sets runtime tictacaae_maxresults, optionally for specific node

• riak tictacaae storeheads [enabled|disabled] [<nodename>]

  • If no enabled|disabled then returns runtime tictacaae_storeheads, optionally for specific node
  • If enabled|disabled then sets runtime tictacaae_storeheads, optionally for specific node

• riak tictacaae tokenbucket [enabled|disabled] [<nodename>]

  • If no enabled|disabled then returns runtime aae_tokenbucket, optionally for specific node
  • If enabled|disabled then sets runtime aae_tokenbucket, optionally for specific node

• riak tictacaae rebuildtreeworkers [<n>] [<nodename>]

  • If no <n> then returns runtime af1 pool size, optionally for specific node
  • If <n> then sets runtime af1 pool size, optionally for specific node

• riak tictacaae rebuildstoreworkers [<n>] [<nodename>]

  • If no <n> then returns runtime be pool size, optionally for specific node
  • If <n> then sets runtime be pool size, optionally for specific node

• riak tictacaae aaefoldworkers [<n>] [<nodename>]

  • If no <n> then returns runtime af4 pool size, optionally for specific node
  • If <n> then sets runtime af4 pool size, optionally for specific node

• riak tictacaae rebuild-soon all|<nodename>|<partition-id>

  • Changes NextRebuildDate to (now + delay) on all partitions, all partitions on a specific node, or a specific partition
  • Will wait for next tictacaae_rebuildtick to start

• riak tictacaae rebuild-next all|<nodename>|<partition-id>

  • Changes NextRebuildDate to now on all partitions, all partitions on a specific node, or a specific partition
  • Will wait for next tictacaae_rebuildtick to start

• riak tictacaae rebuild-now all|<nodename>|<partition-id>

  • Changes NextRebuildDate to now on all partitions, all partitions on a specific node, or a specific partition
  • Starts an immediate rebuild check of all partitions, all partitions on a specific node, or a specific partition

• riak tictacaae treestatus [<partition-id>|<nodename>] [by-last-rebuilt|by-next-rebuild|by-partition]

  • Returns tree status as table (aae_progress_status_pp) optionally for a specific partition or node, and optionally sorted by LastRebuildDate, NextRebuildDate or Partition ID

• riak tictacaae treestatus-json [<partition-id>|<nodename>]

  • Returns tree status as JSON blob (aae_progress_status) optionally for a specific partition or node

Admin tasks

• riak tictacaae fold findkeys [bucket all|"bucketname"|{"typename","bucketname"}] [keyrange all|{"from", "to"}] [modified all|{"YYYY-MM-DD hh:mm:ss", "YYYY-MM-DD hh:mm:ss"}] [[siblingcount <n>]|[objectsize <n-in-bytes>]]

  • Returns a JSON array of keys matching filters
  • Will return warning at start if storeheads required to work but storeheads not enabled

• riak tictacaae fold findtombstones [bucket all|"bucketname"|{"typename","bucketname"}] [keyrange all|{"from", "to"}] [modified all|{"YYYY-MM-DD hh:mm:ss", "YYYY-MM-DD hh:mm:ss"}] [[siblingcount <n>]|[objectsize <n-in-bytes>]]

  • Returns a JSON array of tombstones matching filters
  • Will return warning at start if storeheads required to work but storeheads not enabled

• riak tictacaae fold listbuckets

  • Returns a JSON array of bucket names

• riak tictacaae fold countkeys [bucket all|"bucketname"|{"typename","bucketname"}] [keyrange all|{"from", "to"}] [modified all|{"YYYY-MM-DD hh:mm:ss", "YYYY-MM-DD hh:mm:ss"}]

  • Returns the number of keys that match the filters
  • Will return warning at start if storeheads required to work but storeheads not enabled

• riak tictacaae fold counttombstones [bucket all|"bucketname"|{"typename","bucketname"}] [keyrange all|{"from", "to"}] [modified all|{"YYYY-MM-DD hh:mm:ss", "YYYY-MM-DD hh:mm:ss"}]

  • Returns the number of tombstones that match the filters
  • Will return warning at start if storeheads required to work but storeheads not enabled

• riak tictacaae fold objectstatistics [bucket all|"bucketname"|{"typename","bucketname"}] [keyrange all|{"from", "to"}] [modified all|{"YYYY-MM-DD hh:mm:ss", "YYYY-MM-DD hh:mm:ss"}]

  • Returns objecst statistics as a JSON object for Riak objects that match the filters
  • Will return warning at start if storeheads required to work but storeheads not enabled

• riak tictacaae fold erasekeys [bucket all|"bucketname"|{"typename","bucketname"}] [keyrange all|{"from", "to"}] [modified all|{"YYYY-MM-DD hh:mm:ss", "YYYY-MM-DD hh:mm:ss"}]

  • Returns the number of keys erased that match the filters
  • Will return warning at start if storeheads required to work but storeheads not enabled

• riak tictacaae fold reaptombstones [bucket all|"bucketname"|{"typename","bucketname"}] [keyrange all|{"from", "to"}] [modified all|{"YYYY-MM-DD hh:mm:ss", "YYYY-MM-DD hh:mm:ss"}]

  • Returns the number of tombstones reaped that match the filters
  • Will return warning at start if storeheads required to work but storeheads not enabled

• riak tictacaae fold repairkeys [bucket all|"bucketname"|{"typename","bucketname"}] [keyrange all|{"from", "to"}] [modified all|{"YYYY-MM-DD hh:mm:ss", "YYYY-MM-DD hh:mm:ss"}]

  • Performs a read-repair on the keys that match the filters
  • Returns a JSON object with statistics
  • Will return warning at start if storeheads required to work but storeheads not enabled

NextGenRepl runtime management and reporting

Generic

• riak nextgenrepl status <nodename>
  • Returns for all nodes or a specific node the status of realtime, fullsync and sink.

Fullsync

• riak nextgenrepl fullsync <nodename> scope [disabled|all|bucket|type]

  • On specific node, get or set scope

• riak nextgenrepl fullsync <nodename> bucket-filter-name ["name"]

  • On specific node, get or set scope "bucket" filter

• riak nextgenrepl fullsync <nodename> bucket-filter-type ["type"]

  • On specific node, get or set scope "type" filter

• riak nextgenrepl fullsync <nodename> queue-name [<queuename>]

  • On specific node, get or set the fullsync queue name

• riak nextgenrepl fullsync <nodename> bidirectional-queue-name [<queuename>]

  • On specific node, get or set the bidirectional fullsync queue name on the fullsync peer

• riak nextgenrepl fullsync <nodename> max-results [<n>]

  • On specific node, get or set maximum number of AAE segments to do in one go

• riak nextgenrepl fullsync <nodename> range-boost [<n>]

  • On specific node, get or set the booster for range_check queries

• riak nextgenrepl fullsync <nodename> cluster-slice [<n>]

  • On specific node, get or set the planned number of nodes with the same fullsync settings in the cluster

• riak nextgenrepl fullsync <nodename> read-nval [<n>]

  • On specific node, get or set the "r" value for fullsync

• riak nextgenrepl fullsync <nodename> write-nval [<n>]

  • On specific node, get or set the "w" value for fullsync to use on the peer node

• riak nextgenrepl fullsync <nodename> peer [<ip-address>|<fqdn> <port> pb|http

  • On specific node, get or set peer name, port and protocol

• riak nextgenrepl fullsync <nodename> schedule

  • On specific node, get a summary of the check schedule

• riak nextgenrepl fullsync <nodename> schedule auto [n]

  • On specific node, get or set the number of "auto" checks

• riak nextgenrepl fullsync <nodename> schedule all [n]

  • On specific node, get or set the number of "all" checks

• riak nextgenrepl fullsync <nodename> schedule last-day [n]

  • On specific node, get or set the number of "day" checks

• riak nextgenrepl fullsync <nodename> schedule last-hour [n]

  • On specific node, get or set the number of "hour" checks

• riak nextgenrepl fullsync <nodename> schedule since-last [n]

  • On specific node, get or set the number of "range" checks

• riak nextgenrepl fullsync <nodename> schedule skip [n]

  • On specific node, get or set the number of "no" checks

• riak nextgenrepl fullsync <nodename> workers <n>

  • On specific node, get or set the af3 pool size

Realtime

• riak nextgenrepl realtime <nodename> source enable

  • On specific node, enable the node to act as a realtime source

• riak nextgenrepl realtime <nodename> source disable

  • On specific node, disable the node to act as a realtime source

• riak nextgenrepl realtime <nodename> source queues

  • On specific node, show a summary of all queues and statuses

• riak nextgenrepl realtime <nodename> source queue-status <name>

  • On specific node, show the status of a specific queue

• riak nextgenrepl realtime <nodename> source add-queue <name> [any|block-rtq|bucketname <name>|bucketprefix <prefix>|buckettype <type>]

  • On specific node, add a new realtime queue

• riak nextgenrepl realtime <nodename> source enable-queue <name>

  • On specific node, enabled a specific realtime queue

• riak nextgenrepl realtime <nodename> source disable-queue <name>

  • On specific node, disbled a specific realtime queue

• riak nextgenrepl realtime <nodename> source remove-queue <name>

  • On specific node, remove a specific realtime queue

Sink

• riak nextgenrepl realtime <nodename> sink enable

  • On specific node, enable the node to act as a realtime sink

• riak nextgenrepl realtime <nodename> sink disable

  • On specific node, disable the node to act as a realtime sink

• riak nextgenrepl realtime <nodename> sink queue [<name>]

  • On specific node, get or set the default peer queue name

• riak nextgenrepl realtime <nodename> sink peers

  • On specific node, show a summary of the statuses of all peers

• riak nextgenrepl realtime <nodename> sink peer-status <ip-address>|<fqdn>

  • On specific node, show the status of a specific peer

• riak nextgenrepl realtime <nodename> sink add-peer <ip-address>|<fqdn> <port> http|pb [<queuename>]

  • On specific node, add a peer

• riak nextgenrepl realtime <nodename> sink remove-peer <ip-address>|<fqdn>

  • On specific node, remove a specific peer

• riak nextgenrepl realtime <nodename> sink workers <n>

  • On specific node, set the number of workers available for the sink

• riak nextgenrepl realtime <nodename> sink workers-peer-limit <n>

  • On specific node, set the maximum number of workers available for the sink to use on each peer

HTTP API end points

Tictac runtime management and reporting

• /tictacaae/status[/<partition-id>|/<nodename>]

  • Same as CLI riak tictacaae treestatus-json
  • Does not include process IDs unlike the CLI

• /tictacaae/rebuild-soon[/all|/<nodename>|/<partition-id>]

  • Same as CLI riak tictacaae rebuild-soon

• /tictacaae/rebuild-next[/all|/<nodename>|/<partition-id>]

  • Same as CLI riak tictacaae rebuild-next

• /tictacaae/rebuild-now[/all|/<nodename>|/<partition-id>]

  • Same as CLI riak tictacaae rebuild-now

NextGenRepl runtime management and reporting

• /nextgenrepl/status[/<nodename>]
  • Same as riak nextgenrepl status
  • Does not include process IDs if in CLI version

[martinsumner]

Yikes! That's a lot of commands.

One generic concern is the problems we've had with our existing CLI commands. It is a real struggle to test them effectively via riak_test, and what tests that exist are very artificial. There's then been issues with riak admin commands mysteriously failing in a release. I would be particularly worried about implementing aae_folds by a admin command - as there is a lot of parsing to do for all the options, and hence a lot of testing to implement in some way.

The second generic concern is about commands that don't give an immediate response. For example, aae_folds again, they may take some time. What does the operator do during the wait? If they cancel and re-query - that won't cancel the original query, just double the wait for the next result. So how should the user feedback loop be managed for such commands?

Concern 1 - Testing, Easing Implementation

Because of the first problem, I had started to add helper functions to the riak_client module e.g. https://github.com/OpenRiak/riak_kv/blob/openriak-3.2/src/riak_client.erl#L1013-L1020. My thoughts at the time were in the long run to have a riak_console module to move these to, that could be run direct from riak remote_console, where you could riak_console:list_commands(). and riak_console:help(Command). then (say) run riak_console:remove_node_from_coverage().. so a minimal bit of erlang - but the process of adding commands and testing them becomes that bit easier.

I stopped progressing the above when we had an issue with a remote_console session which was disconnected by a network timeout, and caused a CPU core to burn busy until the node was restarted. So probably not the right answer, but testing CLI commands remains a problem if you want to introduce more.

Concern 2 - Responsiveness

One idea I had there was that a user-prompted fold should log the results to a file when they complete, not ask for the user to hang-on and wait. So, we could have a results.log alongside console.log etc - and if you request a query you get given a result reference you can tail the results.log and grep for.

A good example for this would be stats by bucket - so from remote_console one can have a function that will run (the safe version of) list buckets, and then request the stats for each bucket, and then return the stats by bucket. This on most production clusters can take along time - so it would be better if that was spawned to run in the background and print the results into results.log bucket by bucket as they are returned.

[martinsumner]

riak tictacaae storeheads [enabled|disabled] [<nodename>]

If no enabled|disabled then returns runtime tictacaae_storeheads, optionally for specific node
If enabled|disabled then sets runtime tictacaae_storeheads, optionally for specific node
riak tictacaae tokenbucket [enabled|disabled] [<nodename>]

If no enabled|disabled then returns runtime aae_tokenbucket, optionally for specific node
If enabled|disabled then sets runtime aae_tokenbucket, optionally for specific node
riak tictacaae rebuildtreeworkers [<n>] [<nodename>]

If no <n> then returns runtime af1 pool size, optionally for specific node
If <n> then sets runtime af1 pool size, optionally for specific node
riak tictacaae rebuildstoreworkers [<n>] [<nodename>]

If no <n> then returns runtime be pool size, optionally for specific node
If <n> then sets runtime be pool size, optionally for specific node
riak tictacaae aaefoldworkers [<n>] [<nodename>]

If no <n> then returns runtime af4 pool size, optionally for specific node
If <n> then sets runtime af4 pool size, optionally for specific node

None of these things are presently configurable at runtime i.e. the configuration values are read and set at startup, and never checked after startup. Do you intend to change this, as in some cases that would be significant work?

[martinsumner]

Glancing through a few further options, this is also the case for some others you propose.

[martinsumner]

It might be easier to start with the existing documented remote_console commands - https://github.com/OpenRiak/riak_kv/blob/openriak-3.4/docs/NextGenREPL-GettingStarted.md#useful-remote_console-commands - in that these things that are possible to change at run-time, and have been documented as changeable because changing them proved to be useful in a production system.

The peer discovery run-time changes aren't in that doc yet, but can be found here - https://github.com/OpenRiak/riak_kv/blob/openriak-3.4/src/riak_kv_replrtq_peer.erl#L63-L76. These are useful too.

[martinsumner]

The link above (for remote console commands) is broken, as this doc is now updated: https://github.com/OpenRiak/riak_kv/blob/openriak-3.2/docs/NextGenREPL-GettingStarted.md

The remote_console commands are now spread throughout the tutorial

[pjaclark]

CLI commands are essential for a lot of users, even at large multi-national companies. You would not believe the issues we have had with getting people to use riak attach properly.

I think start with the low-hanging fruit (those that require no code changes - just calling existing code) and come up with a testable way to implement these.

On that note, I really like the idea of making the CLI calls essentially a dumb pass-through to remote_console and having all the logic (including parameter checking) in there.

I also very much like the idea of running queries as background tasks and logging to a file. If the response comes back before a timeout, then print to screen. If timeout then print message about checking the log. For ctrl-C use a signal handler in the script file to tell people that the query is still running and that they should check the log file for the response.

[martinsumner]

This reminded me about making this available via the CLI - OpenRiak/riak_kv#45