Suppress non-critical messages to prevent logs spamming

[tanjinx]

Description

suppress these 2 massive logging messages:

"keyspace event resolved: msgdata/msgdata is now consistent (serving: true)"
"disruption in shard msgdata/4780-47c0 resolved (serving: true)"

$ jq .msg jq.log|awk '{print $1}'|sort|uniq -c
     29 "Adding
    138 "[core]
  13000 "disruption
     53 "Error
     91 "HealthCheckUpdate(Serving
   1708 "keyspace
     26 "Removing
   7661 "tablet
     17 "Tablet

$ jq .level jq.log|sort|uniq -c
     57 "error"
  14867 "info"
   7799 "warn"

Related Issue(s)

Checklist

• "Backport to:" labels have been added if this change should be back-ported
• Tests were added or are not required
• Did the new or modified tests pass consistently locally and on the CI
• Documentation was added or is not required

Deployment Notes

[timvaillancourt]

@tanjinx I feel this change is necessary only if this filtering cannot be done outside of Vitess and/or there is no way to remove the artificial limits we've hit with the logging pipeline

What I suggest is we first:

• Check if there is a way to do this filtering in a different layer in the logging pipeline, outside of Vitess
• Ask for an exception to what seems to be an artificial limit/threshold
• Ask for a dedicated search index

Reducing how many logs we index by changing the Vitess code itself has a maintenance cost that can never be paid down and is difficult to re-enable (new build, CI, deploy) if we needed to re-enable these log lines for troubleshooting

That said, if Vitess changes is the only way forward, I suggest this is a v14 change only and it retires with v15. cc @vmogilev / @roderickyao / @venkatraju

[tanjinx]

We need to justify ourselves about the real value of those logs, are they really all critical for us? My experience might be limited, the only one or two messages which were helpful for me to debug vtgate issues during the incident is that the vtgate initialization, the tablet health state change, the query log and query errors.

ie I did not see any value of this message tabletserver is shutdown from a Restore-type tablet. And that message is logged every few seconds on every vtgate in the web and jobqueue pool.

tanjin.xu@z-kube-beach-tanjin-xu-iad-hedgehog:~$ grep "tabletserver is shutdown" vtgate.web.log |wc -l
8668
tanjin.xu@z-kube-beach-tanjin-xu-iad-hedgehog:~$ grep "tabletserver is shutdown" vtgate.web.log |grep "type:RESTORE"|wc -l
5686

[github-actions]

This PR is being marked as stale because it has been open for 30 days with no activity. To rectify, you may do any of the following:

• Push additional commits to the associated branch.
• Remove the stale label.
• Add a comment indicating why it is not stale.

If no action is taken within 7 days, this PR will be closed.