Take replication lag into account while selecting primary candidate

[GuptaManan100]

Description

This PR adds an additional dimension to choosing a new primary for PlannedReparentShard. If a tablet has a lot of replication lag, even more than wait-replicas-timeout, then waiting for replication to catch up would time out, so it is better to not consider that tablet for new primary

A new sub-flag --tolerable-replication-lag has been added to the command PlannedReparentShard that allows users to specify the amount of replication lag that is considered acceptable for a tablet to be eligible for promotion when Vitess makes the choice of a new primary.
This feature is opt-in and not specifying this sub-flag makes Vitess ignore the replication lag entirely.

A new flag in VTOrc with the same name has been added to control the behaviour of the PlannedReparentShard calls that VTOrc issues.

Related Issue(s)

• Fixes Feature Request: PRS should check replica lag and not use tablets if lag > wait-replicas-timeout #14633

Checklist

• "Backport to:" labels have been added if this change should be back-ported
• Tests were added or are not required
• Did the new or modified tests pass consistently locally and on the CI
• Documentation was added or is not required

Deployment Notes

[vitess-bot]

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

• Ensure that the Pull Request has a descriptive title.
• Ensure there is a link to an issue (except for internal cleanup and flaky test fixes), new features should have an RFC that documents use cases and test cases.

Tests

• Bug fixes should have at least one unit or end-to-end test, enhancement and new features should have a sufficient number of tests.

Documentation

• Apply the release notes (needs details) label if users need to know about this change.
• New features should be documented.
• There should be some code comments as to why things are implemented the way they are.
• There should be a comment at the top of each new or modified test to explain what the test does.

New flags

• Is this flag really necessary?
• Flag names must be clear and intuitive, use dashes (-), and have a clear help text.

If a workflow is added or modified:

• Each item in Jobs should be named in order to mark it as required.
• If the workflow needs to be marked as required, the maintainer team must be notified.

Backward compatibility

• Protobuf changes should be wire-compatible.
• Changes to _vt tables and RPCs need to be backward compatible.
• RPC changes should be compatible with vitess-operator
• If a flag is removed, then it should also be removed from vitess-operator and arewefastyet, if used there.
• vtctl command output order should be stable and awk-able.

[ajm188]

one suggestion for using Duration to make the final comparison read a bit more higher-level, up to you if you think it's not worth it!

[GuptaManan100]

@ajm188 that was a great suggestion. I have made the changes. Thank-you! 💕

[mattlord]

I like it! I'm not sure about the default value, so will leave that to you.