Bug Report: Unimplemented desc = method ReadBinlogFilesTimestamps not implemented`

[marf]

Overview of the Issue

Hello, after upgrading to Vitess v18 and running incremental backups using the command

vtctlclient --action_timeout=2h0m0s -- Backup --incremental_from_pos="auto" uk-3946508899

I received the following error:

E1112 10:26:39.077435 273 main.go:105] remote error: rpc error: code = Unknown desc = TabletManager.Backup on uk-3946508899 error: reading timestamps from binlog files [vt-3946508899-bin.000582 vt-3946508899-bin.000583 vt-3946508899-bin.000584 vt-3946508899-bin.000585 vt-3946508899-bin.000586 vt-3946508899-bin.000587 vt-3946508899-bin.000588 vt-3946508899-bin.000589 vt-3946508899-bin.000590 vt-3946508899-bin.000591 vt-3946508899-bin.000592 vt-3946508899-bin.000593 vt-3946508899-bin.000594 vt-3946508899-bin.000595 vt-3946508899-bin.000596 vt-3946508899-bin.000597 vt-3946508899-bin.000598 vt-3946508899-bin.000599 vt-3946508899-bin.000600 vt-3946508899-bin.000601 vt-3946508899-bin.000602 vt-3946508899-bin.000603 vt-3946508899-bin.000604 vt-3946508899-bin.000605 vt-3946508899-bin.000606 vt-3946508899-bin.000607 vt-3946508899-bin.000608 vt-3946508899-bin.000609 vt-3946508899-bin.000610 vt-3946508899-bin.000611 vt-3946508899-bin.000612 vt-3946508899-bin.000613 vt-3946508899-bin.000614 vt-3946508899-bin.000615 vt-3946508899-bin.000616 vt-3946508899-bin.000617 vt-3946508899-bin.000618 vt-3946508899-bin.000619 vt-3946508899-bin.000620 vt-3946508899-bin.000621 vt-3946508899-bin.000622 vt-3946508899-bin.000623 vt-3946508899-bin.000624 vt-3946508899-bin.000625 vt-3946508899-bin.000626 vt-3946508899-bin.000627 vt-3946508899-bin.000628 vt-3946508899-bin.000629 vt-3946508899-bin.000630 vt-3946508899-bin.000631 vt-3946508899-bin.000632 vt-3946508899-bin.000633 vt-3946508899-bin.000634 vt-3946508899-bin.000635 vt-3946508899-bin.000636 vt-3946508899-bin.000637 vt-3946508899-bin.000638 vt-3946508899-bin.000639 vt-3946508899-bin.000640 vt-3946508899-bin.000641 vt-3946508899-bin.000642 vt-3946508899-bin.000643 vt-3946508899-bin.000644 vt-3946508899-bin.000645 vt-3946508899-bin.000646 vt-3946508899-bin.000647 vt-3946508899-bin.000648 vt-3946508899-bin.000649 vt-3946508899-bin.000650 vt-3946508899-bin.000651 vt-3946508899-bin.000652 vt-3946508899-bin.000653 vt-3946508899-bin.000654 vt-3946508899-bin.000655 vt-3946508899-bin.000656 vt-3946508899-bin.000657 vt-3946508899-bin.000658 vt-3946508899-bin.000659 vt-3946508899-bin.000660]: rpc error: code = Unimplemented desc = method ReadBinlogFilesTimestamps not implemented: reading timestamps from binlog files [vt-3946508899-bin.000582 vt-3946508899-bin.000583 vt-3946508899-bin.000584 vt-3946508899-bin.000585 vt-3946508899-bin.000586 vt-3946508899-bin.000587 vt-3946508899-bin.000588 vt-3946508899-bin.000589 vt-3946508899-bin.000590 vt-3946508899-bin.000591 vt-3946508899-bin.000592 vt-3946508899-bin.000593 vt-3946508899-bin.000594 vt-3946508899-bin.000595 vt-3946508899-bin.000596 vt-3946508899-bin.000597 vt-3946508899-bin.000598 vt-3946508899-bin.000599 vt-3946508899-bin.000600 vt-3946508899-bin.000601 vt-3946508899-bin.000602 vt-3946508899-bin.000603 vt-3946508899-bin.000604 vt-3946508899-bin.000605 vt-3946508899-bin.000606 vt-3946508899-bin.000607 vt-3946508899-bin.000608 vt-3946508899-bin.000609 vt-3946508899-bin.000610 vt-3946508899-bin.000611 vt-3946508899-bin.000612 vt-3946508899-bin.000613 vt-3946508899-bin.000614 vt-3946508899-bin.000615 vt-3946508899-bin.000616 vt-3946508899-bin.000617 vt-3946508899-bin.000618 vt-3946508899-bin.000619 vt-3946508899-bin.000620 vt-3946508899-bin.000621 vt-3946508899-bin.000622 vt-3946508899-bin.000623 vt-3946508899-bin.000624 vt-3946508899-bin.000625 vt-3946508899-bin.000626 vt-3946508899-bin.000627 vt-3946508899-bin.000628 vt-3946508899-bin.000629 vt-3946508899-bin.000630 vt-3946508899-bin.000631 vt-3946508899-bin.000632 vt-3946508899-bin.000633 vt-3946508899-bin.000634 vt-3946508899-bin.000635 vt-3946508899-bin.000636 vt-3946508899-bin.000637 vt-3946508899-bin.000638 vt-3946508899-bin.000639 vt-3946508899-bin.000640 vt-3946508899-bin.000641 vt-3946508899-bin.000642 vt-3946508899-bin.000643 vt-3946508899-bin.000644 vt-3946508899-bin.000645 vt-3946508899-bin.000646 vt-3946508899-bin.000647 vt-3946508899-bin.000648 vt-3946508899-bin.000649 vt-3946508899-bin.000650 vt-3946508899-bin.000651 vt-3946508899-bin.000652 vt-3946508899-bin.000653 vt-3946508899-bin.000654 vt-3946508899-bin.000655 vt-3946508899-bin.000656 vt-3946508899-bin.000657 vt-3946508899-bin.000658 vt-3946508899-bin.000659 vt-3946508899-bin.000660]: rpc error: code = Unimplemented desc = method ReadBinlogFilesTimestamps not implemented

In v17 this command was working fine.

Reproduction Steps

This is not related to a specific Vschema.

Binary Version

Version: 18.0.0 (Git revision 9a6f5262f7707ff80ce85c111d2ff686d85d29cc branch 'HEAD') built on Mon Nov  6 12:16:43 UTC 2023 by runner@fv-az422-64 using go1.21.3 linux/amd64

Operating System and Environment details

PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Linux 5.15.0-82-generic

x86_64

Log Fragments

No response

[shlomi-noach]

Thank you for submitting this issue. This seems like a compatibility issue. A v18 vtctldclient will fail when the underlying mysql daemon is 17.0 because it calls Mysqld.ReadBinlogFilesTimestamps, which is not available in 17.0.

This will have to be fixed in a patch release.

[marf]

``

Thank you for submitting this issue. This seems like a compatibility issue. A v18 vtctldclient will fail when the underlying mysql daemon is 17.0 because it calls Mysqld.ReadBinlogFilesTimestamps, which is not available in 17.0.

This will have to be fixed in a patch release.

Thank you! The problem is that also the vttablets in my cluster are on version v18.0.0

Those are the images I am using in my vitess kuberntes cluster:

  images:
    mysqld:
      mysql80Compatible: vitess/lite:v18.0.0-percona80
    mysqldExporter: prom/mysqld-exporter:v0.11.0
    vtadmin: vitess/vtadmin:v18.0.0
    vtbackup: vitess/lite:v18.0.0-percona80
    vtctld: vitess/lite:v18.0.0-percona80
    vtgate: vitess/lite:v18.0.0-percona80
    vtorc: vitess/lite:v18.0.0-percona80
    vttablet: vitess/lite:v18.0.0-percona80

The underlying MySQL daemon should use 18.0 according to those images.

[shlomi-noach]

Then that is something I do not understand. ReadBinlogFilesTimestamps is a gRPC method implemented in v18.0.

[marf]

Then that is something I do not understand. ReadBinlogFilesTimestamps is a gRPC method implemented in v18.0.

Yes, that is indeed very strange since my vitess cluster should be fully updated to v18.0

[shlomi-noach]

The bug is confirmed. There's a missing function implementation.

This is the mainstream fix: #14525, to be followed shortly by a release-18.0 fix (patch release scheduled for next week).

[shlomi-noach]

Backport to release-18.0: #14526

[shlomi-noach]

Fixed by #14526