Fix character set mapping for utf8 and use it in schema-tracking tables

[GuptaManan100]

Description

This PR builds on the changes made in #14904.

In that PR, we used utf8mb3 as the character set and utf8mb3_bin for the collation of schema tracking tables since we wanted them to be case-sensitive. However, we realized that utf8mb3 is deprecated in MySQL 5.7.

In this PR, we change the schema definitions to use utf8 instead of utf8mb3 and also make changes to Vitess to replicate the MySQL behavior where utf8 maps to utf8mb3 in MySQL 5.7 and utf8mb4 in MySQL 8.0.

Tests have been added to verify that the schema-diff creates the correct tables in MySQL 5.7 and MySQL 8.0 and the upgrades and downgrades generate the correct alter statements too.

Related Issue(s)

Checklist

• "Backport to:" labels have been added if this change should be back-ported to release branches
• If this change is to be back-ported to previous releases, a justification is included in the PR description
• Tests were added or are not required
• Did the new or modified tests pass consistently locally and on CI?
• Documentation was added or is not required

Deployment Notes

[vitess-bot]

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

• Ensure that the Pull Request has a descriptive title.
• Ensure there is a link to an issue (except for internal cleanup and flaky test fixes), new features should have an RFC that documents use cases and test cases.

Tests

• Bug fixes should have at least one unit or end-to-end test, enhancement and new features should have a sufficient number of tests.

Documentation

• Apply the release notes (needs details) label if users need to know about this change.
• New features should be documented.
• There should be some code comments as to why things are implemented the way they are.
• There should be a comment at the top of each new or modified test to explain what the test does.

New flags

• Is this flag really necessary?
• Flag names must be clear and intuitive, use dashes (-), and have a clear help text.

If a workflow is added or modified:

• Each item in Jobs should be named in order to mark it as required.
• If the workflow needs to be marked as required, the maintainer team must be notified.

Backward compatibility

• Protobuf changes should be wire-compatible.
• Changes to _vt tables and RPCs need to be backward compatible.
• RPC changes should be compatible with vitess-operator
• If a flag is removed, then it should also be removed from vitess-operator and arewefastyet, if used there.
• vtctl command output order should be stable and awk-able.

[GuptaManan100]

I have made changes based on what I think is the correct way of doing this, but I could be wrong. I would love to have a review on this PR from @vmg or @dbussink to ensure that this is indeed the correct way.

[GuptaManan100]

I am not sure about this change.

[dbussink]

This is a generated file based on the actual MySQL collations. So this change is wrong.

[dbussink]

In that PR, we used utf8mb3 as the character set and utf8mb3_bin for the collation of schema tracking tables since we wanted them to be case-sensitive. However, we realized that utf8mb3 is deprecated in MySQL 5.7.

I don't understand this. We didn't then afterwards realize. It was already known that it's deprecated, but since MySQL itself uses internally still utf8mb3 for all table information (see information_schema usage and in Vitess all usage of

vitess/go/mysql/collations/env.go

Lines 200 to 208 in e9c513e

	// SystemCollation is the default collation for the system tables
	// such as the information schema. This is still utf8mb3 to match
	// MySQLs behavior. This means that you can't use utf8mb4 in table
	// names, column names, without running into significant issues.
	var SystemCollation = TypedCollation{
	Collation: CollationUtf8mb3ID,
	Coercibility: CoerceCoercible,
	Repertoire: RepertoireUnicode,
	}

for example).

So there's no need for any follow up that I know of.

In this PR, we change the schema definitions to use utf8 instead of utf8mb3 and also make changes to Vitess to replicate the MySQL behavior where utf8 maps to utf8mb3 in MySQL 5.7 and utf8mb4 in MySQL 8.0.

This is not true. utf8 is still an alias for utf8mb3 in MySQL 8.0. And given MySQL's new versioning scheme, I don't think it will ever change for 8.0 and any changes will happen in a newer version.

[vmg]

Yeah I also don't think it's correct to change the encoding for these schema tables. The utf8mb3 is indeed deprecated but it's still being used in MySQL for these tables, so this change just breaks compatibility. Obviously it'd be neat to support emoji and extended codepoints in table names (would it tho?) but until MySQL also supports it, what's the point?

[GuptaManan100]

Thank-you both for looking at the PR! I have closed it in light of it not being a change that makes sense.