tikv-migration: support Kafka integration tests

[pingyu]

Related PR: tikv/migration#382

[ti-chi-bot]

I have already done a preliminary review for you, and I hope to help you do a better job.

Review for "tikv-migration: support Kafka integration tests"

Summary

This pull request adds support for Kafka integration tests to the tikv-migration project. It introduces a new pipeline job and a new pod template file, and updates the pull_integration_test.groovy and tikv-migration-latest-presubmits.yaml files.

Potential Problems

There don't appear to be any major problems with this pull request. However, a few things to note:

• It's unclear what the purpose of the PROW_JOB_ID parameter in the pipeline job is. It's not used anywhere in the script, and it's not explained in the comments or the PR description. If it's not necessary, it should be removed.
• The volumeMounts field in the pod template file mounts the /tmp directory from the host into the containers. This could be a security issue if the host is compromised, since any container running in the pod could modify or read files in that directory. A more secure approach would be to use a ConfigMap or a Secret to store any necessary files, and mount that into the containers instead.
• The golang container in the pod template file requests 8GB of memory and 4 CPUs. This seems like a lot of resources for a single container running integration tests. It might be worth investigating whether this can be reduced without impacting the tests.
• The python3-requests container in the pod template file is used for reporting. It's not clear what kind of reporting is being done or where the reports are being sent. It might be worth adding more documentation or comments to explain this.
• Some of the comments in the pipeline script are unclear or incomplete. For example, the priority property in the properties block is commented out with no explanation as to what it does or why it's being commented out.

Fixing Suggestions

• Remove the PROW_JOB_ID parameter from the pipeline job.
• Use a ConfigMap or a Secret to store any necessary files, and mount that into the containers instead of the host's /tmp directory.
• Consider reducing the resource requests for the golang container.
• Add more documentation or comments to explain the purpose of the python3-requests container.
• Improve the comments in the pipeline script to make them more clear and complete.

[purelind]

Please make sure that the dependency binary is downloaded correctly for branch ^cdc-release-.*$ and ^br-release-.*$

[pingyu]

OK. prepare_test_binaries will download TiDB release 6.5 and all CDC & BR releases are designed to be compatible with this release.

[purelind]

/lgtm

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2024-01-08 02:37:46.377311505 +0000 UTC m=+238055.961565192: ☑️ agreed by purelind.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

Approval requirements bypassed by manually added approval.

This pull-request has been approved by: purelind

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• jobs/OWNERS
• pipelines/OWNERS [purelind]
• prow-jobs/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment