refactor(dockerfile): update dockerfile for release 6.5 #469

wuhuizuo · 2024-10-21T04:03:49Z

From for ~6.5.12 we switch the base image to the new one.

Signed-off-by: wuhuizuo [email protected]

This pull request includes various updates and cleanups to the CI scripts and Dockerfiles. The most important changes include the addition of new versions for testing in CI scripts, removal of commented-out experimental code, and updates to Dockerfiles for different products.

CI Script Updates:

Added new versions v6.5.12 and v6.5.11 to the list of versions in several functions (test_get_builder, test_gen_package_artifacts_script, test_gen_package_images_script, test_gen_offline_package_artifacts_script). [1] [2] [3] [4]
Removed commented-out experimental profile code from the test_get_builder and test_gen_package_images_script functions. [1] [2]

Dockerfile Updates:

Removed Dockerfiles for le6.5-linux-amd64-debug/tiem, le6.5-linux-amd64/tiem, le6.5-linux-arm64/dm-monitor-initializer, and le6.5-linux-arm64/tidb-operator. [1] [2] [3] [4]
Updated Dockerfiles for ng-monitoring, pd, tidb-binlog, tidb-dashboard, and tidb-tools to use new base images and added necessary configurations. [1] [2] [3] [4] [5] [6]

From for ~6.5.12 we switch the base image to the new one. Signed-off-by: wuhuizuo <[email protected]>

ti-chi-bot · 2024-10-21T06:35:30Z

I have already done a preliminary review for you, and I hope to help you do a better job.

The pull request is focused on refactoring the Dockerfile for release 6.5. The changes can be categorized into a few parts:

Refactoring of the CI script: The versions of the CI script have been updated to include the versions v6.5.12, v6.5.11, and v6.1.0. Some sections of the script have also been commented out, which may affect the functionalities they were providing.
Deletion of Dockerfiles: Various Dockerfiles for different products like le6.5-linux-amd64-debug/tiem, le6.5-linux-amd64/tiem, le6.5-linux-arm64/dm-monitor-initializer, le6.5-linux-arm64/tidb-operator etc. have been deleted. This could affect the building of these Docker images.
Modification of Dockerfiles: Many Dockerfiles have been modified, mainly to update the base image used. For example, the base image for ng-monitoring-base has been changed from v1.0.1-old to v1.9.2. In addition, some Dockerfiles have been renamed and moved to different directories.
Changes in packages.yaml.tmpl: This file has been updated to reflect the changes made in the Dockerfiles. For example, the Dockerfile URLs have been updated.

Potential problems:

The CI script changes, especially the commented out sections, could impact the functionalities they were providing. It's necessary to ensure that these changes do not affect any dependent services or functionalities.
The deletion of Dockerfiles could affect the building of these Docker images. If there are services or functionalities dependent on these images, it could lead to problems.
The modifications in Dockerfiles, especially the changes in the base image, could lead to unforeseen issues. The new base images may have different configurations or missing packages that could affect the Docker image build or the runtime of the containers.
The changes in packages.yaml.tmpl, especially the updates in Dockerfile URLs, need to be correct and the new URLs should be accessible. Any mistakes here could lead to failures in fetching the Dockerfiles.

Fixing suggestions:

Review the commented out sections of the CI script and ensure they are not affecting any dependent functionalities. If necessary, these sections can be removed or updated.
For the deleted Dockerfiles, ensure that there are no dependent services or functionalities. If there are, alternatives need to be provided.
Thoroughly test the Docker images built using the modified Dockerfiles to ensure they work as expected.
Validate the changes in packages.yaml.tmpl, especially the Dockerfile URLs, to ensure they are correct and accessible.

wuhuizuo · 2024-10-21T06:41:57Z

/hold

wait for decision be made.

wuhuizuo · 2024-11-04T10:18:41Z

/review list the renamed docker files.

ti-chi-bot · 2024-11-04T10:19:14Z

@wuhuizuo:
The renamed Docker files are:

dockerfiles/products/ng-monitoring.Dockerfile -> dockerfiles/products/ng-monitoring/Dockerfile
dockerfiles/products/ng-monitoring/le6.5/Dockerfile -> dockerfiles/products/ng-monitoring/lt6.5.12/Dockerfile
dockerfiles/products/pd.Dockerfile -> dockerfiles/products/pd/Dockerfile
dockerfiles/products/pd/le6.5/Dockerfile -> dockerfiles/products/pd/lt6.5.12/Dockerfile
dockerfiles/products/tidb-binlog.Dockerfile -> dockerfiles/products/tidb-binlog/Dockerfile
dockerfiles/products/tidb-binlog/le6.5/Dockerfile -> dockerfiles/products/tidb-binlog/lt6.5.12/Dockerfile
dockerfiles/products/tidb-dashboard.Dockerfile -> dockerfiles/products/tidb-dashboard/Dockerfile
dockerfiles/products/tidb-dashboard/le6.5/Dockerfile -> dockerfiles/products/tidb-dashboard/lt6.5.12/Dockerfile
dockerfiles/products/tidb-tools.Dockerfile -> dockerfiles/products/tidb-tools/Dockerfile
dockerfiles/products/tidb-tools/le6.5/Dockerfile -> dockerfiles/products/tidb-tools/lt6.5.12/Dockerfile
dockerfiles/products/br.Dockerfile -> dockerfiles/products/tidb/br.Dockerfile
dockerfiles/products/dumpling.Dockerfile -> dockerfiles/products/tidb/dumpling.Dockerfile
dockerfiles/products/tidb/le6.5/br.Dockerfile -> dockerfiles/products/tidb/lt6.5.12/br.Dockerfile
dockerfiles/products/tidb/le6.5/dumpling.Dockerfile -> dockerfiles/products/tidb/lt6.5.12/dumpling.Dockerfile
dockerfiles/products/tidb-lightning.Dockerfile -> dockerfiles/products/tidb/tidb-lightning.Dockerfile

......

Response is trunked for length limits.

In response to this:

/review list the renamed docker files.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

wuhuizuo · 2024-11-04T10:19:16Z

/review list the renamed docker files.

ti-chi-bot · 2024-11-04T10:19:37Z

@wuhuizuo:

Renamed Dockerfiles:

From dockerfiles/products/ng-monitoring.Dockerfile to dockerfiles/products/ng-monitoring/Dockerfile
From dockerfiles/products/pd.Dockerfile to dockerfiles/products/pd/Dockerfile
From dockerfiles/products/tidb-binlog.Dockerfile to dockerfiles/products/tidb-binlog/Dockerfile
From dockerfiles/products/tidb-dashboard.Dockerfile to dockerfiles/products/tidb-dashboard/Dockerfile
From dockerfiles/products/tidb-tools.Dockerfile to dockerfiles/products/tidb-tools/Dockerfile
From dockerfiles/products/br.Dockerfile to dockerfiles/products/tidb/br.Dockerfile
From dockerfiles/products/dumpling.Dockerfile to dockerfiles/products/tidb/dumpling.Dockerfile
From dockerfiles/products/tidb-lightning.Dockerfile to dockerfiles/products/tidb/tidb-lightning.Dockerfile
From dockerfiles/products/tidb.Dockerfile to dockerfiles/products/tidb/tidb.Dockerfile
From dockerfiles/products/tidb-enterprise.Dockerfile to dockerfiles/products/tidb/tidb.enterprise.Dockerfile
From dockerfiles/products/tikv.Dockerfile to dockerfiles/products/tikv/Dockerfile
From dockerfiles/products/tikv-fips.Dockerfile to dockerfiles/products/tikv/fips.Dockerfile
From dockerfiles/products/dm.Dockerfile to dockerfiles/products/tiflow/dm.Dockerfile
From dockerfiles/products/ticdc.Dockerfile to dockerfiles/products/tiflow/ticdc.Dockerfile

In response to this:

/review list the renamed docker files.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

wuhuizuo · 2024-11-04T10:20:35Z

/review list the renamed docker files without the prefix folder "dockerfiles/products".

ti-chi-bot · 2024-11-04T10:21:06Z

@wuhuizuo:

Summary of Proposed Changes:

Refactor Dockerfiles for compatibility with release 6.5.
Added new versions v6.5.12 and v6.5.11 to the list of versions in several functions.
Removed commented-out experimental code in test_get_builder and test_gen_package_images_script.
Removed Dockerfiles for le6.5-linux-amd64-debug/tiem, le6.5-linux-amd64/tiem, le6.5-linux-arm64/dm-monitor-initializer, and le6.5-linux-arm64/tidb-operator.
Updated Dockerfiles for ng-monitoring, pd, tidb-binlog, tidb-dashboard, and tidb-tools to use new base images and added necessary configurations.
Renamed and updated Dockerfiles for several products without prefix folder "dockerfiles/products".

Renamed Dockerfiles:

ng-monitoring.Dockerfile to ng-monitoring/Dockerfile
pd.Dockerfile to pd/Dockerfile
tidb-binlog.Dockerfile to tidb-binlog/Dockerfile
tidb-dashboard.Dockerfile to tidb-dashboard/Dockerfile
tidb-tools.Dockerfile to tidb-tools/Dockerfile
br.Dockerfile to tidb/br.Dockerfile
dumpling.Dockerfile to tidb/dumpling.Dockerfile
tidb-lightning-debug.Dockerfile to tidb/~6.5.12/tidb-lightning.Dockerfile
tidb-release-6.5.Dockerfile to tidb/~6.5.12/tidb.Dockerfile
tidb-enterprise-release-6.5.Dockerfile to tidb/~6.5.12/tidb.enterprise.Dockerfile
tiflash.Dockerfile to tiflash/Dockerfile
dm.Dockerfile to tiflow/dm.Dockerfile
ticdc.Dockerfile to tiflow/ticdc.Dockerfile
tikv.Dockerfile to tikv/Dockerfile

......

Response is trunked for length limits.

In response to this:

/review list the renamed docker files without the prefix folder "dockerfiles/products".

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

ti-chi-bot · 2024-11-04T10:27:07Z

I have already done a preliminary review for you, and I hope to help you do a better job.

The pull request is related to the update of the Dockerfiles and CI scripts for release 6.5. The changes are made to support the new version v6.5.12 and v6.5.11 in the CI scripts, and to refactor the Dockerfiles for a couple of products.

Key Changes:

Added new versions v6.5.12 and v6.5.11 to the list of versions in several functions in CI scripts.
Removed commented-out experimental profile code from the CI scripts.
Removed Dockerfiles for le6.5-linux-amd64-debug/tiem, le6.5-linux-amd64/tiem, le6.5-linux-arm64/dm-monitor-initializer, and le6.5-linux-arm64/tidb-operator.
Updated Dockerfiles for ng-monitoring, pd, tidb-binlog, tidb-dashboard, and tidb-tools to use new base images and added necessary configurations.

Potential Problems:

In the CI scripts, once you're adding new versions to the list, you need to ensure that your testing environment and the CI system itself can support these versions.
Removing Dockerfiles might affect users who are still using these images. They might not be able to build or pull these images anymore. You need to ensure that you've communicated these changes to your users and have provided them with alternatives if necessary.
Updating Dockerfiles to use new base images might introduce unexpected issues if the new base images are not fully compatible with your applications. Rigorous testing is required to ensure there are no compatibility issues.

Fixing Suggestions:

Before adding new versions to the list in the CI scripts, make sure that your testing environment and the CI system itself are ready for these new versions.
After removing Dockerfiles, you need to communicate these changes to your users and provide them with alternatives if necessary.
When updating Dockerfiles to use new base images, make sure to conduct rigorous testing to ensure there are no compatibility issues.

Signed-off-by: wuhuizuo <[email protected]>

ti-chi-bot · 2024-11-04T10:35:55Z

I have already done a preliminary review for you, and I hope to help you do a better job.

This pull request is mainly about updating Dockerfiles and CI scripts for release 6.5.12. Here are the key changes:

CI Script Updates:

The versions v6.5.12 and v6.5.11 are added to the list of versions in several functions.
Commented-out experimental profile code is removed from the test_get_builder and test_gen_package_images_script functions.

Dockerfile Updates:

Dockerfiles for le6.5-linux-amd64-debug/tiem, le6.5-linux-amd64/tiem, le6.5-linux-arm64/dm-monitor-initializer, and le6.5-linux-arm64/tidb-operator are removed.
Dockerfiles for ng-monitoring, pd, tidb-binlog, tidb-dashboard, and tidb-tools are updated to use new base images and added necessary configurations.

Potential Problems:

For the new Dockerfile versions, it is important to ensure that the new base images are compatible with the products and won't introduce unexpected issues.
Removing dockerfiles may affect dependent services, it is important to verify that these Dockerfiles are not being used elsewhere.
The removal of the experimental profile code needs to be validated to ensure that it doesn't affect the other functionalities.

Suggestions:

Verify the new base images compatibility with the products, ideally through automated tests.
Double-check the dependencies of the removed Dockerfiles to ensure no services are affected.
Test the functionality of the CI scripts after removing the experimental profile code to ensure other functionalities are intact.
It would be better to provide more detailed information in the PR description about why these changes were made and what the expected impact is. For example, why Dockerfiles were removed, why the experimental profile code was removed, etc.
If possible, separate the concerns into multiple PRs. One for updating the CI scripts, one for Dockerfile updates, and one for removing Dockerfiles. This makes the changes easier to review and trace back in the future.

wuhuizuo · 2024-11-04T10:41:58Z

/approve

ti-chi-bot · 2024-11-04T10:42:01Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: wuhuizuo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [wuhuizuo]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

…f v6.5.x - hotfix versions for 6.5.7-* - patch versions for ~6.5.12 Signed-off-by: wuhuizuo <[email protected]>

wuhuizuo · 2024-11-04T11:28:10Z

/unhold

ti-chi-bot · 2024-11-04T11:28:18Z

I have already done a preliminary review for you, and I hope to help you do a better job.

The pull request with the title "refactor(dockerfile): update dockerfile for release 6.5" included several updates and cleanups to the CI scripts and Dockerfiles.

Key Changes:

In the CI script, new versions v6.5.12 and v6.5.11 were added to several functions (test_get_builder, test_gen_package_artifacts_script, test_gen_package_images_script, test_gen_offline_package_artifacts_script).
Some commented-out experimental code was removed from the test_get_builder and test_gen_package_images_script functions.
Dockerfiles for le6.5-linux-amd64-debug/tiem, le6.5-linux-amd64/tiem, le6.5-linux-arm64/dm-monitor-initializer, and le6.5-linux-arm64/tidb-operator were removed.
Dockerfiles for ng-monitoring, pd, tidb-binlog, tidb-dashboard, and tidb-tools were updated to use new base images and added necessary configurations.

Potential Problems:

Some Dockerfiles are removed, which may cause problems if any existing services depend on them.
The switch to new base images in Dockerfiles may introduce incompatibilities or unexpected behavior if these images have significant differences from the previous ones.
The removal of the experimental profile code could potentially remove some functionality that was in the process of being tested.
The addition of new versions to the CI scripts could cause issues if these versions have not been thoroughly tested.

Suggestions:

Ensure that no existing services depend on the Dockerfiles that were removed.
Test the Dockerfiles with the new base images thoroughly to ensure compatibility.
If the experimental profile code is still needed in the future, consider creating a separate branch or feature flag for it.
Conduct thorough testing for the newly added versions in the CI scripts to ensure they work as expected.

Signed-off-by: wuhuizuo <[email protected]>

ti-chi-bot · 2024-11-04T11:34:50Z

I have already done a preliminary review for you, and I hope to help you do a better job.

The pull request essentially does the following:

Updates the CI scripts:
- Adds new versions v6.5.12 and v6.5.11 to the list of versions in several functions.
- Removes commented-out experimental profile code from the test_get_builder and test_gen_package_images_script functions.
Updates the Dockerfiles:
- Removes Dockerfiles for le6.5-linux-amd64-debug/tiem, le6.5-linux-amd64/tiem, le6.5-linux-arm64/dm-monitor-initializer, and le6.5-linux-arm64/tidb-operator.
- Updates Dockerfiles for ng-monitoring, pd, tidb-binlog, tidb-dashboard, and tidb-tools to use new base images and added necessary configurations.
Updates the package.yaml.tmpl file:
- Adds a new template "image_dockerfile_folder" to handle the versioning of dockerfiles.
- Updates the URLs for several Dockerfiles based on the new versioning logic.

Potential issues and suggestions:

Versioning handling changes: The changes introduce new versions and change how versions are handled in Dockerfiles. It's crucial to ensure that all systems or scripts that depend on the previous versioning logic are updated to handle the new logic. This includes any CI/CD pipelines, build scripts, or deployment scripts.
Dockerfile deletions: The PR deletes certain Dockerfiles. Make sure that these Dockerfiles are not required anymore. If there are any systems or scripts that depend on these Dockerfiles, they should be updated or deprecated.
Code removed: The PR removes certain commented code. Although this is generally a good practice (as commented code can make the codebase messy), it's essential to ensure that this code is not required in the future. If there's a chance that this code might be used, consider keeping it with clear comments or moving it to a separate branch or documentation.
Testing: Given the nature of the changes (mainly versioning and Dockerfiles), it's crucial to thoroughly test this PR. This includes testing the updated CI scripts, building the Docker images using the updated Dockerfiles and ensuring all services work as expected. Also, test the package generation process with the updated package.yaml.tmpl file.
Documentation: Update relevant documentation to reflect these changes. This includes updating the versioning logic and the process of adding new versions.
Communication: Communicate these changes with the team, especially those who interact with the Dockerfiles, CI scripts, or the package generation process.

Ref: PingCAP-QE/artifacts#469 Signed-off-by: wuhuizuo <[email protected]> Signed-off-by: wuhuizuo <[email protected]>

ti-chi-bot bot added the do-not-merge/work-in-progress label Oct 21, 2024

ti-chi-bot bot requested a review from purelind October 21, 2024 04:03

ti-chi-bot bot added the size/L label Oct 21, 2024

wuhuizuo force-pushed the feature/update-dockerfile-for-release-6.5 branch from 1bdabe3 to af08bd2 Compare October 21, 2024 06:20

ti-chi-bot bot added size/XL and removed size/L labels Oct 21, 2024

wuhuizuo marked this pull request as ready for review October 21, 2024 06:20

ti-chi-bot bot removed the do-not-merge/work-in-progress label Oct 21, 2024

refactor(dockerfile): update dockerfile for release 6.5

b640808

From for ~6.5.12 we switch the base image to the new one. Signed-off-by: wuhuizuo <[email protected]>

wuhuizuo force-pushed the feature/update-dockerfile-for-release-6.5 branch from af08bd2 to b640808 Compare October 21, 2024 06:34

ti-chi-bot bot added the do-not-merge/hold label Oct 21, 2024

wuhuizuo marked this pull request as draft October 21, 2024 06:42

ti-chi-bot bot added the do-not-merge/work-in-progress label Oct 21, 2024

wuhuizuo marked this pull request as ready for review November 4, 2024 10:19

ti-chi-bot bot removed the do-not-merge/work-in-progress label Nov 4, 2024

fix(dockerfiles/products): add symlinks for config migration

3e440ad

Signed-off-by: wuhuizuo <[email protected]>

wuhuizuo force-pushed the feature/update-dockerfile-for-release-6.5 branch from 39f36d4 to 3e440ad Compare November 4, 2024 10:35

ti-chi-bot bot added the approved label Nov 4, 2024

feat(packages): update config for feature hotfix and patch releases o…

d6a6371

…f v6.5.x - hotfix versions for 6.5.7-* - patch versions for ~6.5.12 Signed-off-by: wuhuizuo <[email protected]>

ti-chi-bot bot removed the do-not-merge/hold label Nov 4, 2024

ci: update ci script

a9b81fb

Signed-off-by: wuhuizuo <[email protected]>

ti-chi-bot bot merged commit 4e0b35c into main Nov 4, 2024
3 checks passed

ti-chi-bot bot deleted the feature/update-dockerfile-for-release-6.5 branch November 4, 2024 11:46

wuhuizuo mentioned this pull request Nov 4, 2024

fix(cd): disable stale jobs PingCAP-QE/ci#3202

Merged

ti-chi-bot bot pushed a commit to PingCAP-QE/ci that referenced this pull request Nov 11, 2024

fix(cd): disable stale jobs (#3202)

33082b7

Ref: PingCAP-QE/artifacts#469 Signed-off-by: wuhuizuo <[email protected]> Signed-off-by: wuhuizuo <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor(dockerfile): update dockerfile for release 6.5 #469

refactor(dockerfile): update dockerfile for release 6.5 #469

wuhuizuo commented Oct 21, 2024 •

edited

Loading

ti-chi-bot bot commented Oct 21, 2024

wuhuizuo commented Oct 21, 2024

wuhuizuo commented Nov 4, 2024

ti-chi-bot bot commented Nov 4, 2024

wuhuizuo commented Nov 4, 2024

ti-chi-bot bot commented Nov 4, 2024

wuhuizuo commented Nov 4, 2024

ti-chi-bot bot commented Nov 4, 2024

ti-chi-bot bot commented Nov 4, 2024

ti-chi-bot bot commented Nov 4, 2024

wuhuizuo commented Nov 4, 2024

ti-chi-bot bot commented Nov 4, 2024

wuhuizuo commented Nov 4, 2024

ti-chi-bot bot commented Nov 4, 2024

ti-chi-bot bot commented Nov 4, 2024

refactor(dockerfile): update dockerfile for release 6.5 #469

refactor(dockerfile): update dockerfile for release 6.5 #469

Conversation

wuhuizuo commented Oct 21, 2024 • edited Loading

CI Script Updates:

Dockerfile Updates:

ti-chi-bot bot commented Oct 21, 2024

wuhuizuo commented Oct 21, 2024

wuhuizuo commented Nov 4, 2024

ti-chi-bot bot commented Nov 4, 2024

wuhuizuo commented Nov 4, 2024

ti-chi-bot bot commented Nov 4, 2024

Renamed Dockerfiles:

wuhuizuo commented Nov 4, 2024

ti-chi-bot bot commented Nov 4, 2024

Summary of Proposed Changes:

Renamed Dockerfiles:

ti-chi-bot bot commented Nov 4, 2024

ti-chi-bot bot commented Nov 4, 2024

wuhuizuo commented Nov 4, 2024

ti-chi-bot bot commented Nov 4, 2024

wuhuizuo commented Nov 4, 2024

ti-chi-bot bot commented Nov 4, 2024

ti-chi-bot bot commented Nov 4, 2024

wuhuizuo commented Oct 21, 2024 •

edited

Loading