Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

restart manager when middleware change #732

Closed
wants to merge 1 commit into from

Conversation

ldpliu
Copy link
Contributor

@ldpliu ldpliu commented Dec 1, 2023

Copy link

openshift-ci bot commented Dec 1, 2023

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: ldpliu
Once this PR has been reviewed and has the lgtm label, please assign nirrozenbaum for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ldpliu
Copy link
Contributor Author

ldpliu commented Dec 4, 2023

/cc @yanmxa @clyang82

@openshift-ci openshift-ci bot requested review from clyang82 and yanmxa December 4, 2023 00:31
var pgConnectionCache *postgres.PostgresConnection
var kafkaConnectionCache *kafka.KafkaConnection

const managerDeploymentName = "multicluster-global-hub-manager"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggest to move it into constants.go file

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@@ -78,6 +89,12 @@ func (r *MulticlusterGlobalHubReconciler) reconcileManager(ctx context.Context,
replicas = 2
}

if isMiddleWareChanged(r.MiddlewareConfig) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

middleware is a word. let us make W as lower case

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@@ -151,6 +168,64 @@ func (r *MulticlusterGlobalHubReconciler) reconcileManager(ctx context.Context,
return nil
}

func restartManagerPod(ctx context.Context, kubeClient kubernetes.Interface) error {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

restart the pod will lose the logs. can we just restart the kafkaConsumer and kafkaProducer? I remembered gorm can handle it automatically @yanmxa is that true?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As @yanmxa append here, I think the gorm can not do it now.
Please configm @yanmxa

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@clyang82 @ldpliu Sorry for not replying in time.

  • The gorm can't reload/reconnect the credential automatically for now.
  • I had several ideas for propagating the middleware credentials, and before we choose the restarting manager, There are the following options we should consider(both of them don't need to restart the pod/manager):
    1. The operator calls the non-k8s API of the manager to update their credential. Then the manager rebuilds the connection to these middlewares.
      As far as I know, a lot of services provide a refreshToken API dedicated to updating the expired credentials. So I think we can prioritize this approach.
    2. The global hub manager watches the credentials of these middlewares, and rebuild the connections in the controller.

@ldpliu ldpliu force-pushed the fix-secret-change branch from 254cadd to b17fd11 Compare December 7, 2023 01:08
@ldpliu
Copy link
Contributor Author

ldpliu commented Dec 7, 2023

/test sonarcloud

Copy link

sonarqubecloud bot commented Dec 7, 2023

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

84.2% 84.2% Coverage
0.0% 0.0% Duplication

@ldpliu
Copy link
Contributor Author

ldpliu commented Dec 7, 2023

/test test-e2e

Copy link

openshift-ci bot commented Dec 7, 2023

@ldpliu: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/test-e2e b17fd11 link true /test test-e2e

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@openshift-merge-robot
Copy link
Contributor

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@ldpliu ldpliu closed this Jan 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants