Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

K8SPXC-1462: Restart PXC pods after monitor user password change #1816

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

s10
Copy link
Contributor

@s10 s10 commented Sep 13, 2024

K8SPXC-1462 Powered by Pull Request Badge


Problem:
monitor user could be used not only in pmm sidecar containers, but in custom mysqld-exporter sidecar container running near pxc. If these custom sidecars are using monitor user, they need a pod restart after password change, same as pmm sidecars.

Cause:
monitor user password update causes pxc pods restart only when pmm is enabled

Solution:
Restart PXC pods without checking if a PMM is enabled.

CHECKLIST

Jira

  • Is the Jira ticket created and referenced properly?
  • Does the Jira ticket have the proper statuses for documentation (Needs Doc) and QA (Needs QA)?
  • Does the Jira ticket link to the proper milestone (Fix Version field)?

Tests

  • Is an E2E test/test case added for the new feature/change?
  • Are unit tests added where appropriate?
  • Are OpenShift compare files changed for E2E tests (compare/*-oc.yml)?

Config/Logging/Testability

  • Are all needed new/changed options added to default YAML files?
  • Did we add proper logging messages for operator actions?
  • Did we ensure compatibility with the previous version or cluster upgrade process?
  • Does the change support oldest and newest supported PXC version?
  • Does the change support oldest and newest supported Kubernetes version?

Copy link
Collaborator

@hors hors left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not like this idea. We should not have unneeded restarts. Maybe we can restart it if custom sidecar is used but up to @egegunes

@egegunes
Copy link
Contributor

I agree with @hors, I don't think it's a good idea to restart PXC pods unless we know for sure that restart is needed.
@s10 how do you use monitor password in custom sidecar? env, envFromSecret or volume? maybe we can add a check for these and decide if restart is needed.

@s10
Copy link
Contributor Author

s10 commented Sep 27, 2024

I use envFromSecret:

    sidecars:
    - name: metrics
      image: prom/mysqld-exporter:v0.15.1
      env:
        - name: MYSQLD_EXPORTER_PASSWORD
          valueFrom:
            secretKeyRef:
              name: pxc-dev-1-secrets
              key: monitor
      ports:
        - name: metrics
          containerPort: 9104
      args:
        - "--mysqld.username=monitor"
        - "--mysqld.address=localhost:3306"
        - "--collect.binlog_size"
 ...

@egegunes
Copy link
Contributor

It might be good to check keys for system usernames in env and envFromSecret of sidecar container and decide if it needs a restart. wdyt @hors @spron-in?

@hors
Copy link
Collaborator

hors commented Sep 30, 2024

It might be good to check keys for system usernames in env and envFromSecret of sidecar container and decide if it needs a restart. wdyt @hors @spron-in?

It is a good idea.

@spron-in
Copy link
Collaborator

I understand the desire to minimize the number of restarts, but also want to be sure that we are not overcomplicating it here.
I would not expect a user to change the password for monitor user too often or to change it when monitoring in some form is not enabled.
So we are going to implement more checks for quite a rare operation and the operation that actually justifies pod restart.

Am I looking at it wrong?

@pull-request-size pull-request-size bot added size/M 30-99 lines and removed size/XS 0-9 lines labels Oct 23, 2024
@s10 s10 requested a review from hors October 24, 2024 08:08
@egegunes
Copy link
Contributor

@spron-in i really don't want to restart PXC pods blindly. we can always say user should restart pods themselves but if we want to do it in automatic way, i believe current changes look good.

@egegunes egegunes added this to the v1.16.0 milestone Nov 20, 2024
@egegunes egegunes assigned inelpandzic and unassigned egegunes Nov 20, 2024
@egegunes
Copy link
Contributor

@inelpandzic please review

@JNKPercona
Copy link
Collaborator

Test name Status
affinity-8-0 passed
auto-tuning-8-0 passed
cross-site-8-0 passed
demand-backup-cloud-8-0 passed
demand-backup-encrypted-with-tls-8-0 passed
demand-backup-8-0 failure
haproxy-5-7 passed
haproxy-8-0 passed
init-deploy-5-7 passed
init-deploy-8-0 failure
limits-8-0 passed
monitoring-2-0-8-0 passed
one-pod-5-7 passed
one-pod-8-0 passed
pitr-8-0 passed
pitr-gap-errors-8-0 passed
proxy-protocol-8-0 passed
proxysql-sidecar-res-limits-8-0 passed
pvc-resize-5-7 passed
pvc-resize-8-0 passed
recreate-8-0 failure
restore-to-encrypted-cluster-8-0 passed
scaling-proxysql-8-0 passed
scaling-8-0 passed
scheduled-backup-5-7 passed
scheduled-backup-8-0 passed
security-context-8-0 passed
smart-update1-8-0 passed
smart-update2-8-0 passed
storage-8-0 passed
tls-issue-cert-manager-ref-8-0 passed
tls-issue-cert-manager-8-0 passed
tls-issue-self-8-0 passed
upgrade-consistency-8-0 passed
upgrade-haproxy-5-7 passed
upgrade-haproxy-8-0 passed
upgrade-proxysql-5-7 passed
upgrade-proxysql-8-0 passed
users-5-7 passed
users-8-0 passed
validation-hook-8-0 passed
We run 41 out of 41

commit: 0a78eaf
image: perconalab/percona-xtradb-cluster-operator:PR-1816-0a78eaf7

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants