Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

The error log of the secondary node of the 3 replica mysql cluster is displayed as follows: [Repl] [GTID INFO] The transaction owned GTID gno start 1, end #8858

Open
wallyxjh opened this issue Jan 23, 2025 · 12 comments
Open

The error log of the secondary node of the 3 replica mysql cluster is displayed as follows: [Repl] [GTID INFO] The transaction owned GTID gno start 1, end #8858

wallyxjh opened this issue Jan 23, 2025 · 12 comments
Assignees
@demian0110 @shanshanying @sophon-zt
Labels
kind/bug Something isn't working

Comments

@wallyxjh
Copy link

error-slave.log
@wallyxjh wallyxjh added the kind/bug Something isn't working label Jan 23, 2025
@wallyxjh
Copy link
Author

Image

@wallyxjh
Copy link
Author

I deleted /data/mysql/data/.meta_initialized file in the 3 pods and restarted the database

@shanshanying
Copy link
Contributor

hi @wallyxjh

  1. pls state the version of kb and apecloud-mysql addons you are working with.
  2. and you may use kbcli report cluster <clusterName> --with-logs to pack more info of the cluster and attached it with the issue. so we can have more context.

@shanshanying
Copy link
Contributor

@demian0110 PTAL

@wallyxjh
Copy link
Author

1.kb version 0.8.2 and ac-mysql-8.0.30
2. report logs is here

report-cluster-liebianbao-nklgunml-mysql-2025-01-23-08-17-11.zip

@demian0110
Copy link
Contributor

demian0110 commented Jan 23, 2025 •
edited
Loading

This error has no significant impact. You can execute select * from information_schema.wesql_cluster_global on the leader node to check the status of all nodes in the cluster

@wallyxjh
Copy link
Author

Before the database status was abnormal, I changed the database parameters, but opsrequest was always pending, I deleted opsrequest, and then the database status could be updated as follows: kbcli cluster configure ${name} --set=log_error=/data/mysql/log/mysqld-error.log,slow_query_log=1,slow_query_log_file=/data/mysql/log/slow-query.log,lon g_query_time=1 -n ns-xxx

@wallyxjh
Copy link
Author

Image

@wallyxjh
Copy link
Author

wallyxjh commented Jan 24, 2025 •
edited by shanshanying
Loading

  • kubectl get configuration -n ns-vh09xixu liebianbao-nklgunml-mysql-mysql -oyaml
apiVersion: apps.kubeblocks.io/v1alpha1
kind: Configuration
metadata:
  creationTimestamp: "2024-08-15T16:09:36Z"
  generation: 2
  name: liebianbao-nklgunml-mysql-mysql
  namespace: ns-vh09xixu
  ownerReferences:
  - apiVersion: apps.kubeblocks.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: Cluster
    name: liebianbao-nklgunml-mysql
    uid: c86909ee-a45b-47d4-b92d-38acd516c63b
  resourceVersion: "1458050476"
  uid: c498ea57-4b5b-403e-9cd5-d57cc84fdff9
spec:
  clusterRef: liebianbao-nklgunml-mysql
  componentName: mysql
  configItemDetails:
  - configFileParams:
      my.cnf:
        parameters:
          log_error: /data/mysql/log/mysqld-error.log
          long_query_time: "1"
          slow_query_log: "1"
          slow_query_log_file: /data/mysql/log/slow-query.log
    configSpec:
      constraintRef: mysql8.0-config-constraints
      name: mysql-consensusset-config
      namespace: kb-system
      templateRef: mysql8.0-config-template
      volumeName: mysql-config
    name: mysql-consensusset-config
  - configSpec:
      defaultMode: 292
      name: agamotto-configuration
      namespace: kb-system
      templateRef: apecloud-mysql8-agamotto-configuration
      volumeName: agamotto-configuration
    name: agamotto-configuration
  - configSpec:
      constraintRef: mysql-scale-vttablet-config-constraints
      name: vttablet-config
      namespace: kb-system
      templateRef: vttablet-config-template
      volumeName: mysql-scale-config
    name: vttablet-config
status:
  configurationStatus:
  - lastDoneRevision: "1"
    name: mysql-consensusset-config
    phase: Upgrading
    reconcileDetail:
      currentRevision: "2"
      execResult: Retry
      expectedCount: 3
      policy: simple
      succeedCount: -1
    updateRevision: "2"
  - lastDoneRevision: "1"
    name: agamotto-configuration
    phase: Finished
    reconcileDetail:
      currentRevision: "1"
      errMessage: the configuration file has not been modified, skip reconfigure
      execResult: ""
      expectedCount: -1
      policy: ""
      succeedCount: -1
    updateRevision: "1"
  - lastDoneRevision: "1"
    name: vttablet-config
    phase: Finished
    reconcileDetail:
      currentRevision: "1"
      errMessage: the configuration file has not been modified, skip reconfigure
      execResult: ""
      expectedCount: -1
      policy: ""
      succeedCount: -1
    updateRevision: "1"

@wallyxjh
Copy link
Author

  • kubectl logs -n ns-vh09xixu liebianbao-nklgunml-mysql-mysql-1
    Defaulted container "mysql" out of: mysql, metrics, vttablet, lorry, config-manager
    2025-01-23 06:08:11+00:00 [Note] [MySQL]: Entrypoint script for MySQL Server started.
    2025-01-23 06:08:11+00:00 [Note] [MySQL]: KB_EMBEDDED_WESQL=1
    2025-01-23 06:08:11+00:00 [Note] [Kubeblocks]:
    2025-01-23 06:08:11+00:00 [Note] [Kubeblocks]: KB_MYSQL_VOLUME_DIR=
    2025-01-23 06:08:11+00:00 [Note] [Kubeblocks]: KB_MYSQL_CLUSTER_UID=$(KB_CLUSTER_UID)
    2025-01-23 06:08:11+00:00 [Note] [Kubeblocks]: KB_MYSQL_CONF_FILE=
    2025-01-23 06:08:11+00:00 [Note] [Kubeblocks]: KB_MYSQL_N=3
    2025-01-23 06:08:11+00:00 [Note] [Kubeblocks]: KB_MYSQL_CLUSTER_MEMBERS=
    2025-01-23 06:08:11+00:00 [Note] [Kubeblocks]: KB_MYSQL_0_HOSTNAME=liebianbao-nklgunml-mysql-mysql-0.liebianbao-nklgunml-mysql-mysql-headless
    2025-01-23 06:08:11+00:00 [Note] [Kubeblocks]: KB_MYSQL_1_HOSTNAME=liebianbao-nklgunml-mysql-mysql-1.liebianbao-nklgunml-mysql-mysql-headless
    2025-01-23 06:08:11+00:00 [Note] [Kubeblocks]: KB_MYSQL_2_HOSTNAME=liebianbao-nklgunml-mysql-mysql-2.liebianbao-nklgunml-mysql-mysql-headless
    2025-01-23 06:08:11+00:00 [Note] [Kubeblocks]: KB_MYSQL_CLUSTER_MEMBER_INDEX=
    2025-01-23 06:08:11+00:00 [Note] [Kubeblocks]: KB_MYSQL_CLUSTER_MEMBER_HOST=
    2025-01-23 06:08:11+00:00 [Note] [Kubeblocks]: KB_POD_NAME=liebianbao-nklgunml-mysql-mysql-1
    2025-01-23 06:08:11+00:00 [Note] [Kubeblocks]: KB_MYSQL_CLUSTER_LEADER_HOST=
    2025-01-23 06:08:11+00:00 [Note] [Kubeblocks]: KB_MYSQL_LEADER=
    2025-01-23 06:08:11+00:00 [Note] [Kubeblocks]: KB_MYSQL_CLUSTER_UID_PATH=/data/mysql/.kb_cluster_uid
    2025-01-23 06:08:11+00:00 [Note] [Apecloud MySQL]:
    2025-01-23 06:08:11+00:00 [Note] [Apecloud MySQL]: Welcome to the ApeCloud MySQL Cluster Container
    2025-01-23 06:08:11+00:00 [Note] [Apecloud MySQL]: MYSQL_CLUSTER_MEMBERS=liebianbao-nklgunml-mysql-mysql-0.liebianbao-nklgunml-mysql-mysql-headless:13306;liebianbao-nklgunml-mysql-mysql-1.liebianbao-nklgunml-mysql-mysql-headless:13306;liebianbao-nklgunml-mysql-mysql-2.liebianbao-nklgunml-mysql-mysql-headless:13306
    2025-01-23 06:08:11+00:00 [Note] [Apecloud MySQL]: MYSQL_CLUSTER_MEMBER_INDEX=1
    2025-01-23 06:08:11+00:00 [Note] [Apecloud MySQL]: MYSQL_CLUSTER_MEMBER_HOST=liebianbao-nklgunml-mysql-mysql-1.liebianbao-nklgunml-mysql-mysql-headless
    2025-01-23 06:08:11+00:00 [Note] [Apecloud MySQL]: MYSQL_CLUSTER_LEADER_HOST=
    2025-01-23 06:08:11+00:00 [Note] [Apecloud MySQL]: MYSQL_DATA_DIR=/data/mysql/data
    2025-01-23 06:08:11+00:00 [Note] [Apecloud MySQL]: MYSQL_LOG_DIR=/data/mysql/log
    2025-01-23 06:08:11+00:00 [Note] [Apecloud MySQL]: MYSQL_EXTRA_CONF_FILE=/opt/mysql/my.cnf
    2025-01-23 06:08:11+00:00 [Note] [Apecloud MySQL]: MYSQL_PORT=3306(default)
    2025-01-23 06:08:11+00:00 [Note] [Apecloud MySQL]: MYSQL_CONSENSUS_PORT=13306
    2025-01-23 06:08:11+00:00 [Note] [Apecloud MySQL]:
    2025-01-23 06:08:11+00:00 [Note] [Apecloud MySQL]: MYSQL_CLUSTER_INFO=liebianbao-nklgunml-mysql-mysql-0.liebianbao-nklgunml-mysql-mysql-headless:13306;liebianbao-nklgunml-mysql-mysql-1.liebianbao-nklgunml-mysql-mysql-headless:13306;liebianbao-nklgunml-mysql-mysql-2.liebianbao-nklgunml-mysql-mysql-headless:13306@2
    2025-01-23 06:08:11+00:00 [Note] [Apecloud MySQL]:
    2025-01-23 06:08:11+00:00 [Note] [Apecloud MySQL]: Injecting configuration /opt/mysql/my.cnf to /etc/my.cnf
    2025-01-23 06:08:11+00:00 [Note] [Apecloud MySQL]: Injecting configuration /etc/mysql/wesql_cluster.cnf to /etc/my.cnf
    2025-01-23 06:08:12+00:00 [Note] [Apecloud MySQL]: DATADIR=/data/mysql/data/
    2025-01-23 06:08:12+00:00 [Note] [Apecloud MySQL]: PID_FILE=/var/run/mysqld/mysqld.pid
    2025-01-23 06:08:12+00:00 [Note] [Apecloud MySQL]: SOCKET=/var/run/mysqld/mysqld.sock
    2025-01-23 06:08:12+00:00 [Note] [Apecloud MySQL]: Generating post logrotate shell script: /apecloud/mysql/scripts//post_logroate.sh
    2025-01-23 06:08:12+00:00 [Note] [Apecloud MySQL]: error_log=/data/mysql/log/mysqld-error.log
    2025-01-23 06:08:12+00:00 [Note] [Apecloud MySQL]: slow_log=/data/mysql/log/slow-query.log
    2025-01-23 06:08:12+00:00 [Note] [Apecloud MySQL]: general_log=/data/mysql/data/liebianbao-nklgunml-mysql-mysql-1.log
    2025-01-23 06:08:12+00:00 [Note] [Apecloud MySQL]: Switching to dedicated user 'mysql'
    2025-01-23 06:08:12+00:00 [Note] [MySQL]: Entrypoint script for MySQL Server started.
    2025-01-23 06:08:12+00:00 [Note] [MySQL]: KB_EMBEDDED_WESQL=1
    2025-01-23 06:08:12+00:00 [Note] [Apecloud MySQL]: DATADIR=/data/mysql/data/
    2025-01-23 06:08:12+00:00 [Note] [Apecloud MySQL]: PID_FILE=/var/run/mysqld/mysqld.pid
    2025-01-23 06:08:12+00:00 [Note] [Apecloud MySQL]: SOCKET=/var/run/mysqld/mysqld.sock
    2025-01-23 06:08:12+00:00 [Note] [Apecloud MySQL]: JEMALLOC_ENABLED=1
    2025-01-23 06:08:13+00:00 [Note] [Apecloud MySQL]: Using jemalloc: LD_PRELOAD=/usr/local/lib/libjemalloc.so.2
    2025-01-23 06:08:13+00:00 [Note] [Apecloud MySQL]: Using persisted data
    2025-01-23 06:08:13+00:00 [Note] [Apecloud MySQL]: Initializing consensus meta: 'mysqld --cluster-info=liebianbao-nklgunml-mysql-mysql-0.liebianbao-nklgunml-mysql-mysql-headless:13306;liebianbao-nklgunml-mysql-mysql-1.liebianbao-nklgunml-mysql-mysql-headless:13306;liebianbao-nklgunml-mysql-mysql-2.liebianbao-nklgunml-mysql-mysql-headless:13306@2 --cluster-id=1 --cluster-start-index=1 --cluster-force-change-meta=ON --upgrade-wesql'
    '/var/lib/mysql/mysql.sock' -> '/var/run/mysqld/mysqld.sock'
    2025-01-23 06:08:19+00:00 [Note] [Apecloud MySQL]: Starting WeSQL server: 'mysqld'

@wallyxjh
Copy link
Author 

kubectl logs -n ns-vh09xixu liebianbao-nklgunml-mysql-mysql-1
Defaulted container "mysql" out of: mysql, metrics, vttablet, lorry, config-manager
2025-01-23 06:08:11+00:00 [Note] [MySQL]: Entrypoint script for MySQL Server started.
2025-01-23 06:08:11+00:00 [Note] [MySQL]: KB_EMBEDDED_WESQL=1
2025-01-23 06:08:11+00:00 [Note] [Kubeblocks]:
2025-01-23 06:08:11+00:00 [Note] [Kubeblocks]: KB_MYSQL_VOLUME_DIR=
2025-01-23 06:08:11+00:00 [Note] [Kubeblocks]: KB_MYSQL_CLUSTER_UID=$(KB_CLUSTER_UID)
2025-01-23 06:08:11+00:00 [Note] [Kubeblocks]: KB_MYSQL_CONF_FILE=
2025-01-23 06:08:11+00:00 [Note] [Kubeblocks]: KB_MYSQL_N=3
2025-01-23 06:08:11+00:00 [Note] [Kubeblocks]: KB_MYSQL_CLUSTER_MEMBERS=
2025-01-23 06:08:11+00:00 [Note] [Kubeblocks]: KB_MYSQL_0_HOSTNAME=liebianbao-nklgunml-mysql-mysql-0.liebianbao-nklgunml-mysql-mysql-headless
2025-01-23 06:08:11+00:00 [Note] [Kubeblocks]: KB_MYSQL_1_HOSTNAME=liebianbao-nklgunml-mysql-mysql-1.liebianbao-nklgunml-mysql-mysql-headless
2025-01-23 06:08:11+00:00 [Note] [Kubeblocks]: KB_MYSQL_2_HOSTNAME=liebianbao-nklgunml-mysql-mysql-2.liebianbao-nklgunml-mysql-mysql-headless
2025-01-23 06:08:11+00:00 [Note] [Kubeblocks]: KB_MYSQL_CLUSTER_MEMBER_INDEX=
2025-01-23 06:08:11+00:00 [Note] [Kubeblocks]: KB_MYSQL_CLUSTER_MEMBER_HOST=
2025-01-23 06:08:11+00:00 [Note] [Kubeblocks]: KB_POD_NAME=liebianbao-nklgunml-mysql-mysql-1
2025-01-23 06:08:11+00:00 [Note] [Kubeblocks]: KB_MYSQL_CLUSTER_LEADER_HOST=
2025-01-23 06:08:11+00:00 [Note] [Kubeblocks]: KB_MYSQL_LEADER=
2025-01-23 06:08:11+00:00 [Note] [Kubeblocks]: KB_MYSQL_CLUSTER_UID_PATH=/data/mysql/.kb_cluster_uid
2025-01-23 06:08:11+00:00 [Note] [Apecloud MySQL]:
2025-01-23 06:08:11+00:00 [Note] [Apecloud MySQL]: Welcome to the ApeCloud MySQL Cluster Container
2025-01-23 06:08:11+00:00 [Note] [Apecloud MySQL]: MYSQL_CLUSTER_MEMBERS=liebianbao-nklgunml-mysql-mysql-0.liebianbao-nklgunml-mysql-mysql-headless:13306;liebianbao-nklgunml-mysql-mysql-1.liebianbao-nklgunml-mysql-mysql-headless:13306;liebianbao-nklgunml-mysql-mysql-2.liebianbao-nklgunml-mysql-mysql-headless:13306
2025-01-23 06:08:11+00:00 [Note] [Apecloud MySQL]: MYSQL_CLUSTER_MEMBER_INDEX=1
2025-01-23 06:08:11+00:00 [Note] [Apecloud MySQL]: MYSQL_CLUSTER_MEMBER_HOST=liebianbao-nklgunml-mysql-mysql-1.liebianbao-nklgunml-mysql-mysql-headless
2025-01-23 06:08:11+00:00 [Note] [Apecloud MySQL]: MYSQL_CLUSTER_LEADER_HOST=
2025-01-23 06:08:11+00:00 [Note] [Apecloud MySQL]: MYSQL_DATA_DIR=/data/mysql/data
2025-01-23 06:08:11+00:00 [Note] [Apecloud MySQL]: MYSQL_LOG_DIR=/data/mysql/log
2025-01-23 06:08:11+00:00 [Note] [Apecloud MySQL]: MYSQL_EXTRA_CONF_FILE=/opt/mysql/my.cnf
2025-01-23 06:08:11+00:00 [Note] [Apecloud MySQL]: MYSQL_PORT=3306(default)
2025-01-23 06:08:11+00:00 [Note] [Apecloud MySQL]: MYSQL_CONSENSUS_PORT=13306
2025-01-23 06:08:11+00:00 [Note] [Apecloud MySQL]:
2025-01-23 06:08:11+00:00 [Note] [Apecloud MySQL]: MYSQL_CLUSTER_INFO=liebianbao-nklgunml-mysql-mysql-0.liebianbao-nklgunml-mysql-mysql-headless:13306;liebianbao-nklgunml-mysql-mysql-1.liebianbao-nklgunml-mysql-mysql-headless:13306;liebianbao-nklgunml-mysql-mysql-2.liebianbao-nklgunml-mysql-mysql-headless:13306@2
2025-01-23 06:08:11+00:00 [Note] [Apecloud MySQL]:
2025-01-23 06:08:11+00:00 [Note] [Apecloud MySQL]: Injecting configuration /opt/mysql/my.cnf to /etc/my.cnf
2025-01-23 06:08:11+00:00 [Note] [Apecloud MySQL]: Injecting configuration /etc/mysql/wesql_cluster.cnf to /etc/my.cnf
2025-01-23 06:08:12+00:00 [Note] [Apecloud MySQL]: DATADIR=/data/mysql/data/
2025-01-23 06:08:12+00:00 [Note] [Apecloud MySQL]: PID_FILE=/var/run/mysqld/mysqld.pid
2025-01-23 06:08:12+00:00 [Note] [Apecloud MySQL]: SOCKET=/var/run/mysqld/mysqld.sock
2025-01-23 06:08:12+00:00 [Note] [Apecloud MySQL]: Generating post logrotate shell script: /apecloud/mysql/scripts//post_logroate.sh
2025-01-23 06:08:12+00:00 [Note] [Apecloud MySQL]: error_log=/data/mysql/log/mysqld-error.log
2025-01-23 06:08:12+00:00 [Note] [Apecloud MySQL]: slow_log=/data/mysql/log/slow-query.log
2025-01-23 06:08:12+00:00 [Note] [Apecloud MySQL]: general_log=/data/mysql/data/liebianbao-nklgunml-mysql-mysql-1.log
2025-01-23 06:08:12+00:00 [Note] [Apecloud MySQL]: Switching to dedicated user 'mysql'
2025-01-23 06:08:12+00:00 [Note] [MySQL]: Entrypoint script for MySQL Server started.
2025-01-23 06:08:12+00:00 [Note] [MySQL]: KB_EMBEDDED_WESQL=1
2025-01-23 06:08:12+00:00 [Note] [Apecloud MySQL]: DATADIR=/data/mysql/data/
2025-01-23 06:08:12+00:00 [Note] [Apecloud MySQL]: PID_FILE=/var/run/mysqld/mysqld.pid
2025-01-23 06:08:12+00:00 [Note] [Apecloud MySQL]: SOCKET=/var/run/mysqld/mysqld.sock
2025-01-23 06:08:12+00:00 [Note] [Apecloud MySQL]: JEMALLOC_ENABLED=1
2025-01-23 06:08:13+00:00 [Note] [Apecloud MySQL]: Using jemalloc: LD_PRELOAD=/usr/local/lib/libjemalloc.so.2
2025-01-23 06:08:13+00:00 [Note] [Apecloud MySQL]: Using persisted data
2025-01-23 06:08:13+00:00 [Note] [Apecloud MySQL]: Initializing consensus meta: 'mysqld --cluster-info=liebianbao-nklgunml-mysql-mysql-0.liebianbao-nklgunml-mysql-mysql-headless:13306;liebianbao-nklgunml-mysql-mysql-1.liebianbao-nklgunml-mysql-mysql-headless:13306;liebianbao-nklgunml-mysql-mysql-2.liebianbao-nklgunml-mysql-mysql-headless:13306@2 --cluster-id=1 --cluster-start-index=1 --cluster-force-change-meta=ON --upgrade-wesql'
'/var/lib/mysql/mysql.sock' -> '/var/run/mysqld/mysqld.sock'
2025-01-23 06:08:19+00:00 [Note] [Apecloud MySQL]: Starting WeSQL server: 'mysqld'

@shanshanying
Copy link
Contributor

@sophon-zt PTAL

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@demian0110 demian0110

@shanshanying shanshanying

@sophon-zt sophon-zt

Labels
kind/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@demian0110 @wallyxjh @shanshanying @sophon-zt