Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Charm stuck in Initialising mysqld #310

Closed
gboutry opened this issue Sep 25, 2023 · 11 comments
Closed

Charm stuck in Initialising mysqld #310

gboutry opened this issue Sep 25, 2023 · 11 comments
Labels
bug Something isn't working

Comments

@gboutry
Copy link

gboutry commented Sep 25, 2023

Steps to reproduce

  1. Deploy sunbeam

Expected behavior

Charm should become ready

Actual behavior

Charm has an unhandled exception and never becomes ready

Versions

Operating system: Ubuntu 22.04.3 LTS

Juju CLI: 3.2.3-genericlinux-amd64

Juju agent: 3.2.3

Charm revision: 99 (8.0/candidate)

microk8s: MicroK8s v1.26.8 revision 5883

Log output

2023-09-25T11:48:38.238Z [container-agent] 2023-09-25 11:48:38 INFO juju.worker.uniter.operation runhook.go:186 ran "start" hook (via hook dispatching script: dispatch)
2023-09-25T11:48:39.462Z [container-agent] 2023-09-25 11:48:39 INFO juju.worker.uniter.operation runhook.go:186 ran "database-relation-changed" hook (via hook dispatching script: dispatch)
2023-09-25T11:48:40.972Z [container-agent] 2023-09-25 11:48:40 INFO juju.worker.uniter.operation runhook.go:186 ran "database-relation-joined" hook (via hook dispatching script: dispatch)
2023-09-25T11:48:42.028Z [container-agent] 2023-09-25 11:48:42 INFO juju.worker.uniter.operation runhook.go:186 ran "database-relation-changed" hook (via hook dispatching script: dispatch)
2023-09-25T11:48:43.152Z [container-agent] 2023-09-25 11:48:43 INFO juju.worker.uniter.operation runhook.go:186 ran "database-relation-changed" hook (via hook dispatching script: dispatch)
2023-09-25T11:48:44.168Z [container-agent] 2023-09-25 11:48:44 INFO juju.worker.uniter.operation runhook.go:186 ran "database-relation-changed" hook (via hook dispatching script: dispatch)
2023-09-25T11:48:45.239Z [container-agent] 2023-09-25 11:48:45 INFO juju.worker.uniter.operation runhook.go:186 ran "database-relation-joined" hook (via hook dispatching script: dispatch)
2023-09-25T11:48:46.475Z [container-agent] 2023-09-25 11:48:46 INFO juju.worker.uniter.operation runhook.go:186 ran "upgrade-relation-changed" hook (via hook dispatching script: dispatch)
2023-09-25T11:48:47.577Z [container-agent] 2023-09-25 11:48:47 INFO juju.worker.uniter.operation runhook.go:186 ran "database-peers-relation-changed" hook (via hook dispatching script: dispatch)
2023-09-25T11:48:48.682Z [container-agent] 2023-09-25 11:48:48 INFO juju.worker.uniter.operation runhook.go:186 ran "database-relation-changed" hook (via hook dispatching script: dispatch)
2023-09-25T11:48:49.741Z [container-agent] 2023-09-25 11:48:49 INFO juju.worker.uniter.operation runhook.go:186 ran "database-relation-joined" hook (via hook dispatching script: dispatch)
2023-09-25T11:48:50.720Z [container-agent] 2023-09-25 11:48:50 INFO juju.worker.uniter.operation runhook.go:186 ran "database-relation-changed" hook (via hook dispatching script: dispatch)
2023-09-25T11:48:52.625Z [container-agent] 2023-09-25 11:48:52 INFO juju.worker.uniter.operation runhook.go:186 ran "database-relation-joined" hook (via hook dispatching script: dispatch)
2023-09-25T11:48:53.936Z [container-agent] 2023-09-25 11:48:53 INFO juju.worker.uniter.operation runhook.go:186 ran "database-relation-changed" hook (via hook dispatching script: dispatch)
2023-09-25T11:48:55.189Z [container-agent] 2023-09-25 11:48:55 INFO juju.worker.uniter.operation runhook.go:186 ran "database-relation-changed" hook (via hook dispatching script: dispatch)
2023-09-25T11:48:59.796Z [container-agent] 2023-09-25 11:48:59 INFO juju.worker.uniter.operation runhook.go:186 ran "database-relation-changed" hook (via hook dispatching script: dispatch)
2023-09-25T11:49:03.521Z [container-agent] 2023-09-25 11:49:03 INFO juju.worker.uniter.operation runhook.go:186 ran "database-relation-joined" hook (via hook dispatching script: dispatch)
2023-09-25T11:49:04.888Z [container-agent] 2023-09-25 11:49:04 INFO juju.worker.uniter.operation runhook.go:186 ran "database-relation-joined" hook (via hook dispatching script: dispatch)
2023-09-25T11:49:06.082Z [container-agent] 2023-09-25 11:49:06 INFO juju.worker.uniter.operation runhook.go:186 ran "database-relation-changed" hook (via hook dispatching script: dispatch)
2023-09-25T11:49:07.706Z [container-agent] 2023-09-25 11:49:07 INFO juju.worker.uniter.operation runhook.go:186 ran "database-relation-changed" hook (via hook dispatching script: dispatch)
2023-09-25T11:50:15.766Z [container-agent] 2023-09-25 11:50:15 INFO juju-log Configuring instance
2023-09-25T11:50:35.725Z [container-agent] 2023-09-25 11:50:35 ERROR juju-log Uncaught exception while in charm code:
2023-09-25T11:50:35.725Z [container-agent] Traceback (most recent call last):
2023-09-25T11:50:35.725Z [container-agent]   File "/var/lib/juju/agents/unit-mysql-0/charm/./src/charm.py", line 724, in <module>
2023-09-25T11:50:35.725Z [container-agent]     main(MySQLOperatorCharm)
2023-09-25T11:50:35.725Z [container-agent]   File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/main.py", line 441, in main
2023-09-25T11:50:35.725Z [container-agent]     _emit_charm_event(charm, dispatcher.event_name)
2023-09-25T11:50:35.725Z [container-agent]   File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/main.py", line 149, in _emit_charm_event
2023-09-25T11:50:35.725Z [container-agent]     event_to_emit.emit(*args, **kwargs)
2023-09-25T11:50:35.725Z [container-agent]   File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/framework.py", line 344, in emit
2023-09-25T11:50:35.725Z [container-agent]     framework._emit(event)
2023-09-25T11:50:35.725Z [container-agent]   File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/framework.py", line 841, in _emit
2023-09-25T11:50:35.725Z [container-agent]     self._reemit(event_path)
2023-09-25T11:50:35.725Z [container-agent]   File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/framework.py", line 930, in _reemit
2023-09-25T11:50:35.725Z [container-agent]     custom_handler(event)
2023-09-25T11:50:35.725Z [container-agent]   File "/var/lib/juju/agents/unit-mysql-0/charm/./src/charm.py", line 531, in _on_mysql_pebble_ready
2023-09-25T11:50:35.725Z [container-agent]     if not self._configure_instance(container):
2023-09-25T11:50:35.725Z [container-agent]   File "/var/lib/juju/agents/unit-mysql-0/charm/./src/charm.py", line 486, in _configure_instance
2023-09-25T11:50:35.725Z [container-agent]     if workload_version := self._mysql.get_mysql_version():
2023-09-25T11:50:35.725Z [container-agent]   File "/var/lib/juju/agents/unit-mysql-0/charm/lib/charms/mysql/v0/mysql.py", line 1756, in get_mysql_version
2023-09-25T11:50:35.725Z [container-agent]     output = self._run_mysqlsh_script("\n".join(get_version_commands))
2023-09-25T11:50:35.725Z [container-agent]   File "/var/lib/juju/agents/unit-mysql-0/charm/src/mysql_k8s_helpers.py", line 672, in _run_mysqlsh_script
2023-09-25T11:50:35.725Z [container-agent]     stdout, _ = process.wait_output()
2023-09-25T11:50:35.725Z [container-agent]   File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/pebble.py", line 1354, in wait_output
2023-09-25T11:50:35.725Z [container-agent]     exit_code: int = self._wait()
2023-09-25T11:50:35.725Z [container-agent]   File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/pebble.py", line 1294, in _wait
2023-09-25T11:50:35.725Z [container-agent]     change = self._client.wait_change(self._change_id, timeout=timeout)
2023-09-25T11:50:35.725Z [container-agent]   File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/pebble.py", line 1794, in wait_change
2023-09-25T11:50:35.725Z [container-agent]     return self._wait_change_using_wait(change_id, timeout)
2023-09-25T11:50:35.725Z [container-agent]   File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/pebble.py", line 1815, in _wait_change_using_wait
2023-09-25T11:50:35.725Z [container-agent]     return self._wait_change(change_id, this_timeout)
2023-09-25T11:50:35.725Z [container-agent]   File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/pebble.py", line 1829, in _wait_change
2023-09-25T11:50:35.725Z [container-agent]     resp = self._request('GET', f'/v1/changes/{change_id}/wait', query)
2023-09-25T11:50:35.725Z [container-agent]   File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/pebble.py", line 1564, in _request
2023-09-25T11:50:35.725Z [container-agent]     response = self._request_raw(method, path, query, headers, data)
2023-09-25T11:50:35.725Z [container-agent]   File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/pebble.py", line 1599, in _request_raw
2023-09-25T11:50:35.725Z [container-agent]     response = self.opener.open(request, timeout=self.timeout)
2023-09-25T11:50:35.725Z [container-agent]   File "/usr/lib/python3.10/urllib/request.py", line 519, in open
2023-09-25T11:50:35.725Z [container-agent]     response = self._open(req, data)
2023-09-25T11:50:35.725Z [container-agent]   File "/usr/lib/python3.10/urllib/request.py", line 536, in _open
2023-09-25T11:50:35.725Z [container-agent]     result = self._call_chain(self.handle_open, protocol, protocol +
2023-09-25T11:50:35.725Z [container-agent]   File "/usr/lib/python3.10/urllib/request.py", line 496, in _call_chain
2023-09-25T11:50:35.725Z [container-agent]     result = func(*args)
2023-09-25T11:50:35.725Z [container-agent]   File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/pebble.py", line 286, in http_open
2023-09-25T11:50:35.725Z [container-agent]     return self.do_open(_UnixSocketConnection, req,  # type:ignore
2023-09-25T11:50:35.725Z [container-agent]   File "/usr/lib/python3.10/urllib/request.py", line 1352, in do_open
2023-09-25T11:50:35.725Z [container-agent]     r = h.getresponse()
2023-09-25T11:50:35.725Z [container-agent]   File "/usr/lib/python3.10/http/client.py", line 1375, in getresponse
2023-09-25T11:50:35.725Z [container-agent]     response.begin()
2023-09-25T11:50:35.725Z [container-agent]   File "/usr/lib/python3.10/http/client.py", line 318, in begin
2023-09-25T11:50:35.725Z [container-agent]     version, status, reason = self._read_status()
2023-09-25T11:50:35.725Z [container-agent]   File "/usr/lib/python3.10/http/client.py", line 279, in _read_status
2023-09-25T11:50:35.725Z [container-agent]     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
2023-09-25T11:50:35.725Z [container-agent]   File "/usr/lib/python3.10/socket.py", line 705, in readinto
2023-09-25T11:50:35.725Z [container-agent]     return self._sock.recv_into(b)
2023-09-25T11:50:35.725Z [container-agent] TimeoutError: timed out
2023-09-25T11:50:36.020Z [container-agent] 2023-09-25 11:50:36 ERROR juju.worker.uniter.operation runhook.go:180 hook "mysql-pebble-ready" (via hook dispatching script: dispatch) failed: exit status 1
2023-09-25T11:50:36.020Z [container-agent] 2023-09-25 11:50:36 ERROR juju.worker.uniter pebblepoller.go:101 pebble poll failed for container "mysql": failed to send pebble-ready event: hook failed
2023-09-25T11:50:37.302Z [container-agent] 2023-09-25 11:50:37 INFO juju.worker.uniter.operation runhook.go:186 ran "database-relation-changed" hook (via hook dispatching script: dispatch)
2023-09-25T11:50:39.650Z [container-agent] 2023-09-25 11:50:39 INFO juju.worker.uniter.operation runhook.go:186 ran "mysql-pebble-ready" hook (via hook dispatching script: dispatch)
2023-09-25T11:50:40.446Z [container-agent] 2023-09-25 11:50:40 INFO juju-log database:72: Adding pebble layer
2023-09-25T11:50:48.575Z [container-agent] 2023-09-25 11:50:48 INFO juju.worker.uniter.operation runhook.go:186 ran "database-relation-changed" hook (via hook dispatching script: dispatch)
2023-09-25T11:50:49.457Z [container-agent] 2023-09-25 11:50:49 INFO juju-log Adding pebble layer
2023-09-25T11:50:49.857Z [container-agent] 2023-09-25 11:50:49 INFO juju.worker.uniter.operation runhook.go:186 ran "mysql-pebble-ready" hook (via hook dispatching script: dispatch)
2023-09-25T11:50:50.814Z [container-agent] 2023-09-25 11:50:50 INFO juju.worker.uniter.operation runhook.go:186 ran "database-relation-changed" hook (via hook dispatching script: dispatch)
2023-09-25T11:50:51.795Z [container-agent] 2023-09-25 11:50:51 INFO juju.worker.uniter.operation runhook.go:186 ran "database-relation-joined" hook (via hook dispatching script: dispatch)
2023-09-25T11:50:52.881Z [container-agent] 2023-09-25 11:50:52 INFO juju.worker.uniter.operation runhook.go:186 ran "database-relation-changed" hook (via hook dispatching script: dispatch)
2023-09-25T11:50:53.843Z [container-agent] 2023-09-25 11:50:53 INFO juju.worker.uniter.operation runhook.go:186 ran "database-relation-changed" hook (via hook dispatching script: dispatch)
2023-09-25T11:50:54.793Z [container-agent] 2023-09-25 11:50:54 INFO juju.worker.uniter.operation runhook.go:186 ran "database-relation-joined" hook (via hook dispatching script: dispatch)
2023-09-25T11:50:55.784Z [container-agent] 2023-09-25 11:50:55 INFO juju.worker.uniter.operation runhook.go:186 ran "database-relation-joined" hook (via hook dispatching script: dispatch)
2023-09-25T11:50:56.665Z [container-agent] 2023-09-25 11:50:56 INFO juju.worker.uniter.operation runhook.go:186 ran "database-relation-changed" hook (via hook dispatching script: dispatch)
2023-09-25T11:50:57.583Z [container-agent] 2023-09-25 11:50:57 INFO juju.worker.uniter.operation runhook.go:186 ran "database-relation-changed" hook (via hook dispatching script: dispatch)
2023-09-25T11:50:58.488Z [container-agent] 2023-09-25 11:50:58 INFO juju.worker.uniter.operation runhook.go:186 ran "database-relation-joined" hook (via hook dispatching script: dispatch)
2023-09-25T11:50:59.457Z [container-agent] 2023-09-25 11:50:59 INFO juju.worker.uniter.operation runhook.go:186 ran "database-relation-changed" hook (via hook dispatching script: dispatch)
2023-09-25T11:51:00.397Z [container-agent] 2023-09-25 11:51:00 INFO juju.worker.uniter.operation runhook.go:186 ran "database-relation-changed" hook (via hook dispatching script: dispatch)

Status log:

juju show-status-log mysql/0
Time                   Type       Status       Message
25 Sep 2023 11:49:06Z  juju-unit  executing    running database-relation-changed hook for glance-mysql-router/0
25 Sep 2023 11:49:08Z  juju-unit  executing    running mysql-pebble-ready hook
25 Sep 2023 11:50:36Z  juju-unit  idle         
25 Sep 2023 11:50:36Z  juju-unit  executing    running database-relation-changed hook
25 Sep 2023 11:50:37Z  juju-unit  executing    running mysql-pebble-ready hook
25 Sep 2023 11:50:39Z  juju-unit  executing    running database-relation-changed hook
25 Sep 2023 11:50:48Z  juju-unit  executing    running mysql-pebble-ready hook
25 Sep 2023 11:50:49Z  juju-unit  executing    running database-relation-changed hook for nova-mysql-router/0
25 Sep 2023 11:50:50Z  juju-unit  executing    running database-relation-joined hook for cinder-mysql-router/0
25 Sep 2023 11:50:51Z  juju-unit  executing    running database-relation-changed hook for placement-mysql-router/0
25 Sep 2023 11:50:52Z  juju-unit  executing    running database-relation-changed hook for keystone-mysql-router/0
25 Sep 2023 11:50:53Z  juju-unit  executing    running database-relation-joined hook for horizon-mysql-router/0
25 Sep 2023 11:50:54Z  juju-unit  executing    running database-relation-joined hook for cinder-ceph-mysql-router/0
25 Sep 2023 11:50:55Z  juju-unit  executing    running database-relation-changed hook for horizon-mysql-router/0
25 Sep 2023 11:50:56Z  juju-unit  executing    running database-relation-changed hook for cinder-ceph-mysql-router/0
25 Sep 2023 11:50:57Z  juju-unit  executing    running database-relation-joined hook for nova-api-mysql-router/0
25 Sep 2023 11:50:58Z  juju-unit  executing    running database-relation-changed hook for cinder-mysql-router/0
25 Sep 2023 11:50:59Z  juju-unit  executing    running database-relation-changed hook for nova-api-mysql-router/0
25 Sep 2023 11:51:27Z  juju-unit  idle         
25 Sep 2023 11:51:27Z  workload   maintenance  Initialising mysqld

Additional context

I does not happen at every run, only some of them, but if it happens, mysql never become ready.

@gboutry gboutry added the bug Something isn't working label Sep 25, 2023
@github-actions
Copy link
Contributor

@gboutry
Copy link
Author

gboutry commented Sep 25, 2023

Another run with debug logs:
mysql2.log

relation data:

jhack show-relation mysql:database-peers --format json
{
  "type": "peer",
  "endpoints": [
    {
      "url": "mysql:database-peers",
      "relation_id": 8,
      "meta": {
        "scale": 1,
        "units": [
          0
        ],
        "leader_id": 0
      },
      "application_data": {
        "cluster-name": "cluster-58a3017b136b545d62e8d3b9fb03c6eb",
        "cluster-set-domain-name": "cluster-set-58a3017b136b545d62e8d3b9fb03c6eb",
        "secret-id": "secret://4cad2829-f96b-45b8-8277-47786a891317/ck8o22698o8tjbgj5gtg"
      },
      "units_data": {
        "mysql/0": {
          "unit-status": "alive"
        }
      },
      "model": null,
      "other_model": null
    }
  ]
}

@paulomach
Copy link
Contributor

@gboutry thanks for the report. Did you saw that on other juju version?

@gboutry
Copy link
Author

gboutry commented Sep 25, 2023

@gboutry thanks for the report. Did you saw that on other juju version?

I haven't, but I can try more

@motjuste
Copy link

I was using microk8s 1.24/stable with Juju 3.1.6, and kept facing this issue. Moving to microk8s 1.26/stable or 1.26-strict/stable worked.

@paulomach
Copy link
Contributor

@motjuste , this is potentially related to issue #337 , which is fixed on 8.0/beta. Can you test with it?

@motjuste
Copy link

I have unfortunately moved to 1.26 microk8s ... will update if I manage to get something running somewhere.

@gboutry
Copy link
Author

gboutry commented Feb 13, 2024

I have not seen this error for weeks now, seems fixed from my side.

@motjuste
Copy link

motjuste commented Feb 19, 2024

I have found something weird ... this problem occurs every time if I enable the mayastor add-on in microk8s 1.26 ... and simply enabling it causes this problem. Once I disable mayastor, the problem with the deployed MLFlow does not fix itself, but a fresh deployment after destroying the deployed apps of the MLFlow charm ends up working.

@swasenius
Copy link

swasenius commented Apr 17, 2024

Dear all
I seem to have the same issue, but I don't know what mayastor is so hard to disable it. What else could possibly cause this?

$ juju status mysql-k8s
Model     Controller                 Cloud/Region                      Version  SLA          Timestamp
kubeflow  datalake-kubeflow-test-v3  datalake-kubeflow-test.k8s.local  3.4.0    unsupported  21:48:59+03:00

App        Version  Status   Scale  Charm      Channel     Rev  Address        Exposed  Message
mysql-k8s           waiting      1  mysql-k8s  8.0/stable  113  100.99.195.98  no       waiting for units to settle down

Unit          Workload     Agent      Address          Ports  Message
mysql-k8s/0*  maintenance  executing  100.120.219.253         installing charm software
unit-katib-db-mysql-0: 21:48:19 ERROR unit.katib-db-mysql/0.juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-katib-db-mysql-0/charm/./src/charm.py", line 786, in <module>
    main(MySQLOperatorCharm)
  File "/var/lib/juju/agents/unit-katib-db-mysql-0/charm/venv/ops/main.py", line 436, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-katib-db-mysql-0/charm/venv/ops/main.py", line 144, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-katib-db-mysql-0/charm/venv/ops/framework.py", line 351, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-katib-db-mysql-0/charm/venv/ops/framework.py", line 853, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-katib-db-mysql-0/charm/venv/ops/framework.py", line 942, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-katib-db-mysql-0/charm/./src/charm.py", line 583, in _on_mysql_pebble_ready
    self._reconcile_pebble_layer(container)
  File "/var/lib/juju/agents/unit-katib-db-mysql-0/charm/./src/charm.py", line 345, in _reconcile_pebble_layer
    self._mysql.wait_until_mysql_connection()
  File "/var/lib/juju/agents/unit-katib-db-mysql-0/charm/venv/tenacity/__init__.py", line 289, in wrapped_f
    return self(f, *args, **kw)
  File "/var/lib/juju/agents/unit-katib-db-mysql-0/charm/venv/tenacity/__init__.py", line 379, in __call__
    do = self.iter(retry_state=retry_state)
  File "/var/lib/juju/agents/unit-katib-db-mysql-0/charm/venv/tenacity/__init__.py", line 325, in iter
    raise retry_exc.reraise()
  File "/var/lib/juju/agents/unit-katib-db-mysql-0/charm/venv/tenacity/__init__.py", line 158, in reraise
    raise self.last_attempt.result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/var/lib/juju/agents/unit-katib-db-mysql-0/charm/venv/tenacity/__init__.py", line 382, in __call__
    result = fn(*args, **kwargs)
  File "/var/lib/juju/agents/unit-katib-db-mysql-0/charm/src/mysql_k8s_helpers.py", line 236, in wait_until_mysql_connection
    if check_port and not self.check_mysqlsh_connection():
  File "/var/lib/juju/agents/unit-katib-db-mysql-0/charm/lib/charms/mysql/v0/mysql.py", line 2561, in check_mysqlsh_connection
    self._run_mysqlsh_script("\n".join(connect_commands))
  File "/var/lib/juju/agents/unit-katib-db-mysql-0/charm/src/mysql_k8s_helpers.py", line 666, in _run_mysqlsh_script
    stdout, _ = process.wait_output()
  File "/var/lib/juju/agents/unit-katib-db-mysql-0/charm/venv/ops/pebble.py", line 1436, in wait_output
    exit_code: int = self._wait()
  File "/var/lib/juju/agents/unit-katib-db-mysql-0/charm/venv/ops/pebble.py", line 1375, in _wait
    change = self._client.wait_change(self._change_id, timeout=timeout)
  File "/var/lib/juju/agents/unit-katib-db-mysql-0/charm/venv/ops/pebble.py", line 1893, in wait_change
    return self._wait_change_using_wait(change_id, timeout)
  File "/var/lib/juju/agents/unit-katib-db-mysql-0/charm/venv/ops/pebble.py", line 1914, in _wait_change_using_wait
    return self._wait_change(change_id, this_timeout)
  File "/var/lib/juju/agents/unit-katib-db-mysql-0/charm/venv/ops/pebble.py", line 1928, in _wait_change
    resp = self._request('GET', f'/v1/changes/{change_id}/wait', query)
  File "/var/lib/juju/agents/unit-katib-db-mysql-0/charm/venv/ops/pebble.py", line 1655, in _request
    response = self._request_raw(method, path, query, headers, data)
  File "/var/lib/juju/agents/unit-katib-db-mysql-0/charm/venv/ops/pebble.py", line 1690, in _request_raw
    response = self.opener.open(request, timeout=self.timeout)
  File "/usr/lib/python3.10/urllib/request.py", line 519, in open
    response = self._open(req, data)
  File "/usr/lib/python3.10/urllib/request.py", line 536, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/usr/lib/python3.10/urllib/request.py", line 496, in _call_chain
    result = func(*args)
  File "/var/lib/juju/agents/unit-katib-db-mysql-0/charm/venv/ops/pebble.py", line 311, in http_open
    return self.do_open(_UnixSocketConnection, req,  # type:ignore
  File "/usr/lib/python3.10/urllib/request.py", line 1352, in do_open
    r = h.getresponse()
  File "/usr/lib/python3.10/http/client.py", line 1375, in getresponse
    response.begin()
  File "/usr/lib/python3.10/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.10/http/client.py", line 279, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.10/socket.py", line 705, in readinto
    return self._sock.recv_into(b)
TimeoutError: timed out
unit-katib-db-mysql-0: 21:48:39 ERROR juju.worker.uniter.operation hook "mysql-pebble-ready" (via hook dispatching script: dispatch) failed: exit status 1
unit-katib-db-mysql-0: 21:48:39 ERROR juju.worker.uniter pebble poll failed for container "mysql": failed to send pebble-ready event: hook failed


@paulomach
Copy link
Contributor

Duplicates #535

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

4 participants