Intermittent broken backend connections when using have_ssl=true #4500
Comments
|
Hi @amickael, fronted and backend connections are isolated from each other, enabling or disabling Thanks, Javier. |
|
Hi @JavierJF, Thanks for replying. We don't see any corresponding errors in the MySQL errorlog which is the oddest part about this. We also have a much longer |
|
Hi @amickael, it's weird that you can't find errors at MySQL side, this scenario has two possibilities:
(EDIT: There are other possible scenarios, but for the sake of simplicity, let's consider these two for now, until it's clear that it's neither one of these posibilities) In both cases, the connection is being terminated in an abnormal way, and MySQL should at least have an entry in the error log for this. This logging wont take place with the default verbosity configuration, you should increase it for seeing them: Hope this helps to reveal the cause of the broken connections. Thanks, regards, Javier. |
@JavierJF is that entirely true? My understanding is that SSL errors and such are queued on a per-thread basis. Do the backend and frontend conns not share the same proxysql thread? |
|
Hi @joshuahunt, it's not entirely true, the scenario you describe is a possible one, but as my previous comment mentions:
First it would be required to rule out other potential scenarios, before starting making hypothesis without having enough evidence of the problem itself. For this issue, MySQL error log in combination with ProxySQL error log (with the previous requested flag), should be the first piece of evidence for diagnosing a potential issue. Thanks, Javier. |
|
Thanks @JavierJF, we increased the MySQL error log verbosity and still did not see any related errors. We will take a look at the way the client is terminating the connections. When you say:
Is there anything specific we should be looking for? |
|
Hi @amickael, MySQL will log whenever a connection hasn't been correctly closed by a client. This goes for when MySQL is going to kill the connection due to a timeout: Or, in case the client broken protocol in any way, or disconnected in a unhealthy way, it's also usual to see a error like the following in MySQL error log: I was interested in if any error of this shape was present in MySQL error log (when set to Thanks, regards. Javier. |
|
We're experiencing a similar issue (also version 2.5.5). In the proxysql logs we're seeing a lot of errors in the following format: this generally correlates with the following logs on mysql server: and in general mysql log we can correlate the events with: It is hard to tell if the general log is from the retry tho. In the application we see related: Turning off have_ssl immediately quiets down the whole thing. We have tried changing packet size and keepalive settings and all kinds of timeouts, but that had no effect. at this point our config looks like |
|
Hi @artemvovk, thanks for adding more details about the behavior you are experiencing. As you mention, the
When you say Turning off have_ssl immediately quiets down the whole thing you mean the errors on both sides, or just the errors in client side? Can you confirm if the errors in the server side are no longer present after Also, even if the previous hold, what would really help to investigate the issue without a clear reproduction case would be a Thank you, Javier. |
|
Hello, with have_ssl=false I am seeing that the application clients stop receiving errors and proxysql logs about "Lost connection during query" stop. On MySQL servers we still see occasional "communication packet" errors, but, as far as I can tell, those come from the ProxySQL connection max age setting (unless that got less noisy recent updates?). I will work on a more concise reproducible case, but these errors happen on our busier ProxySQL servers and happen intermittently, so it is hard to pin down what sequence of events definitively causes the dropped query. The best I might be able to come up with is a tcpdump from mysql server and proxysql from a set period of time. It'll take a day or two to get that. We have narrowed down this to correlate with proxysql version updates between 2.5.5 and 2.6.4. With proxysql version downgrade the error rate for the applications disappears. |
|
Hi @artemvovk, just checking for keeping the issue alive. Were you able to get the Thank you, Javier. |
|
I have just submitted an issue which seems similar to this one so wanted to note it here #4556 |
|
Hi @artemvovk and @amickael, without the evidence provided by Please let us know if that is the case and the issue is already fixed, since it looks like it may be already misleading other users. Thank you! Regards, Javier. |
|
Much apprecated @JavierJF, we will upgrade to the latest release and report back. |
|
Hey @JavierJF, we deployed the latest release and it seems to have resolved this issue! We no longer see the errors described above when using the same configuration. I will close this issue, appreciate all your support 🙇 |
|
Hi @amickael, yw! I'm glad the issue has been resolved. Thanks for following up and closing the issue. Regards, Javier. |
Recently we have been trying to enable frontend SSL support for ProxySQL, i.e.
have_ssl=true. We have had backend SSL enabled for a few weeks (use_ssl=1) without any issues, however when settinghave_ssl=truewe intermittently get the following errors:On the application side we use Django with
mysqlclient/MySQLdb, and these errors are surfaced as:Disabling frontend SSL immediately causes the errors to stop.
We are using version
2.5.5and our ProxySQL configuration is as follows:The text was updated successfully, but these errors were encountered: