Pausing/Resuming partitions causes lots of errors

[stampy88]

Hi,

I have noticed a problem with pausing and resuming of partitions that is causing lots of errors. When I stopped doing the pause/resume, the errors went away. Here is a screenshot of the Franz prometheus metrics when the application was actively pausing and resuming partitions:

Here is a screenshot after commenting out the pause/resume keeping everything else the same:

Here is a sample summary of errors during a 5 minute period from a pod:

 [
   {
    "msg": "read Fetch v6",
    "err": "context canceled",
    "count": 827
  },
  {
    "msg": "unable to open connection to broker",
    "err": "context canceled",
    "count": 117
  },
  {
    "msg": "wrote Fetch v6",
    "err": "context canceled",
    "count": 4
  },
  {
    "msg": "wrote Fetch v6",
    "err": "use of closed network connection",
    "count": 18
  },
  {
    "msg": "wrote Fetch v6",
    "err": "write tcp 10.73.0.182:35770->xx.xx.147.1:9093: use of closed network connection",
    "count": 1
  },
  {
    "msg": "wrote Fetch v6",
    "err": "write tcp 10.73.1.0:45126->xx.xx.147.1:9093: use of closed network connection",
    "count": 1
  },
  {
    "msg": "wrote Fetch v6",
    "err": "write tcp 10.73.1.0:54114->xx.xx.147.1:9093: use of closed network connection",
    "count": 1
  }
]

[twmb]

So we discussed in discord, and the reason this occurs is because the client takes a simplistic way to ensure partitions are not returned after they are paused (solving your original issue #489). To eliminate the chance that any in flight fetch request could be returning data for paused partitions, the client cuts all active fetch requests at the end of every pause.

An alternative approach than what's done in c3b083b would be to let these fetches return and strip the data after. An easyish way to do this would be, whenever we receive a fetch response, if we have paused partitions while processing the response, drop any response for paused partitions. This may not be too difficult.

An advanced approach would be to go further and try to do intelligence on what is inflight -- so that you are not waiting for a fetch response for data entirely for paused partitions. This is too complicated.

[twmb]

I think #601 solves this

[stampy88]

I think what you have done makes sense. That is a better way to handle it. Thanks for turning it around quickly.