Read error looking for ack: EOF

[evalencia]

Hi all,
OS: Ubuntu 14.04.1 LTS
logstash-forwarder: logstash-forwarder_0.4.0_amd64.deb

Im deploying the new logtash-forwarder debian package from master and I keep seeing the following error:
==> /var/log/logstash-forwarder/logstash-forwarder.err <==
2015/03/06 04:46:30.432719 Setting trusted CA from file: /etc/ssl/logstash-forwarder/ca-cert.crt
2015/03/06 04:46:30.433663 Connecting to [10.0.0.100]:5043 (logstash.connect.domain.com)
2015/03/06 04:46:30.506904 Connected to 10.0.0.100
2015/03/06 04:46:30.623772 Read error looking for ack: EOF
2015/03/06 04:46:30.623868 Loading client ssl certificate: /etc/ssl/logstash-forwarder/logstash-forwarder.crt and /etc/ssl/logstash-forwarder/logstash-forwarder.key
2015/03/06 04:46:30.772146 Setting trusted CA from file: /etc/ssl/logstash-forwarder/ca-cert.crt
2015/03/06 04:46:30.773207 Connecting to [10.0.0.100]:5043 (logstash.connect.domain.com)
2015/03/06 04:46:30.838942 Connected to 10.0.0.100
2015/03/06 04:46:30.914806 Read error looking for ack: EOF
2015/03/06 04:46:30.914892 Loading client ssl certificate: /etc/ssl/logstash-forwarder/logstash-forwarder.crt and /etc/ssl/logstash-forwarder/logstash-forwarder.key
2015/03/06 04:46:31.063929 Setting trusted CA from file: /etc/ssl/logstash-forwarder/ca-cert.crt
2015/03/06 04:46:31.064843 Connecting to [10.0.0.100]:5043 (logstash.connect.domain.com)
2015/03/06 04:46:31.238920 Connected to 10.0.0.100
2015/03/06 04:46:31.304081 Read error looking for ack: EOF
2015/03/06 04:46:31.304166 Loading client ssl certificate: /etc/ssl/logstash-forwarder/logstash-forwarder.crt and /etc/ssl/logstash-forwarder/logstash-forwarder.key
2015/03/06 04:46:31.453161 Setting trusted CA from file: /etc/ssl/logstash-forwarder/ca-cert.crt
2015/03/06 04:46:31.454245 Connecting to [10.0.0.100]:5043 (logstash.connect.domain.com)
2015/03/06 04:46:31.631891 Connected to 10.0.0.100
2015/03/06 04:46:31.632169 Read error looking for ack: EOF
2015/03/06 04:46:31.632213 Loading client ssl certificate: /etc/ssl/logstash-forwarder/logstash-forwarder.crt and /etc/ssl/logstash-forwarder/logstash-forwarder.key
2015/03/06 04:46:31.781060 Setting trusted CA from file: /etc/ssl/logstash-forwarder/ca-cert.crt
2015/03/06 04:46:31.782044 Connecting to [10.0.0.100]:5043 (logstash.connect.domain.com)
2015/03/06 04:46:31.871092 Connected to 10.0.0.100

[rdemey]

I'm having the exact same issue. I built an RPM package from the master on a CentOS 6.6 i686 ( logstash-forwarder-0.4.0-1.i686.rpm ), installed it and configured it like my other servers.

I'm running Logstash 1.4.2 on a CentOS 7.

Note that my other machines are not displaying this behavior.

2015/03/06 13:02:26.519881 Registrar: processing 1 events
2015/03/06 13:03:06.453740 Read error looking for ack: EOF
2015/03/06 13:03:06.453910 Setting trusted CA from file: /etc/httpd/conf/cert/DigiCertCA.crt
2015/03/06 13:03:06.455650 Connecting to [192.168.2.30]:5000 (sub.company.com)
2015/03/06 13:03:06.516296 Connected to 192.168.2.30
2015/03/06 13:03:06.518909 Registrar: processing 1 events
2015/03/06 13:03:18.953079 Read error looking for ack: EOF
2015/03/06 13:03:18.953242 Setting trusted CA from file: /etc/httpd/conf/cert/DigiCertCA.crt
2015/03/06 13:03:18.955250 Connecting to [192.168.2.30]:5000 (sub.company.com)
2015/03/06 13:03:19.017604 Connected to 192.168.2.30
2015/03/06 13:03:19.020071 Registrar: processing 1 events
2015/03/06 13:03:28.952725 Read error looking for ack: EOF
2015/03/06 13:03:28.952820 Setting trusted CA from file: /etc/httpd/conf/cert/DigiCertCA.crt
2015/03/06 13:03:28.954575 Connecting to [192.168.2.30]:5000 (sub.company.com)
2015/03/06 13:03:29.015474 Connected to 192.168.2.30
2015/03/06 13:03:29.022845 Registrar: processing 1 events
2015/03/06 13:03:38.952713 Read error looking for ack: EOF
2015/03/06 13:03:38.952812 Setting trusted CA from file: /etc/httpd/conf/cert/DigiCertCA.crt
2015/03/06 13:03:38.954270 Connecting to [192.168.2.30]:5000 (sub.company.com)
2015/03/06 13:03:38.989218 Connected to 192.168.2.30
2015/03/06 13:03:38.989359 Read error looking for ack: EOF
2015/03/06 13:03:38.989408 Setting trusted CA from file: /etc/httpd/conf/cert/DigiCertCA.crt
2015/03/06 13:03:38.991046 Connecting to [192.168.2.30]:5000 (sub.company.com)
2015/03/06 13:03:39.051691 Connected to 192.168.2.30
2015/03/06 13:03:39.054130 Registrar: processing 4 events
2015/03/06 13:05:13.953443 Read error looking for ack: EOF
2015/03/06 13:05:13.953561 Setting trusted CA from file: /etc/httpd/conf/cert/DigiCertCA.crt
2015/03/06 13:05:13.955505 Connecting to [192.168.2.30]:5000 (sub.company.com)

[TheHank]

Same issue here, installed 0.4.0 through the Debian repository and getting this error. The strange thing is that i get stuff reported to the logstash server. (side note that i'm "sure" is not related to this, only syslog entries gets forwarded, have nginx access and error logs also added to the config. Also feels like the syslog entries only loops, have tailed the stdout on logstash, and same information gets recorded over and over again. So maybe that the forwarder is DDOSing the logstash server.)

2015/03/06 16:45:54.085117 Setting trusted CA from file: /etc/pki/tls/certs/logstash-forwarder.crt
2015/03/06 16:45:54.085818 Connecting to [192.168.0.155]:5000 (logstash.i.company.com)
2015/03/06 16:45:54.145838 Connected to 192.168.0.155
2015/03/06 16:45:55.114476 Read error looking for ack: EOF

[evalencia]

The issue I've seen is that it doesn't keep the offset and the same message keeps getting logged, at least thats what I'm initially seeing.

[jordansissel]

@evalencia If "Read error looking for ack: EOF" happens, logstash-forwarder will retransmit the last payload because that payload was not acknowledged.

I don't know what you mean, otherwise, by "doesn't keep the offset".

[jordansissel]

Let's move discussion to #293.