Duplicate log messages

[skyjur]

I'm getting many duplicated log records on CloudWatch for almost every message. Any idea what can be cause of it? There aren't any duplclicates when I'm inspecting it with journalctl. I only have singe proccess running and configs are the minimum possible as described in readme.

[apparentlymart]

Hi @skyjur! Sorry for this weird behavior.

The program attempts to create batches of log entries to minimize the number of requests to the Cloudwatch Logs API under high throughput, but perhaps that batching behavior is misbehaving for some reason. Is the number of logs in your system either notably high or notably low?

Is the program producing any interesting log messages of its own? (We try to avoid writing anything to stdout and stderr in normal operation to minimize the risk of feedback loops if systemd is copying those streams back into the logs, so some messages may appear only in Cloudwatch Logs itself.)

[skyjur]

Hi @apparentlymart , thanks for quick response.

I guess number of messages is pretty high (currently hundreds per minute, but will increase even further). Perhaps systemd is not the right place for my logs in the first place. I think I should probably use metrics api, instead of logs, since my primary goal is to have few charts in dashboard with numbers of events happening in my app.

[apparentlymart]

Hi @skyjur!

My experience running this application was only at low volumes of logs and it's very possible that there are bugs at higher volumes; though I'm not sure if I'd consider "hundreds per minute" as a lot, I suspect it was more than was going on in the deployment I'm familiar with since, as you said, we were using metrics (via influxdb) rather than logs for the higher-volume stuff, using the logs just for exceptional information.

Unfortunately there's no-one actively working on this codebase right now since it's been doing its job for some years now without changes and so everyone's moved on. If you're seeing unexplained misbehavior then it may be better to seek another solution here since unfortunately it's unlikely that any of the former maintainers of this program will be able to spend time digging in and debugging for the foreseeable future. 😖

[aymanosman]

I seem to be getting every log-event repeated exactly twice. Even with a very low number of log events happening. My setup is AWS EC2, ubuntu, systemd. I'm just experimenting with the library, so it's not a high priority.

[whereisaaron]

It would be worth eliminating the possibility you have two agents running on that instance, or you have this agent and the AWS agent is installed.