Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Default timestamp_format to be compatible with zero padding and non zero padding #870

Merged
merged 18 commits into from
Oct 16, 2023
Merged

Default timestamp_format to be compatible with zero padding and non zero padding #870

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
ab921ff
default to %d for both %d and %-d
ymtaye Sep 28, 2023
639f943
revert original test case changes
ymtaye Sep 29, 2023
cccfe1b
add default for %-m and timestamp translation unit tests
ymtaye Sep 29, 2023
110e542
fix typos
ymtaye Sep 29, 2023
02b4a2d
fix unit tests
ymtaye Sep 29, 2023
e3569b7
add debug messaging
ymtaye Sep 29, 2023
34b3111
make timestamp_layout compatible
ymtaye Oct 4, 2023
5d23156
remove excessive logging
ymtaye Oct 4, 2023
70588d9
rebase
ymtaye Oct 16, 2023
00bf542
fix linter
ymtaye Oct 5, 2023
32138e4
address PR suggestions
ymtaye Oct 5, 2023
1ec6073
fix lint
ymtaye Oct 5, 2023
06170ff
add log sampling
ymtaye Oct 9, 2023
f9eb9df
rebase and add log throttling
ymtaye Oct 9, 2023
0fb678d
address PR comments
ymtaye Oct 11, 2023
7c3a189
fix timeformatting
ymtaye Oct 12, 2023
3201655
address PR suggestions
ymtaye Oct 13, 2023
d4ec621
rebase and address nitpicks
ymtaye Oct 16, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions plugins/outputs/cloudwatchlogs/pusher.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,9 @@ import (
)

const (
reqSizeLimit = 1024 * 1024
reqEventsLimit = 10000
logWarnInterval = 1 * 24 * time.Hour
reqSizeLimit = 1024 * 1024
reqEventsLimit = 10000
warnOldTimeStamp = 1 * 24 * time.Hour
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor nit: Still think threshold should be somewhere in the name, but it isn't blocking.

)

var (
Expand Down Expand Up @@ -417,8 +417,8 @@ func (p *pusher) convertEvent(e logs.LogEvent) *cloudwatchlogs.InputLogEvent {
t = p.lastValidTime
if !p.lastUpdateTime.IsZero() {
// Check when timestamp has an interval of 5 days.
if time.Since(p.lastUpdateTime) > logWarnInterval {
p.Log.Warnf("Unable to parse timestamp, using last valid timestamp %v: which is at least older by 1 day for log group %v: ", p.lastValidTime, p.Group)
if time.Since(p.lastUpdateTime) > warnOldTimeStamp {
p.Log.Warnf("Unable to parse timestamp, using last valid timestamp found in the logs %v: which is at least older than 1 day for log group %v: ", p.lastValidTime, p.Group)
}
}
} else {
Expand Down
4 changes: 2 additions & 2 deletions translator/tocwconfig/sampleConfig/no_skip_log_timestamp.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,8 @@
log_group_name = "amazon-cloudwatch-agent.log"
pipe = false
retention_in_days = -1
timestamp_layout = "15:04:05 06 Jan 02"
timestamp_regex = "(d{2}:d{2}:d{2} d{2} w{3} d{2})"
timestamp_layout = ["15:04:05 06 Jan _2"]
timestamp_regex = "(d{2}:d{2}:d{2} d{2} w{3} s{0,1} d{1,2})"

[outputs]

Expand Down
4 changes: 2 additions & 2 deletions translator/tocwconfig/sampleConfig/no_skip_log_timestamp_windows.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,8 @@
log_group_name = "amazon-cloudwatch-agent.log"
pipe = false
retention_in_days = -1
timestamp_layout = "15:04:05 06 Jan 02"
timestamp_regex = "(d{2}:d{2}:d{2} d{2} w{3} d{2})"
timestamp_layout = ["15:04:05 06 Jan _2"]
timestamp_regex = "(d{2}:d{2}:d{2} d{2} w{3} s{0,1} d{1,2})"

[outputs]

Expand Down