[receiver/awsfirehosereceiver] Add support to ingest logs from services that send directly to firehose #36184
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mwear
reviewed
Nov 12, 2024
axw
reviewed
Nov 21, 2024
Comment on lines
+6
to
+10
| type firehoseLog struct { | ||
| Timestamp int64 `json:"timestamp"` | ||
| FirehoseARN string `json:"firehoseARN"` | ||
| Message string `json:"message"` | ||
| } |
There was a problem hiding this comment.
Nothing produces JSON with these fields does it? IIUC it's just for passing around in memory? If that's the case, I'd suggest dropping the json:"..." tags, as they confused me, and may confuse others.
This is unlike cwlog, where the cWLog struct holds the JSON structure of logs coming from CloudWatch.
| builders := make(map[resourceAttributes]*resourceLogsBuilder) | ||
| for _, record := range records { | ||
| var log firehoseLog | ||
| log.Message = string(record) |
There was a problem hiding this comment.
This will lead to unparsed JSON stored in the log record body, right? Should we be attempting to decode the record as JSON, and storing that as a structured body?
|
This PR was marked stale due to lack of activity. It will be closed in 14 days. |
|
Closed as inactive. Feel free to reopen if this PR is still being worked on. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Add support for logs sent from services directly to Firehose. For example VPC flow logs can be sent directly to Firehose and our
awsfirehosereceivershould be able to process these requests on top of logs from CloudWatch.This introduces a new
record_type: firehoselogsto separate the log type fromrecord_type: cwlogs. Since there can be logs from different AWS services send directly to Firehose, the format will be different too. Thisrecord_type: firehoselogswill process the request, store the whole record data intomessagefield, store the timestamp and firehose ARN.Link to tracking issue
#36379
Testing
Unit tests and manual smoke tests.
Documentation
Added a section in the README file.