diff --git a/README.md b/README.md index 3ca4588..4040d54 100644 --- a/README.md +++ b/README.md @@ -1,2 +1,55 @@ -# aws-cloudwatch-firehose -Forwards logs from cloudwatch to NewRelic through firehose +# aws-unified-firehose +Forwards logs from cloudwatch to NewRelic through Amazon Data Firehose + +## Features + +- Collects logs from Amazon CloudWatch. +- Forwards logs to NewRelic using Amazon Data Firehose. +- Allows users to attach custom attributes to the logs to make it easier to search, filter, analyze, and parse the logs +- Scalable and reliable log forwarding. +- Stores license key in Secret Manager by default. + +## Requirements + +- SAM CLI - [Install the SAM CLI](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install.html) + +## Deployment + +To try this integration out you can use the `sam` cli to deploy the cloudformation template (`firehose-template.yml`). Make sure **aws is properly authenticated with an account of your choice**. + + +#### CloudFormation Parameters + +- `NewRelicRegion` : Can either be `US` or `EU` depending on which endpoint to be used to push logs to New Relic + - For this param `US` is default +- `LicenseKey`: Used when forwarding logs to New Relic +- `LogGroupConfig` : String representation of JSON array of objects of your CloudWatch LogGroup(s) and respective filter (if applicable) to set the Lambda function trigger. + - Example : ```[{"LogGroupName":"group1"}, {"LogGroupName":"group2", "FilterPattern":"ERROR"}, {"LogGroupName":"group3", "FilterPattern":"INFO"}]``` +- `LoggingFirehoseStreamName` : Name of new Data Firehose Delivery Stream (must be unique per AWS account in the same AWS Region) + - The default value will be `NewRelic-Logging-Delivery-Stream` +- `LoggingS3BackupBucketName`: S3 Bucket Destination for failed events (must be globally unique across all AWS accounts in all AWS Regions within a partition) + - The default value will be `firehose-logging-backup` +- `EnableCloudWatchLoggingForFirehose`: Can either be `true` or `false` to enable CloudWatch logging for the Amazon Data Firehose stream. Enabling logging can help in troubleshooting issues in pushing data through firehose stream. `false` by default +- `NewRelicAccountId` : The New Relic Account ID to which the logs will be pushed +- `CommonAttributes` : Common attributes to be added to all logs. This should be a JSON object. + - Example : ```[{"AttributeName": "name1", "AttributeValue": "value1"}, {"AttributeName": "name2", "AttributeValue": "value2}]``` +- `StoreNRLicenseKeyInSecretManager` : Can either be `true` or `false` depending on which cloud formation stack decides whether to store your license key in the environment variables or to create a new secret in aws secrets manger. + - For this param `true` is default + +## Building and packaging +To build and package, follow these steps: +1. Authenticate with your aws account details +2. Create an S3 bucket with a unique name, e.g., `test123`. +3. Build the project: + ```sh + sam build --template-file firehose-template.yaml + ``` +4. The build will be located by default at `.aws-sam/build`, and a template file will be created with the name `template.yaml`. +5. Package the project: + ```sh + sam package --s3-bucket test123 --template-file .aws-sam/build/template.yaml --output-template-file firehose-template.yaml --region us-east-2 + ``` +6. Copy the main template file to the S3 bucket: + ```sh + aws s3 cp .aws-sam/build/firehose-template.yaml s3://test123/firehose-template.yaml + ``` \ No newline at end of file diff --git a/firehose-cloudwatch-trigger-stack.yaml b/firehose-cloudwatch-trigger-stack.yaml index 46ccba2..f0d0ec9 100644 --- a/firehose-cloudwatch-trigger-stack.yaml +++ b/firehose-cloudwatch-trigger-stack.yaml @@ -2,7 +2,7 @@ AWSTemplateFormatVersion: '2010-09-09' Parameters: LogGroupConfig: - Description: "String representation of JSON array of objects of your CloudWatch Loggroup(s) and respective filter (if applicable)" + Description: "JSON array of objects representing your LogGroup and Filters (if applicable). For example: [{\"LogGroupName\":\"logGroup1\",\"FilterPattern\":\"filter1\"}]" Type: String LogGroupArns: Description: "Comma-separated list of CloudWatch Log Group ARNs to create subscription to Data Firehose" diff --git a/firehose-template.yaml b/firehose-template.yaml index f9a5c96..69f357f 100644 --- a/firehose-template.yaml +++ b/firehose-template.yaml @@ -58,7 +58,7 @@ Parameters: AllowedPattern: "[0-9]+" ConstraintDescription: must only contain numbers LogGroupConfig: - Description: "String representation of JSON array of objects of your CloudWatch Loggroup(s) and respective filter (if applicable)" + Description: "JSON array of objects representing your LogGroup and Filters (if applicable). For example: [{\"LogGroupName\":\"logGroup1\",\"FilterPattern\":\"filter1\"}]" Type: String LoggingFirehoseStreamName: Type: String @@ -84,7 +84,7 @@ Parameters: - "false" CommonAttributes: Type: String - Description: "String representation of JSON array of objects of custom attributes to organize your logs and make it easier for you to search, filter, analyze, and parse your logs" + Description: "JSON array of objects representing your custom attributes. For example: [{\"AttributeName\":\"key1\",\"AttributeValue\":\"value1\"}]" Default: "" StoreNRLicenseKeyInSecretManager: Type: String