Note – Parsing Logs and Events with AWS Native Tools – SCS-C02 Study Guide

Note

If you choose the bucket you created earlier in this chapter, either remove the lifecycle policy that will delete any files created in the bucket after 24 hours or check the logs sent to that bucket before those log files are deleted.

  • After choosing your S3 bucket for delivery, scroll down to the bottom of the page and click the Create delivery stream button.

Your Amazon Kinesis Data Firehose has been set up, but it still needs to deliver data. There are two additional steps to accomplish before Kinesis Data Firehose sends logs to the S3 bucket you have designated when setting up the firehose. To create the IAM role, use the terminal.

  • Open up your terminal console or file editor and create a file named FirehoseTrustPolicy.json.
  • In the file, add the following text:

{

“Statement”: {

“Effect”: “Allow”,

“Principal”: { “Service”: “logs.us-east-2.amazonaws.com”},

“Action”: “sts:AssumeRole”

}

}

After creating the file, you can use the AWS CLI to create the role using the following command:

aws iam create-role –role-name FireHoseTrustRole –assume-role-policy-document file://FirehoseTrustPolicy.json

If the command succeeds, an ARN should be returned to you on the command line.

Next, you need to craft the access policy for the Kinesis, open up your text editor again, create a file named FireHoseAccess.json, and use the contents below to populate the file. Be sure to replace the account number in the statement (123456789012) with your own account number:

{

“Statement”:[

{

“Effect”:”Allow”,

“Action”:[“firehose:*”],

“Resource”:[“arn:aws:firehose:us-east-2:123456789

012:deliverystream/packt-chapter9-firehose”]

},

{

“Effect”:”Allow”,

“Action”:[“iam:PassRole”],

“Resource”:[“arn:aws:iam:: 123456789012:role/FireHoseTrustRole”]

}

]

}

Once you have the policy created, you can attach the policy to the role using the following AWS CLI command:

aws iam put-role-policy –role-name FireHoseTrustRole –policy-name FirehoseAccessPolicy –policy-document file://FirehoseAccess.json

After creating the IAM role, you will need to go back to the CloudWatch Logs service and configure it start sending the logs to the Firehose service. Follow the steps below:

  1. Switch back to your AWS Management Console and navigate to the CloudWatch service. You can get there quickly by using the following URL: http://console.aws.amazon.com/cloudwatch/
  2. Once you have reached the CloudWatch page, check the top-right corner of the page to confirm that you are in the Ohio region since you have used us-east-2 configuring other parts of this exercise. After confirming the region, use the left-side navigation to find the main header of Logs and choose the sub-menu item of Log groups.

Figure 9.10: CloudWatch Logs Menu

  • From the list of all your log groups on the page, click the name of a log group where there are active logs streaming or where you can have logs generated to that log group. (In the example, a simple Lambda function will be invoked to easily generate log files.) This will take you to the details of the log group.
  • Now on the details page of the log group, click on the Actions button, which is located directly beneath the name of the log group.

Clicking on the actions button will cause a drop-down menu to appear. Inside that drop-down menu, you will see a menu choice named Subscription filters, which is also a dropdown.

Figure 9.11: Action menu in Log group details

  • Click on the Subscription Filters dropdown to make another set of options appear, including Create Kinesis Firehose subscription filter. Use this choice to start the process of connecting your log group to your Kinesis Firehose.

Figure 9.12: Subscription filters for log groups

You should now be on the Create a Kinesis Firehose subscription filter page.

  • In the first box labeled Choose Destination, keep the Destination account of Current account selected. Under the Kinesis Firehose delivery stream heading, click in the textbox, and select your packt-chapter9-firehose stream that you created earlier in this exercise.

Figure 9.13: Destination selection for Firehose delivery stream

  • Scroll down on the page to the box labeled Grant permission. Under the heading Select an existing role, once again, click inside of the box, and you should be able to select the role you created earlier in this exercise named FireHoseTrustRole.

Figure 9.14: Grant permission screen for Firehose delivery

The next box on the page should be labeled Configure log format and filters. For these settings, you will need to add a subscription filter name.

  • Add the name Chapter9-Firehose.

Figure 9.15: Log format configuration for Firehose filter

  • After you have filled in these values, scroll down to the bottom of the page and click the Start streaming button.

Once you have the subscription filter set up, you only need to send some logs to the log group. Kinesis Firehose should pick up the logs via the subscription filter and place them in the S3 bucket you designated. You could even set your CloudWatch Logs group to expire in 3, 5, or 7 days since the logs are placed in long-term storage elsewhere.

After exploring so many different ways to move your log data from CloudWatch to other durable and more cost-effective storage such as S3, you now need a way to search through those stored files quickly should the need arise. The following section will examine just the tool needed to accomplish this: Amazon Athena.