Retaining and Expiring CloudWatch Logs – CloudWatch and CloudWatch Metrics – SCS-C02 Study Guide

Retaining and Expiring CloudWatch Logs

As logs come into the CloudWatch service, they are stored in a log group and by default never expire. This can lead to extra charges for your account, especially for items such as developmental logs that don’t need to be saved in the account for extended periods of time due to compliance requirements.

If you would like for the logs of a particular log group to expire automatically, you can customize this configuration. The setting is customizable from 1 day to 10 years (3,653 days).

Figure 8.3: CloudWatch log groups log retention options

You can change this setting from any CloudWatch log group by clicking on the current retention setting. This will bring up a dialog window like the one shown in Figure 8.3 from which you can choose your new log retention setting. Any logs in the log group older than the setting would then be deleted.

Now that you know how to adjust the retention settings for the logs that come into the CloudWatch Logs service, you are ready to install the CloudWatch agent on an EC2 instance to capture both logs and metrics.

Installing and Using the CloudWatch Logging Agent

One of the best ways to understand how CloudWatch sends and collects logs, along with how the unified agent works, is to install the agent on an EC2 instance. The following tutorial will take you through the steps of standing up an EC2 instance and installing the agent onto the instance with the assistance of AWS Systems Manager. Once this has been completed, you can send some traffic to the instance and then look at the generated logs.

Note

A CloudWatch agent configuration file will be created and stored in the Systems Manager parameter store during the configuration. This part of the process is only required once. Once this configuration file has been created, the same file can be used on all your EC2 instances.

Creating the Necessary Roles

Start the process of configuring the CloudWatch logging agent by creating two new IAM roles. The first will be used to collect the log data and then pass that data on to the CloudWatch service. The second role will be communicating with the Systems Manager service to create and store your agent configuration file. This secondary role is especially crucial if you are trying to roll out a unified system with your EC2 instances and want to collect the same log files on each instance quickly. Another case where this is helpful is if you have a handful of different logging scenarios (e.g., where different flavors of Linux have logs stored in different paths). You need to have a separate configuration file for each operating system. All you need to maintain is the top-level configuration file and then have SSM do the actual installation on the instances.

Begin by creating the first IAM role. To make your role, open up the AWS Management Console. You can create the two roles that are needed through the following steps:

  1. To quickly get to the IAM service, use the URL https://packt.link/2CFNv.
  2.  Once on the IAM dashboard, choose Roles from the left-hand-side menu.

Figure 8.4: IAM dashboard menu

  • In the top-right corner, click the Create Role button.
  • Now that you’re on the Select trusted entity page, under Trusted entity type, ensure that AWS service is selected.

Figure 8.5: Trusted entity type screen

  • At the bottom of the page, select EC2 as the use case. Once you have done that, you can click the Next button at the bottom of the page.

Figure 8.6: Use case screen for IAM

After pressing the Next button, you should be on the Add permissions page. You will need to add two AWS-managed policies to this role.

  • Search for the policies by name and then select the box to the left side of the policy name. The policies are named CloudWatchAgentServerPolicy and AmazonSSMManagedInstanceCore. Once both policies are selected, press the Next button at the bottom of the page to proceed.

Figure 8.7: Policy name for roles

You now have the opportunity to enter the role details.

  • For the role name, enter CloudWatchAgentServerRole. Once done, scroll down to the bottom of the page and ensure that both policies have been attached to the role. Then press the blue Create role button.

You have now created the first role needed. Now go through the process of creating the second role. It is very similar to the previous steps, so refer to the preceding steps or the screenshots if you need to.

If you have just finished creating the previous role, you should be back on the IAM page showing the roles in your account. On this page, perform the following steps:

  1. Click on the Create role button in the top-right corner of the page.
  2. Once on the Select trusted entity page, choose Trusted entity type for AWS service on the top half of the page. Scroll to the bottom of the page and choose EC2 for Use case. Once you have finished these selections, click on the blue Next button at the bottom of the page.
  3. Moving on to the Add permissions page, use the search box to add the policies named CloudWatchAgentAdminPolicy and AmazonSSMManagedInstanceCore to the role. Select the box at the side of each policy to add them. Once done, click on the blue Next button at the bottom of the page.
  4. For the role details, enter CloudWatchAgentAdminRole for the role name. Double-check that both policies have been added to the role before scrolling down the page; then click the blue Create role button.

You have now created the IAM roles needed for the CloudWatch logging agent. The next section will take you through the process of creating an instance and then installing the CloudWatch agent on that instance.