Summary In this chapter, you learned how to review and analyze many log files using native tools found in AWS. You also had a brief overview of how costs can affect your decisions when choosing the correct storage solution for your long-term log storage for your account and your organization. This overview was presented through […]

Running Queries with Amazon Athena With so much valuable data within CloudTrail and other logs, finding effective ways to query the data for specific entries is always a top priority. The Amazon Athena service makes running ad hoc queries on extensive datasets much more straightforward. When discussing storing logs on the S3 service earlier in […]

Note If you choose the bucket you created earlier in this chapter, either remove the lifecycle policy that will delete any files created in the bucket after 24 hours or check the logs sent to that bucket before those log files are deleted. Your Amazon Kinesis Data Firehose has been set up, but it still […]

Using Amazon Kinesis to Process Logs Amazon Kinesis Firehose is a fully managed service that can scale automatically with the volume of the data throughput that you are sending to it. It can scale automatically if large quantities of data suddenly appear and then scale back down again. Data can be encrypted and sent to […]

Comparing Costs of Storing Logs in S3 versus CloudWatch Logs You may think that because you are studying for a test that emphasizes security, the pricing does not matter. However, this is not the case because keeping costs under control is everyone’s responsibility in an organization. The ability to choose the proper storage for the […]

Creating a Lifecycle Policy for an S3 Bucket This exercise uses the AWS Management Console. To create a Lifecycle policy on a new S3 bucket, perform the following steps: Figure 9.2: Configuration for S3 bucket screen This will bring up the Lifecycle rules section in the middle of the main screen. It should have a […]

S3 Glacier Flexible Retrieval The S3 Glacier Flexible Retrieval storage tier provides you with a low-cost, durable storage archive with low fees for data retrieval. There are three tiers of retrieval speeds to bring back your data for access to your S3 bucket. However, unlike the Glacier storage service available from AWS, you do not […]

S3 Standard Infrequent Access (S3 Standard-IA) If you have data you don’t access frequently but still need to retrieve in real time, the S3 Standard-IA storage tier provides an excellent option. If you decide to use this storage tier, consider the following: the files and objects you would store in your S3 bucket would need […]

Storing Logs on S3 One of the optimal places to store log files—either initially or after placing them in another store—is the S3 storage service. This service provides both the resiliency and durability desired for long-term storage and objects such as log files. Where it can fall short is the ability to search the various […]

The previous chapter showed you how the CloudWatch service can help collect and store logs from a myriad of services in AWS. You are now ready to turn your attention to the most cost-effective ways to retain those log files for long-term storage, along with the methods to pull out the necessary data from them. […]