AWS Log Aggregator on the Cheap

Question

Our CIO had a heart attack upon seeing our AWS bill.

I need to aggregate Apache and Tomcat logs from multiple EC2 (in scaling group) -- what could be the best way to initiate this without breaking the bank? The goal of the logs is to view events by IP address, account names, view the transaction flows (diagnostic/audit logging -- not so much as performance metrics).

ELK is out of the equation (political). Cloudwatch is allowed + anything else.

Answer 1

Depends on volume and access patterns, but pushing the logs to S3 and using Athena to query them is a good shout.

Its cheap because S3 is a really cheap datastore, and Athena is server-less, meaning you only pay for the queries you run.

Make sure you convert the logs to a compressed data format (like Apace Parquet) to save even more dosh.

https://aws.amazon.com/athena

https://docs.aws.amazon.com/athena/latest/ug/querying-apache-logs.html

https://aws.amazon.com/blogs/big-data/analyzing-data-in-s3-using-amazon-athena/