Nginx Access Log To Kinesis

Question

I am looking for a scalable solution. Nginx will server a pixel (1x1 gif) with a query string to an html page. This query string will be in the nginx access logs. I need to stream, or send this data to Amazon kinesis so that we can then process it later. I have done some reading about Logstash, Fluentd, Ect. Is anyone doing this? What is the recommended why to turn the access log into events that can then be processed?

Thanks Brian

Answer 1

This is definitely possible with Fluentd. Amazon published its own plugin for AWS Kinesis. Fluentd has a feature to tail the nginx logs. Please check the URLs below.

• http://docs.fluentd.org/articles/kinesis-stream
• http://docs.fluentd.org/articles/in_tail
• https://github.com/awslabs/aws-fluent-plugin-kinesis