using cloud services to aggregate and group real-time statistics in a time window to trigger notifications

Question

I'm trying to build a real-time achievements processor for things like:

1. every time there is a new participant in a thread, send a notification to the last 3 participants

2. group and aggregate activity stream notifications by type per day

This description of event stream processing seems like a good fit for what I need https://en.wikipedia.org/wiki/Event_stream_processing

If the use case were just to update or trigger from single events, I can use one of the many cloud queue or publisher services from amazon or azure, things like Kinesis or SQS and use say an AWS lambda function to process messages from the queue. Azure seems like it offers something called an Event Hub which can act as the data stream broadcaster. Essentially, have a cloud queue of all actions/events and multiple notification processors as subscribers to the events stream(s) and the logic triggers and aggregations and achievement awards are encapsulated in each achievement processor.

However, since I need to group items by some arbitrary rules (each achievement can have many grouping parameters), I can't just simply look at the latest event in the action queue to process each achievement in real-time. Would I have to keep a set in memory to make this efficient? The alternative is to have each achievement processor do a database lookup with every event (e.g. to select all events for the day that match this type) but I'm worried if I do that it will not be very performant. I've heard mention of things like spark streaming and snowplow, so I'm wondering if there is both a pattern and a product on either AWS or Azure cloud services that can be useful to solve this in a very scalable and simple manner - and if the existing data streaming services on azure and aws (event hubs and kinesis) would fit this data-aggregation use case.

Answer 1

Both Azure and AWS now offer something that can fit this use case:

https://azure.microsoft.com/en-us/services/stream-analytics/

and

https://aws.amazon.com/kinesis/analytics/