Find missing date ranges in timeline using java

Question

I have a custom java sync that fetch data by date range thoght SOAP service running on tomcat. Ex:

getDataByDateRange(startDate,endDate)
getDataByDateRange('2016-01-01 10:00:00.00000','2016-01-01 11:00:00.00000')

I want to write a control program to check if any range has been missed by any kind of runtime or server error.

How can I find the missing date ranges?

Thanks.

Visually Example:

TimeLine       : [------------------------------------------------------------------]
Processed Dates: [----1---][---2----]---[-3-][--4---]---[----5---][---6--]-----------
Missing Dates  : -------------------[-1-]-----------[-2-]----------------[-----3----]

TimeLine: 
1: '2016-01-01 10:00:00.00000','2016-02-01 09:00:00.00000'

Processed Dates: 
1: '2016-01-01 10:00:00.00000','2016-01-01 11:00:00.00000'
2: '2016-01-01 11:00:00.00000','2016-01-01 12:00:00.00000'
3: '2016-01-01 13:00:00.00000','2016-01-01 13:30:00.00000'
4: '2016-01-01 13:30:00.00000','2016-01-01 14:30:00.00000'
5: '2016-01-01 15:30:00.00000','2016-01-01 16:30:00.00000'
6: '2016-01-01 16:30:00.00000','2016-01-01 17:00:00.00000'

Missing Dates:
1: '2016-01-01 12:00:00.00000','2016-01-01 13:00:00.00000'
2: '2016-01-01 14:30:00.00000','2016-01-01 15:30:00.00000'
3: '2016-01-01 17:00:00.00000','2016-01-02 09:00:00.00000'

Answer 1

According to your comment I post my previous comment as answer. This solution uses my library Time4J (including the range-module):

// prepare parser
ChronoFormatter<PlainTimestamp> f = 
  ChronoFormatter.ofTimestampPattern( // five decimal digits
    "uuuu-MM-dd HH:mm:ss.SSSSS", PatternType.CLDR, Locale.ROOT);

// parse input to intervals - here the overall time window
TimestampInterval timeline = 
  TimestampInterval.between(
    f.parse("2016-01-01 10:00:00.00000"), 
    f.parse("2016-02-01 09:00:00.00000"));

// for more flexibility - consider a for-each-loop
TimestampInterval i1 =
    TimestampInterval.between(f.parse("2016-01-01 10:00:00.00000"), f.parse("2016-01-01 11:00:00.00000"));
TimestampInterval i2 =
    TimestampInterval.between(f.parse("2016-01-01 11:00:00.00000"), f.parse("2016-01-01 12:00:00.00000"));
TimestampInterval i3 =
    TimestampInterval.between(f.parse("2016-01-01 13:00:00.00000"), f.parse("2016-01-01 13:30:00.00000"));
TimestampInterval i4 =
    TimestampInterval.between(f.parse("2016-01-01 13:30:00.00000"), f.parse("2016-01-01 14:30:00.00000"));
TimestampInterval i5 =
    TimestampInterval.between(f.parse("2016-01-01 15:30:00.00000"), f.parse("2016-01-01 16:30:00.00000"));
TimestampInterval i6 =
    TimestampInterval.between(f.parse("2016-01-01 16:30:00.00000"), f.parse("2016-01-01 17:00:00.00000"));

// apply interval arithmetic
IntervalCollection<PlainTimestamp> icoll =
    IntervalCollection.onTimestampAxis().plus(Arrays.asList(i1, i2, i3, i4, i5, i6));
List<ChronoInterval<PlainTimestamp>> missed = icoll.withComplement(timeline).getIntervals();

// result
System.out.println(missed); 
// [[2016-01-01T12/2016-01-01T13), [2016-01-01T14:30/2016-01-01T15:30), [2016-01-01T17/2016-02-01T09)]

The core of the whole interval arithmetic is just done by the code fragment icoll.withComplement(timeline). The rest is only about creation of intervals. By applying a for-each-loop you can surely minimize again the count of lines in presented code.

The output is based on the canonical description of the intervals implicitly using toString(), for example: [2016-01-01T12/2016-01-01T13) The square bracket denotes a closed boundary while the round bracket to the right end denotes an open boundary. So we have here the standard case of half-open timestamp intervals (without timezone). While other interval types are possible I have chosen that type because it corresponds to the type of your input strings.

If you plan to combine this solution with Joda-Time in other parts of your app then keep in mind that a) there is not yet any special bridge between both libraries available and b) the conversion looses microsecond precision (Joda-Time only supports milliseconds) and c) Time4J has much more power than Joda-Time (for almost everything). Anyway, you can do this as conversion (important if you don't want to do the effort of bigger rewriting of your app):

ChronoInterval<PlainTimestamp> missed0 = missed.get(0);
PlainTimestamp tsp = missed0.getStart().getTemporal();
LocalDateTime ldt = // joda-equivalent
    new LocalDateTime(
        tsp.getYear(), tsp.getMonth(), tsp.getDayOfMonth(), 
        tsp.getHour(), tsp.getMinute(), tsp.getSecond(), tsp.get(PlainTime.MILLI_OF_SECOND));
System.out.println(ldt); // 2016-01-01T10:00:00.000

About a Joda-only solution:

Joda-Time does only support instant intervals, not timestamp intervals without timezone. However, you could simulate that missing interval type by hardwiring the timezone to UTC (using fixed offset).

Another problem is missing support for five decimal digits. You can circumvent it by this hack:

DateTime start = 
  DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ss.SSS")
    .withZoneUTC()
    .parseDateTime("2016-01-01 16:30:00.00000".substring(0, 23));
System.out.println(start); // 2016-01-01T16:30:00.000Z
DateTime end = ...;
Interval interval = new Interval(start, end);

The other more critical element of a solution is almost missing - interval arithmetic. You have to sort the intervals first by start instant (and then by end instant). After sorting, you can iterate over all intervals such that you find the gaps. The best thing Joda-Time can do for you here is giving you methods like isBefore(anotherInstant) etc. which you can use in your own solution. But it gets pretty much bloated.

Answer 2

Given that the frequency of date ranges is one hour, you can start with range start date, iterate till range end date and write a method that checks for an entry with dates. You can use DateUtils to add hour to date, as shown in the below pseudo code:

Date startDate  = startDate;
Date endDate = endDate;
while (startDate.before(endDate){
    if(!exists(startDate, DateUtils.addHours(startDate, 1), entries)){
        //Add into missing entries
    }
    startDate = DateUtils.addHours(startDate, 1);
}