Splunk Java integration with JUnit

Question

I have made a module that lets me integrate with splunk using the Java SDK. We're using 1.2.1.0 via maven.

My module seems to work great. However, I want to put junit unit tests around it. I created tests that attempt to get back the most recent event, but unless I put a large sleep, I never get what I just put in, usually an event before it. I also tried it by event count on the index (I use a test index) but that also doesn't update properly. Any idea of a good way to do a JUnit test that I can actually validate and assert on?

I'm using spring in this app, so I have a singleton service to do this logging. Here's the service implementation:

/*
 * Copyright (c) 2015 POS Portal, Inc.
 * 180 Promenade Circle, Ste 215, Sacramento, CA 95834, USA
 * All rights reserved.
 *
 * This software is the confidential and proprietary information
 * of POS Portal, Inc.
 * 
 */
package com.posportal.splunk.impl;

import java.util.Date;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import com.posportal.splunk.api.SplunkLog;
import com.posportal.splunk.enumeration.SplunkLoggingLevel;
import com.splunk.Receiver;
import com.splunk.Service;
import com.splunk.ServiceArgs;

/**
 * Represents a splunk logging instance.  This is a spring singleton for logging. 
 * 
 * We log to the index given, so it is acceptable to have a singleton on a per index basis.  
 * 
 * Splunk configuration including index name are injected via Spring. 
 * @author Michael Wenk
 *
 */
public class SplunkLogImpl implements SplunkLog {
    private static Logger log = LoggerFactory.getLogger(SplunkLogImpl.class); 
    private String host;
    private String user;
    private String password;
    private String scheme;
    private String indexName;

    private int port;

    private boolean disabled = false; 

    private Service splunksvc = null;
    private Receiver receiver = null; 

    @Override
    public void logMessage(String msg, SplunkLoggingLevel level) {
        if (disabled) {
            log.warn("Splunk system disabled.  Splunk message would be: " + msg + " on level: " + level);
        } else {
            if (receiver == null) {
                initService();
            }
            String formattedMessageData = formatMessage(msg, level); 
            receiver.log(indexName, formattedMessageData);
        }
    }

    private String formatMessage(String msg, SplunkLoggingLevel level) {
        String fmt = "timestamp=\"%s\",level=\"%s\",data=\"%s\""; 
        Date now = new Date(); 
        return String.format(fmt, now.toString(), level.toString(), msg); 
    }

    private void initService() {
        ServiceArgs loginArgs = new ServiceArgs();
        loginArgs.setUsername(user);
        loginArgs.setPassword(password);
        loginArgs.setHost(host);
        loginArgs.setScheme(scheme);
        loginArgs.setPort(port);

        splunksvc = Service.connect(loginArgs);
        receiver = splunksvc.getReceiver(); 
    }

    public void setHost(String host) {
        this.host = host;
    }

    public void setUser(String user) {
        this.user = user;
    }

    public void setPassword(String password) {
        this.password = password;
    }

    public void setScheme(String scheme) {
        this.scheme = scheme;
    }

    public void setIndexName(String indexName) {
        this.indexName = indexName;
    }

    public void setPort(int port) {
        this.port = port;
    }

    @Override
    public void setDisabled(boolean disabled) {
        this.disabled = disabled;       
    }

    @Override
    public boolean isDisabled() {
        return disabled;
    }
}

Here's the unit test code:

/*
 * Copyright (c) 2015 POS Portal, Inc.
 * 180 Promenade Circle, Ste 215, Sacramento, CA 95834, USA
 * All rights reserved.
 *
 * This software is the confidential and proprietary information
 * of POS Portal, Inc.
 * 
 */
package com.posportal.splunk.test;

import static org.junit.Assert.assertNotNull;
import static org.junit.Assert.assertTrue;

import java.util.Map;

import javax.annotation.Resource;

import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.test.context.ContextConfiguration;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;

import com.posportal.splunk.api.SplunkLog;
import com.posportal.splunk.enumeration.SplunkLoggingLevel;
import com.splunk.Job;
import com.splunk.JobArgs;
import com.splunk.JobArgs.ExecutionMode;
import com.splunk.Service;
import com.splunk.ServiceArgs;

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration("classpath:pos-splunk-test-ctx.xml")
public class SplunkTest {
    //private static Logger log = LoggerFactory.getLogger(SplunkTest.class); 
    @Resource(name = "splunkLog")
    private SplunkLog splunk;

    @Resource(name = "splunkConfig")
    private Map<String,String> splunkConfig; 

    @Test
    public void testInitialMessage() throws InterruptedException
    {
        if (splunk.isDisabled())
        {
            String msg = "This is my test";
            splunk.logMessage(msg, SplunkLoggingLevel.INFO);
        }
        else
        {
            int startCount = getEventCountFromIndex(); 
            String msg = "This is my test";
            assertNotNull(splunk); 
            splunk.logMessage(msg, SplunkLoggingLevel.INFO);
            Service svc = getService(); 
            assertNotNull(svc); 
            // Sleep for a while. 
            Thread.sleep(4000);

            int finalCount = getEventCountFromIndex(); 


            assertTrue(finalCount > startCount); 

        }

    }

    @Test
    public void testDisabled() {
        if (!splunk.isDisabled())
        {
            splunk.setDisabled(true);
            splunk.logMessage("This is a disabled test", SplunkLoggingLevel.INFO);
            // Can't assert unfortunately, 
            //FIXME see if I can assert using log4j itself. 
        }
    }
    private int getEventCountFromIndex() {
        String searchString = "search index="+ splunkConfig.get("indexName");
        JobArgs jargs = new JobArgs();
        jargs.setExecutionMode(ExecutionMode.BLOCKING);
        Service svc = getService(); 
        Job j = svc.getJobs().create(searchString, jargs);
        return j != null ? j.getEventCount() : -1; 
    }
    @Test
    public void testSecondMessage() throws  InterruptedException
    {
        if (splunk.isDisabled())
        {
            String msg = "This is my second test";
            splunk.logMessage(msg, SplunkLoggingLevel.INFO);
        }
        else
        {
            int startCount = getEventCountFromIndex(); 
            String msg = "This is my second test";
            assertNotNull(splunk); 
            splunk.logMessage(msg, SplunkLoggingLevel.INFO);
            Service svc = getService(); 
            assertNotNull(svc); 
            // Sleep for a while. 
            Thread.sleep(4000);
            int finalCount = getEventCountFromIndex(); 
            assertTrue(finalCount > startCount); 
        }
    }


    private Service getService() {
        ServiceArgs loginArgs = new ServiceArgs();
        loginArgs.setUsername(splunkConfig.get("user"));
        loginArgs.setPassword(splunkConfig.get("password"));
        loginArgs.setHost(splunkConfig.get("host"));
        loginArgs.setScheme(splunkConfig.get("scheme"));
        int port = Integer.parseInt(splunkConfig.get("port"));
        loginArgs.setPort(port);
        Service service = Service.connect(loginArgs);
        return service;
    }

}

If you note the sleeps in there. If I don't sleep, I get no increase in the # of events in the index. And unfortunately sometimes 4 seconds is not enough.

Answer 1

The timing delay you're seeing is Splunk indexing the event, which involves writing to disk and can take time. The solution is, as you've already tried, wait for Splunk to finish.

I suggest setting an upper bound for how many times you want to try, and check every x seconds (the sleep time).

In the Splunk SDK for Java (see SearchJobTest.java and SDKTestCase.java), we essentially do the same thing with our assertEventuallyTrue() method:

public static boolean assertEventuallyTrue(EventuallyTrueBehavior behavior) {
    int remainingTries = behavior.tries;
    while (remainingTries > 0) {
        boolean succeeded = behavior.predicate();
        if (succeeded) {
            return true;
        } else {
            remainingTries -= 1;
            try {
                Thread.sleep(behavior.pauseTime);
            } catch (InterruptedException e) {}
        }
    }
    Assert.fail(behavior.timeoutMessage);
    return false;
}

Answer 2

I guess this is unit testing so you should not integrate with splunk for unit testing. Maybe what you try is acceptance test.

For your unit test you need to abstract from Splunk. So just create your own layer of abstraction from Splunk or mock the Splunk implementation.

So you can test if all your arguments are respected and go into creating the service and also you can test if all your log messages are good enough formatted and send to the service at all.

So in the end this is all that needs to be done. Abstract from Splunk or mock it and you will be fine.