Could you recommend an unstructured data indexing software? [closed]

Question

I am collecting logs from several custom made applications. Each application has it's own log format. What I'm looking for is a central tool which would allow me to search through all of my logs. This means the tool would have to be able to define a different regex (or alike) for each log file (marking where a record begins, ends, and what are the fields). I've been trying Splunk, but I'm not happy with it, since performance are slow, I'm limited (free version) with the amount of indexed data per-day, and it's not as flexible as I want it to be. Could you recommend a software (preferably free or cheap) for the task?

Answer 1

You can try Lucene. It is free. It is written in Java, and it allows full-text search over large amount of data. It is not a complete application, but rather a library, so you have to write code that uses it to index and to search your logs. You may have to define different document types or at least different indexing functions for your logs, but then search works beautifully.