Eliminate the top-level field in Logstash

Question

I am using Logstash and one of my applications is sending me fields like:

[message][UrlVisited]
[message][TotalDuration]
[message][AccountsProcessed]

I'd like to be able to collapse these fields, removing the top level message altogether. So the above fields will become:

[UrlVisited]
[TotalDuration]
[AccountsProcessed]

Is there a way to do this in Logstash?

Answer 1

Assuming the names of all such subfields are known in advance you can use the mutate filter:

filter {
  mutate {
    rename => ["[message][UrlVisited]", "UrlVisited"]
  }
  mutate {
    rename => ["[message][TotalDuration]", "TotalDuration"]
  }
  mutate {
    rename => ["[message][AccountsProcessed]", "AccountsProcessed"]
  }
  mutate {
    remove_field => ["message"]
  }
}

Alternatively, use a ruby filter (which works even if you don't know the field names):

filter {
  ruby {
    code => "
      event.get('message').each {|k, v|
        event.set(k, v)
      }
      event.remove('message')
    "
  }
}

This example works on Logstash 2.4 and later. For earlier versions use event['message'].each ... and event[k] = v instead.