Logstash : send logs using curl

Question

I have an ELK server to manage my logs. Generally they are send to automaticaly. All works fine. But for some reason, sometimes, we need to send a log file.

I'm trying to do like this :

curl -F "file=@<my_file_path>" http://<ELK_server_host>:5001

And this conf (part that is usefull here) on the server side :

input {
    ...
    tcp {
        port => 5001
        type => "curl"
        codec => line { charset => "UFT-8" }
    }
    .....
}

filter {
    .....
    if "curl" == [type] {
        multiline {
            pattern => "^%{TIMESTAMP_ISO8601}"
            negate => true
            what => previous
        }
        grok {.....}
        date {.....}
    }
    .....
}

output {
    stdout { codec => rubydebug }
    elasticsearch {
        host => localhost
    }
}

This work really good BUT there is two little difficulties :

• The last (not empty) line of the file is ignored
• Curl command never stop. I have to kill with Ctrl+C.

Maybe this two points are connected but what is there a solution? Maybe the multiline filter is a problem?

Answer 1

Part 1:

It's hard to answer this without more information. But I'd expect the last line of your logfile doesn't end with a newline character? The logstash input codec "line" is looking for a newline at the end of each line.

To add one in Windows:

echo. >> path\to\your_log_file

Part 2:

cURL is made for talking to HTTP servers. Logstash input "tcp" is not an HTTP server and doesn't send the HTTP response cURL is expecting at the end of the upload.

You'd be better off using a tool like Netcat that is made to send data over TCP.

P.S

"UFT-8" looks like a typo.