How do I limit the amount of job console log on Jenkins?

Question

From time to time I encounter jobs that end-up putting Jenkins down because they log too much to the console, sometimes due to an more or less endless loop or too much verbosity.

I am looking for a solution that would prevent/stop/kill spamming jobs.

We are using Jenkins pipelines and all these occasional spamming jobs are inside sh(), which makes me think that maybe we should add some kind of smart wrapper.

Please, do not suggest fixing the spamming source instead. I asked this question because I want to prevent this kind of issue from outside the potential spammer script, that's because we have hundreds of jobs and because is impossible to avoid a sooner or later spamming bug caused by a developer mistake.

Shortly: we cannot control what is run inside the shell script, but we can control how the shell is run.

Answer 1

There is a plugin that stops jobs if their console log has exceeded a defined size limit.

It is called "Build log file size checker"

If your normally successful job logs are consistent in size, and do not vary hugely, then this could help you distinguish the looping or spamming ones.

Answer 2

After many hours wasted with various attempts to address this I was able to come up with a working solution that looks like:

def sh2(script) {
    sh '''( ''' + script + ''' ) 2>&1 | tee output.log | awk -v offset=${MAX_LINES:-200} '{ if (NR <= offset) print; else { a[NR] = $0; delete a[NR-offset] } } END { for (i=NR-offset+1; i<=NR; i++) { if (!match(a[i],"^[[:space:]]*$")) print a[i]} }
}

You can find the latest version at https://github.com/ssbarnea/rhos-ci/blob/master/vars/sh2.groovy in case I will find other bugs or improvements.

As you can see it will limit both head and tail to 200 lines if MAX_LINES is not defined. If defined it will be used.

This solution is the only one that does not repeat output when the total number of lines is <200. Also it does work in streaming mode, so you do see the output in real time.

Note: the linked version has extra features, like ability to log full output to auto-sequenced log files.