Parse Output of Command into Variable LIVE (Network Traffic Monitoring)

Question

I am writing a network monitoring script in bash. The base command I am using is ettercap -T -M ARP -i en1 // //. Then I pipe egrep --color 'Host:|GET' into it.

A sample output I am getting looks like this:

GET /images/srpr/logo11w.png HTTP/1.1.
Host: www.google.com.
GET /en-us/us/products HTTP/1.1.
Host: www.caselogic.com.

My desired output is this:

Title: logo11w.png
URL: www.google.com/images/srpr/logo11w.png HTTP/1.1.

Title: Products - Case Logic
URL: www.caselogic.com/en-us/us/products

Things to notice: HTTP/1.1. and the . at the end of the host are gone. They also are formed into one URL and there is a blank line after each Title/URL listing. I attempted forming them into one URL by parsing the commands output into a variable with

var=`sudo ettercap -T -M ARP -i en1 // // | egrep --color 'Host:|GET'` | echo $var

but obviously that doesn't work because the input to the variable is a command the isn't done until the user requests a stop (CTRL + C).

To get the title of an HTML page, I use the command wget -qO- 'https://url.goes/here' | perl -l -0777 -ne 'print $1 if /<title.*?>\s*(.*?)\s*<\/title/si'. If it is something that doesn't have a title, such as an image, no title is fine.

Any help is greatly appreciated, and sorry if what I wrote is hard to read, feel free to ask questions.

Answer 1

Try this:

title_host.pl

#!/usr/bin/env perl

use warnings;
use strict;
use WWW::Mechanize;

my $mech = WWW::Mechanize->new();

my ($get,$host,$title);
while (<>) {
    if (m|^GET (\S+) |) {
        $get = $1;
    } elsif ( m|^Host: (\S+)\.| ) {
        $host = $1;
    } else {
        # Unrecognized line...reset
        $get = $host = $title = '';
    }

    if ($get and $host) {
        my ($title) = $get =~ m|^.*\/(.+?)$|; # default title
        my $url = 'http://' . $host . $get;
        $mech->get($url);
        if ($mech->success) {
            # HTML may have title, images will not
            $title = $mech->title() || $title;
        }
        print "Title: $title\n";
        print "URL: $url\n";
        print "\n";

        $get = $host = $title = '';
    }
}

input

GET /images/srpr/logo11w.png HTTP/1.1.
Host: www.google.com.
GET /en-us/us/products HTTP/1.1.
Host: www.caselogic.com.

now just pipe your input into the perl script:

cat input | perl title_host.pl

output:

Title: logo11w.png
URL: http://www.google.com/images/srpr/logo11w.png

Title: Products - Case Logic
URL: https://www.caselogic.com/en-us/us/products