Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Request multithreading with array

This script reads a file of urls to do multithreading HTTP requests.

How can I use an array with urls to make multithreading requests?

My array will have something like:

@array = ("https://example.com/xsd","https://example.com/xys","https://example.com/des","https://example.com/hduei");

I need to remove the function of reading file with urls, but I can not.

#!/usr/bin/perl
use strict;
use warnings;
use threads;
use threads::shared;

use Fcntl              qw( LOCK_EX );
use IO::Handle         qw( );
use LWP::UserAgent     qw( );
use Thread::Queue 3.01 qw( );

use constant NUM_WORKERS => 20;

my $output_lock :shared;
my $output_fh;
sub write_to_output_file {
   lock($output_lock);
   print($output_fh @_);
   $output_fh->flush();
}

sub worker {
   my ($ua, $url) = @_;
   my $response = $ua->get($url);
   write_to_output_file("$url\n")
      if $response->success
      && $response->content =~ /Exist/;       
}

{
   $output_fh = \*STDOUT;  # Or open a file.

   my $q = Thread::Queue->new();
   for (1..NUM_WORKERS) {
      async {
         my $ua = LWP::UserAgent->new( timeout => 15 );
         while (my $job = $q->dequeue()) {
            worker($ua, $job);
         }
      };
   }

   while (<>) {
      chomp;
      $q->enqueue($_);
   }

   $q->end();
   $_->join() for threads->list();
}
like image 371
Henrique Avatar asked Jan 23 '26 03:01

Henrique


1 Answers

This part of the code you've shown reads filenames from the command line arguments, and then reads all the lines in these files. It then iterates over the lines.

   while (<>) { # <--- here
      chomp;
      $q->enqueue($_);
   }

You can replace that with an array, which of course needs a for loop. Make sure to remove the chomp, as there won't be a need to remove newlines.

   foreach (@urls) {
      $q->enqueue($_);
   }
like image 64
simbabque Avatar answered Jan 24 '26 15:01

simbabque



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!