How can I create a directory and fetch a file over FTP into that directory using Perl?

Question

I have a file that looks like this:

ftp://url1/files1.tar.gz dir1
ftp://url2/files2.txt dir2
.... many more...

What I want to do are these steps:

1. Create directory based on column 2
2. Unix 'cd' to that directory
3. Download file with 'wget' based on column1

But how come this approach of mine doesn't work

while(<>) {
  chomp;
  my ($url,$dir) = split(/\t/,$_);
  system("mkdir $dir");
  system("cd $dir");   
  system("wget $url"); # This doesn't get executed
}

What's the right way to do it?

Answer 1

Use native Perl solutions where possible:

• cd can be done with chdir
• mkdir can be done with mkdir
• mkdir -p (don't die if dir exists, recursive creation) can be done with File::Path which comes with Perl
• wget can be done with LWP::Simple

How I would implement this:

use File::Spec::Functions qw(catfile);  # adds a '/' between things (or '\' on Windows)
use LWP::Simple qw(mirror);
use File::Path qw(mkpath);
use File::Basename;
use URI;

while (<>) {
    chomp;
    my ($url, $dir) = split /\t/;
    mkpath($dir);

    # Use the 'filename' of the $url to save 
    my $file = basename(URI->new($url)->path);
    mirror($url, catfile($dir, $file));
}

If you do this, you get:

• Portability between platforms
• Portability between shells
• Perl exception handling (via return values or die)
• Perl input/output (no need to escape anything)
• FLexibility in the future (if you change the way you want to calculate filenames, or how you store the web content, or if you want to run web requests in parallel)