http://www.perlmonks.org?node_id=1066927


in reply to Re: Net::FTP::Recursive code not downloading files
in thread Net::FTP::Recursive code not downloading files

Thanks. If I replace the rget call with this:
$ftp->rget( FlattenTree => 1, MatchDirs => qr/PowerEdge (R810|R610|R720|R620 +|M620|M1000E)/, MatchFiles => qr/\.txt$/, );
I get it downloading directories, but it doesn't seem to find any text files, as it doesn't look like it's recursing deep enough. Is there a setting for how deep to go? I get this (snippet shown):
Net::FTP::Recursive=GLOB(0x2355f08)>>> LIST Net::FTP::Recursive=GLOB(0x2355f08)<<< 150 Opening BINARY mode data co +nnection. Net::FTP::Recursive=GLOB(0x2355f08)<<< 226 Transfer complete. drwxrwxrwx 1 owner group 0 Aug 29 2012 Chassis Sys +tem Management drwxrwxrwx 1 owner group 0 Aug 29 2012 Diagnostics -rwxrwxrwx 1 owner group 144462 Aug 29 2012 index.html drwxrwxrwx 1 owner group 0 Sep 25 2012 Legacy drwxrwxrwx 1 owner group 0 Aug 29 2012 Network drwxrwxrwx 1 owner group 0 Aug 29 2012 Rack Soluti +ons drwxrwxrwx 1 owner group 0 Aug 29 2012 SAS Drive drwxrwxrwx 1 owner group 0 Aug 29 2012 SCSI non-RA +ID drwxrwxrwx 1 owner group 0 Aug 29 2012 Serial ATA drwxrwxrwx 1 owner group 0 Aug 29 2012 Systems Man +agement drwxrwxrwx 1 owner group 0 Aug 29 2012 Tape Automa +tion drwxrwxrwx 1 owner group 0 Aug 29 2012 Tape Drives Net::FTP::Recursive=GLOB(0x2355f08)>>> PWD Net::FTP::Recursive=GLOB(0x2355f08)<<< 257 "/Browse_For_Drivers/Server +s, Storage & Networking/PowerEdge/PowerEdge M1000E" is current direct +ory. Returned from rget in /Browse_For_Drivers/Servers, Storage & Networkin +g/PowerEdge. Net::FTP::Recursive=GLOB(0x2355f08)>>> CDUP Net::FTP::Recursive=GLOB(0x2355f08)<<< 250 CDUP command successful. Net::FTP::Recursive=GLOB(0x2355f08)>>> CWD PowerEdge M620 Net::FTP::Recursive=GLOB(0x2355f08)<<< 250 CWD command successful. Calling rget in /Browse_For_Drivers/Servers, Storage & Networking/Powe +rEdge Net::FTP::Recursive=GLOB(0x2355f08)>>> PASV Net::FTP::Recursive=GLOB(0x2355f08)<<< 227 Entering Passive Mode (143, +166,135,12,247,80) Net::FTP::Recursive=GLOB(0x2355f08)>>> LIST Net::FTP::Recursive=GLOB(0x2355f08)<<< 150 Opening BINARY mode data co +nnection. Net::FTP::Recursive=GLOB(0x2355f08)<<< 226 Transfer complete. drwxrwxrwx 1 owner group 0 Aug 29 2012 Application drwxrwxrwx 1 owner group 0 Aug 29 2012 BIOS drwxrwxrwx 1 owner group 0 Aug 29 2012 Chipset drwxrwxrwx 1 owner group 0 Aug 29 2012 Diagnostics drwxrwxrwx 1 owner group 0 Aug 29 2012 Drivers for + OS Deployment drwxrwxrwx 1 owner group 0 Aug 29 2012 Enterprise +Solutions drwxrwxrwx 1 owner group 0 Aug 29 2012 ESM drwxrwxrwx 1 owner group 0 Aug 29 2012 Fibre Chann +el drwxrwxrwx 1 owner group 0 Aug 29 2012 Firmware -rwxrwxrwx 1 owner group 147115 Aug 29 2012 index.html drwxrwxrwx 1 owner group 0 Aug 29 2012 Lifecycle C +ontroller drwxrwxrwx 1 owner group 0 Aug 29 2012 Network drwxrwxrwx 1 owner group 0 Sep 25 2012 PCIe SSS drwxrwxrwx 1 owner group 0 Aug 29 2012 SAS Drive drwxrwxrwx 1 owner group 0 Aug 29 2012 SAS RAID drwxrwxrwx 1 owner group 0 Aug 29 2012 SCSI non-RA +ID drwxrwxrwx 1 owner group 0 Aug 29 2012 Serial ATA drwxrwxrwx 1 owner group 0 Aug 29 2012 Systems Man +agement drwxrwxrwx 1 owner group 0 Aug 29 2012 Video Net::FTP::Recursive=GLOB(0x2355f08)>>> PWD Net::FTP::Recursive=GLOB(0x2355f08)<<< 257 "/Browse_For_Drivers/Server +s, Storage & Networking/PowerEdge/PowerEdge M620" is current director +y. Returned from rget in /Browse_For_Drivers/Servers, Storage & Networkin +g/PowerEdge. Net::FTP::Recursive=GLOB(0x2355f08)>>> CDUP Net::FTP::Recursive=GLOB(0x2355f08)<<< 250 CDUP command successful. Net::FTP::Recursive=GLOB(0x2355f08)>>> CWD PowerEdge R610

-- Burvil

Replies are listed 'Best First'.
Re^3: Net::FTP::Recursive code not downloading files
by tangent (Parson) on Dec 12, 2013 at 23:11 UTC
    As wazat points out the problem is with MatchDirs, though not with path names, as you can see from the output that it is entering the directories you have in MatchDirs. Once it gets in to one of those directories though, it won't go any further because subsequent directories are not in MatchDirs - it checks every directory for a match before recursing, not just the top level.

    The solution is to get rid of the MatchDirs and call your code for each directory, setting the path in $params{dir}.
    e.g. dir => "Browse_For_Drivers/Servers, Storage & Networking/PowerEdge/PowerEdge R810"

Re^3: Net::FTP::Recursive code not downloading files
by wazat (Monk) on Dec 12, 2013 at 22:31 UTC

    I suspect your MatchDirs may be preventing the recursion. If the matching doesn't use the full path then the subdirectory names won't include the parent directory names.

    MatchDirs => qr/PowerEdge R810|R610|R720|R620|M620|M1000E)/,

    If that is the case, then you need to run your code on these 6 directories separately.

      Why would I need to run the code for each directory separately? Isn't that the whole point of recursion, so it will drill down into each directory until there are no more levels? The thing is, I don't know where the text files will be within the directory structure, so I have to traverse every directory and subdirectory.

      Also, on point about the regex. I tried changing it to the following:

      MatchDirs => qr/.*PowerEdge (R810|R610|R720|R620|M620|M1000E).*/,
      but I get the same thing.

      -- Burvil

        Net::FTP::Recursive won't descend into to subdirectory unless the name of the subdirectory matches your pattern. For example, if there is a directory

        PowerEdge R810/TextFiles

        Net::FTP::Recursive will descend into "PowerEdge R810" since it matches. However the subdirectory "TextFiles" does not match your pattern, so Net::FTP::Recursive will ignore it. Net::FTP::Recursive dos not consider a child directory to match simply because a parent did.

        What I was suggesting, was that if you happen to know the where the various "PowerEdge" directories are, you could search each of them separately, omitting the MatchDirs option, thus circumventing the limitations of MatchDirs

        Why would I need to run the code for each directory separately? Isn't that the whole point of recursion, so it will drill down into each directory until there are no more levels?
        MatchDirs is not doing what you think - it is not saying "if a directory matches this pattern go in and recurse". What it is saying is "for every directory do not enter unless its name matches this pattern". Changing the pattern will not help as it tests against the directory name not the path.

        Peeking into the Module's relevant code may help:

        elsif ( $file->is_directory() ) { if( ( $options{MatchDirs} and $filename !~ $options{MatchDirs} ) or ( $options{OmitDirs} and $filename =~ $options{OmitDirs} )){ next FILE; }