Beefy Boxes and Bandwidth Generously Provided by pair Networks
XP is just a number
 
PerlMonks  

Re: How Could I Speed Up An Archive Script?

by arden (Curate)
on Jun 22, 2004 at 16:09 UTC ( #368773=note: print w/ replies, xml ) Need Help??


in reply to How Could I Speed Up An Archive Script?

new_monk, a few different things come to mind with your script.

Since each $schema is from a different server, why not perform each of the FTP downloads at the same time? I doubt you're using up all of your machine's bandwidth for a single FTP, you might save some time by performing multiple FTPs at the same time. Also, one of the longer FTP sessions could still be downloading (relatively light on the CPU) while another $schema has moved on to the compression stage (heavy on CPU, light on disk).

Before you do the $ftp->get on the files, you have an if statement that is going to be TRUE for everything. You have if($file1 !~ /.stderr/ || $file1 !~ /.log/) which says "if NOT .stderr OR if NOT .log"; well the file isn't likely to be BOTH, so it'll return true every time. I doubt that's what you really wanted to happen.

After you've compressed your files, you write the resultant .zip file out to the local harddrive, then copy it to another drive based on $schema twice, then you delete the .zip file. Why not just write it to where you want it with  $zip->writeToFileNamed( "G:/some folder/$schema/$logname ); instead? Then you can copy it to the second location.

I don't think you're going to speed up the compression much. You might wish to include a few more print statements so you know where in your script the process is. Statements like  print "Starting FTP $host\n"; and  print "Finished FTP, starting compression on $schema\n; would help. Maybe even a  print "writing compressed file $logname to disk\n; too. It'll help you to figure out where your bottleneck is at.

- - arden.
p.s. please include <READMORE> tags on large sections of code. It really matters if your node gets front-paged!


Comment on Re: How Could I Speed Up An Archive Script?
Select or Download Code
Re^2: How Could I Speed Up An Archive Script?
by Roy Johnson (Monsignor) on Jun 22, 2004 at 17:08 UTC
    Also note that the dot in the regexes is not a literal, but matches any non-newline.

    We're not really tightening our belts, it just feels that way because we're getting fatter.
Re^2: How Could I Speed Up An Archive Script?
by new_monk (Sexton) on Jun 22, 2004 at 18:51 UTC
    Thank you for your assistance oh wise ones.

    I thought "!~" is the same as "=~,"? I have debugged my code before, the bottelneck is definitely where the file is getting added to the zip file. Getting the files from the server takes but just a few seconds.

    This is not going to speed things up but how do I spawn threads in Perl?

    On the copying files, I like to create it locally first then copy it to my backup location and to the network sharedrive. I do this in case the fileserver is down, I will have a copy of it locally.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://368773]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others cooling their heels in the Monastery: (12)
As of 2014-12-22 20:58 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    Is guessing a good strategy for surviving in the IT business?





    Results (130 votes), past polls