Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot
 
PerlMonks  

Reading progress of "copy" executed asynchronously

by neWerminder (Novice)
on Aug 08, 2018 at 09:51 UTC ( #1220067=perlquestion: print w/replies, xml ) Need Help??
neWerminder has asked for the wisdom of the Perl Monks concerning the following question:

Hello, i have quite a problem to manage and i cant really manage to do so myself so im seeking for help here. What i need to do is to get % of file copied somehow displayed (or captured to variable even better). So on windows im using cmd's "copy" command. copy s.txt b.txt /z Displays nice percentages of progress. Now.. i could add >output.txt at the end to get that percentages to file.. but thats quite a spam of them and thats not what i want to achieve. Generally i tried with backticks and with open command and i cant really get it to work. In best case i received 100% at the end of copy but i dont know how to execute copy and read how much % there is. Any help with making it happening would be greatly appreciated.

Replies are listed 'Best First'.
Re: Reading progress of "copy" executed asynchronously
by afoken (Abbot) on Aug 08, 2018 at 17:01 UTC

    I would avoid running copy at all. Copying a file isn't that hard. File::Copy does almost all that you need. All that's missing is a callback hook for progress, and adding that to a copy of File::Copy::copy() should be easy, especially if you remove syscopy support.

    Alexander

    --
    Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)
Re: Reading progress of "copy" executed asynchronously
by zentara (Archbishop) on Aug 08, 2018 at 10:39 UTC
    Hi, have you considered trying a pipe viewer? I use linux exclusively, but I'm sure Windows has one. I would try running the whole piped command and capturing it's STDOUT. You might have to regex the STDOUT for %. Possibly IO::Capture::Stdout would help.

    I'm not really a human, but I play one on earth. ..... an animated JAPH
Re: Reading progress of "copy" executed asynchronously
by Corion (Pope) on Aug 08, 2018 at 10:43 UTC

    A funny way to cheat would be to simply generate the progress in your program by checking the size of the target file and comparing that to the size of the source file. This avoids needing to read the (console) output of the copy process, but you don't necessarily get any indication of errors that way.

      Windows shows all the disk space as used immediately, and doesn't even update the file timestamps during the copy process. It does update the modification time to be equal to the source file's modification time after the copy completes, but that's not useful. The free space on disk is also set to the final value immediately.

Re: Reading progress of "copy" executed asynchronously
by hippo (Canon) on Aug 08, 2018 at 09:57 UTC

    Have you looked at Term::ProgressBar? That's probably where I would start. It seems to cover most of the bases.

    PS. For clarity, I've assumed here that you would actually prefer not to shell out to do the copy.

Re: Reading progress of "copy" executed asynchronously
by QM (Parson) on Aug 08, 2018 at 09:56 UTC
    In *nix, I'd use tee. In Windows, I think cmd might help.

    -QM
    --
    Quantum Mechanics: The dreams stuff is made of

Re: Reading progress of "copy" executed asynchronously
by bliako (Hermit) on Aug 08, 2018 at 14:23 UTC
    use strict; use warnings; my $cmd = 'copy.exe'; my @args = ('from', 'to', '/z'); open(EXE, '-|', $cmd, @args) || die "$!"; while(<EXE>){ print "read from program output: ".$_."\n"; } print "done.\n"; close(EXE);

    (Update1) Alternatively, if in unix/bash, create a script which forks a copy process and a filestat process. When the copy process finishes, the filestat process is killed. You can use the above perl script to run it.

    #!/bin/bash FROM=$1 TO=$2 cp "${FROM}" "${TO}" & wPID1=$! echo "$0 : copy-process, pid is $wPID1" (for((;;)); do stat -c '%s' "${TO}"; sleep 1; done) & wPID2=$! echo "$0 : filestat-process, pid is $wPID2" wait ${wPID1} # wait for copy-process to finish echo "$0 : copy-process is finished" kill ${wPID2} echo "$0 : filestat-process killed"

    You could also use the same logic of the bash script in Perl.

      hey, none of the above works on Windows unfortunately. First option is out due to -| which apparently is not supported on windows. I receive "List form of pipe open not implemented". Second option wont work because windows for some reason shows that size of file "to" is same as size of file "from" in the moment copying started.. ;/ at least in any of methods i know.

        I was not aware Windows reports incorrect filesizes. Then you have to rely on copy's reporting. Use this pipe-open form instead:

        use strict; use warnings; my $cmd = 'copy.exe 'from', 'to', '/z' |'; open(EXE, $cmd) || die "$!"; while(<EXE>){ print "read from program output: ".$_."\n"; } print "done.\n"; close(EXE);

        Or use IPC::Run or any other IPC::* module mentioned here or there.

Re: Reading progress of "copy" executed asynchronously
by bliako (Hermit) on Aug 08, 2018 at 21:59 UTC

    Fork a child to filestat (unix stat is required or windows equivalent) while parent copies on...:

    #!/usr/bin/env perl use strict; use warnings; # assumes file src is 'aaa' and dest is 'bbb' my $cmd1 = 'cp'; # path to OS filecopy command my @args1 = ('aaa', 'bbb'); my $cmd2 = 'stat'; # path to OS file-stat command my @args2 = qw/-c '%s' bbb/; # unix stat params to print size of targe +t file 'bbb' my $pid = fork(); die "fork! $!" if not defined $pid; if( $pid > 0 ){ # child my $keepon = 1; $SIG{TERM} = sub { $keepon = 0; print "sigterm received\n"; }; sleep 2; # let the copy begin first ... while($keepon){ open(EXE2, '-|', $cmd2, @args2) or die "$!"; my $sta = ''; while(<EXE2>){ $sta .= $_ } print "$0 : file size now is $sta\n"; close(EXE2); sleep 1; } print "child out of loop and dead...\n"; } else { # parent print "$0 : starting copy ...\n"; system($cmd1, @args1) == 0 or die "system failed $cmd1, $!"; print "parent exiting\n"; kill -15, $pid; # kill the child :( }
Re: Reading progress of "copy" executed asynchronously
by karlgoethebier (Monsignor) on Aug 08, 2018 at 12:26 UTC

    Probably a totally useless guess: See File::Tee and IPC::Run. Sorry for typos - written on my iPad in a hurry 😎

    Regards, Karl

    «The Crux of the Biscuit is the Apostrophe»

    perl -MCrypt::CBC -E 'say Crypt::CBC->new(-key=>'kgb',-cipher=>"Blowfish")->decrypt_hex($ENV{KARL});'Help

Re: Reading progress of "copy" executed asynchronously
by neWerminder (Novice) on Aug 08, 2018 at 10:24 UTC
    Hello,

    i do not need progress bar so i have not checked it at all - numeric representation of progress is just fine, well even better. As for tee/cmd - i believe executing copy x to y is actually executing cmd - or i do not know something?

    Generally i need to copy a file which often is pretty big - can be over 1TB. So i need a way to execute copy and yet be able to read and write somewhere its progress. That write is quite crucial - it cant be just displayed. I need to be able to process it further on while copying is still occuring.

Re: Reading progress of "copy" executed asynchronously
by neWerminder (Novice) on Aug 08, 2018 at 10:51 UTC
    Hi,

    @Zentara i managed to capture stdout via open but i cant capture it asynchronously. I get output from STDOUT after file is copied, i need to get it while its copied and process it further on.

    @Corion - that is actually nice hint but wont work (tested ;/) at least while copying file localy (might need to test through network). For some weird reason it shows size and size on disk of copied file as if it would be already fully copied ;o

      Hi, I don't know how it is done on Windows, but I suspect you need to turn off buffering of STDOUT to get immediate readout. Try using $|=1 to turn off STDOUT buffering. Also see Is print faster than syswrite? and Re^3: explanation for sysread and syswrite for examples on using syswrite and sysread. If you write your own copying program with syswrite and turn stdout buffering off, you should be able to monitor your instantaneous progress. Look at this example from the perlmonks archives:
      #!/usr/bin/perl -w use strict; use IPC::Open2; use IO::Select; $| = 1; # autoflush STDOUT # Declare filehandles and command to use: my ($r, $w); my $cmd = 'ping 127.0.0.1'; # Open the process and set the selector: my $pid = open2($r, $w, $cmd); my $selector = IO::Select->new($r); while (1) # infinite loop (use "last" to break out) { if ($selector->can_read(0)) { my $chars; my $bytes_read = sysread($r, $chars, 4046); print "$bytes_read $chars\n"; } # Do anything you want in between reads here... } __END__ The advantage to this script is that, if your commands (like "whois", "dig", and "ping") happen to pause, the loop won't automatically break out. The disadvantage to this script is that it might be difficult figuring out when a command has finished, or just has delayed output (in which case you might have to put in a few sleep() calls). Either way, I think that this script here does a better job of helping you visualize what is going on -- you just need to be mindful of the fact that some programs don't flush their output right away, and that it's not a simple matter to tell if the program has stopped running altogether.

      I'm not really a human, but I play one on earth. ..... an animated JAPH
Re: Reading progress of "copy" executed asynchronously
by neWerminder (Novice) on Aug 09, 2018 at 03:59 UTC
    Thank you all for quick replies. Whenever i get a minute to sit back to it ill try your suggestions and get back to you. Thank you all again! :)

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1220067]
Approved by marto
Front-paged by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others pondering the Monastery: (4)
As of 2018-08-18 06:49 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Asked to put a square peg in a round hole, I would:









    Results (185 votes). Check out past polls.

    Notices?