Beefy Boxes and Bandwidth Generously Provided by pair Networks
Don't ask to ask, just ask
 
PerlMonks  

Comment on

( #3333=superdoc: print w/ replies, xml ) Need Help??
Illuminatus, yes that is in fact what I am mostly worried about. In fact I think the Mem Usage here is due to one of the threads doing a regex replace, adding backslashes to prepare a file to be imported into mysql.

Edit: Just to clarify, I know that the CPU use should also be high when doing a regex on a large file with many matches, however, the CPU usage of the perl.exe process is always high, even when no regex operations are being performed.

My do_handler function just routes the work to a function in the separate Worker module (I couldn't figure out the syntax to call it directly)

Here's the relevant code from the Worker module:
package File::FileListerWorker; sub main_run { # arguments: 1) job id from Pooler 2) path to scan 3) Path to FileLi +st.exe 4)table name # returns: 1) return 1; #### - START: MAIN EXECUTABLE CODEBLOCK - #### my $jobid = shift; # read scanpaths from the ARGs my $scanpaths_line = shift; # get path to scan my $fl_path = shift; # define path to FileList.exe my $arg_tbl_name = shift; # table to load data into my $dbh = &mysql_conn(); # connect to mysql my $mysql_tablename = &mysql_create_table( $arg_tbl_name, $dbh ); +# create table (IF NOT EXISTS) my @ta = localtime(time); # get time my $time_string = $ta[4] . $ta[3] . ( $ta[5] + 1900 ) . "_" . $ta[2] . $ta[1] . $t +a[0]; # time string my $temp_filename = # make unique filename "\\\\directoryToPlaceTempFileIn\\" . $jobid . "_" . $time_string . ".csv"; &do_scan( $fl_path, $scanpaths_line, $temp_filename ); # call to s +ub that does the scan my $mysql_temp_filename = &fix_file_4mysql($temp_filename); # add +backslashes &mysql_load_data( $mysql_temp_filename, $mysql_tablename, $dbh ); +# mysql import file &cleanup($temp_filename); #delete temp file to preserve space &mysql_disc($dbh); # disconnect mysql #### - END: MAIN EXECUTABLE CODEBLOCK - #### return 1; } sub do_scan { #arguments: 1)location of FileList.exe 2)path to scan 3)output fi +le path #returns: none my $fl_path = shift; my $pth = shift; chomp($pth); my $temp_filename = shift; system(qq{$fl_path $pth >$temp_filename}); }

In reply to Re^2: Multithreaded Script CPU Usage by Zenshai
in thread Multithreaded Script CPU Usage by Zenshai

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • Outside of code tags, you may need to use entities for some characters:
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?
    Username:
    Password:

    What's my password?
    Create A New User
    Chatterbox?
    and the web crawler heard nothing...

    How do I use this? | Other CB clients
    Other Users?
    Others surveying the Monastery: (8)
    As of 2014-12-20 16:16 GMT
    Sections?
    Information?
    Find Nodes?
    Leftovers?
      Voting Booth?

      Is guessing a good strategy for surviving in the IT business?





      Results (96 votes), past polls