Re^2: MCE: How to access variables globally

[biohisham]

Kia ora!

Yes, this is the crux of it but I avoided gathering from mce_loop_f in this way

%hash2 = mce_loop_f {
    MCE->gather(split ' ', $$_);
} $data_file;

Because (and I can be totally wrong in assuming so) gather will only return %hash2 in this instance while I am also interested in returning $counter2. The docs show gather can be called multiple times, doing so will complicate teasing apart the returned output from mce_loop_f{}. It would be great if gather can behave a bit like a sub so WYSIWYG

#hypothetical code:
#alas if gather can gather two or more data types
(%hash2, $counter2) = mce_loop_f {
    my $internal_counter2++;
    MCE->gather(split '\s', $$_, $internal_counter2); #return data typ
+es
} $DATA_F;

[kcott]

If all you need is a count of the number of iterations, and the keys are unique, you can use something as simple as:

$ perl -E 'my %x = (a=>1, b=>2, c=>3); say "Count: ", 0+keys(%x)'
Count: 3

If the situation is more complex than that — non-unique keys; lines skipped for some reason; and so on — you'll need to provide more information or I'm only guessing (and I don't really want to waste time doing that). You should show some sample input: keep it short but still realistic with example exception cases. Then show the expected output from that input.

If duplicate keys are encountered, should they be skipped or should their value overwrite the previous value. Other reasons that lines might be skipped are: they're blank, are comments, don't match /^\S+\s+\S+$/, or something else. What else is special that I should know about?

When I saw your OP code, I thought the first (non-MCE) loop, and the two counters, were just for testing. Clearly, that was a poor guess; please help me out here.

[biohisham]

Thanks a lot Ken for your help; your guess is on-spot. The keys reflect States that do repeat in different lines/cols and associated with them prob and errorProb values against a time stamp. So the idea is to turn the states into columns of state name\tProb[state]\t Error[state]and align them against the time. The original file is generated by a mathematical modelling (Boolean modelling of networks) software and requires changing its format to be able to do further analysis. Its a large file hence I split it and I am reading it in chunks to avoid the OOM killer.

The header looks like this

Time    TH    ErrorTH    H    HD=0    State    Proba    ErrorProba    
+State    Proba    ErrorProba    State    State    Proba    ErrorProba

Here is how my minimum working example looks like

Read more... (57 kB)

[marioroy]

Greetings, biohisham,

Gathering nested data is possible with a gather callback. MCE::Relay is used to relay the index line counter. I commented out the section regarding past index line. Starting from 0 each time made me wonder. I changed it to relay index line.

Please note that $ret_FullGlobal, $ret_ProbState, and $ret_ErrProbState hashes are HoH-H and HoH. Important! Remember to "not" gather HoH->A and HoA structures containing "only" high indexes. Perl is likely to reserve memory for the empty array slots. For your use case, HoH-H and HoH are your friends with regards to gathering nested structure. The gather callback can then append/populate the global HoH->A and HoA structures.

Read more... (6 kB)

[kcott]

"Thanks a lot Ken for your help; your guess is on-spot."

OK, that's great.

Now that I see you're working with CSV data, one further piece of advice: take a look at Text::CSV (and note that if you also have Text::CSV_XS installed, it will run faster).