Re^3: XML::Twig and threads

[remiah]

I understand your situation at last.
So, copying original and reuse it will be like this.

my $t= XML::Twig->new();
$t->parsefile($inputFile);
my $someData =$t->root->first_child; #someData

for my $i(1 .. $loop) {
    for ( $someData->children_copy( 'managedObject') ){
        handle_managedObject($t, $_);
    }

    print "Iteracja: $i / $loop \t->  OK\n";
    $bID++;
    $someID = 0;
}

It becomes slower than original with my machine. Because deep coping Elt object takes too much time for large XML. I wonder is this same at your environment? Or maybe you already know this ...

As BrowserUK says, use strict and warning please.

[BrowserUk]

I think the real problem is even worse in that with the real, 10MB & 100MB XML files, he is moving his process into swapping hence everything slows to a crawl.

[remiah]

Hello, BrowserUK.

I see. Example XML is just 613KB.

Copying Twig object is terribly slow. I guess Data::Dumper or dclone of storable will not do any good, because it is just huge.

    for ( $someData->children_copy( 'managedObject') ){
        handle_managedObject($t, $_);
    }

Without copy, it is very fast.

    for ( $someData->children( 'managedObject') ){
        handle_managedObject($t, $_);
    }

So, I vaguely imagined rewriting managedObject sub using regex, for example ...

my ($t, $element)=@_;

# create rewrite rules using Twig 
my %rewrite_rules =(
    q/name="name"/ => "some value",
);


#replace with regex
my $buffer=$element->sprint; #get plain text of element
for (keys %rewrite_rules){
    $buffer =~ s/ $_  (.*?)  >  .*?  (?=<)
                /${_} ${1} $rewrite_rules{$_}/sx;
}

#just print out without changing $element
print $fh $buffer;

I will do like this, if I were.

Regards and thanks for your response.

[BrowserUk]

I vaguely imagined rewriting managedObject sub using regex,

Given the size of the OPs target files -- 100MB -- I'd drop the use of XML parser entirely and process the file as text.

That statement is tantamount to blasphemy around here, but it would be quicker, simpler and would work. And nobody would think twice if the files weren't labeled "XML".