Re: How to improve speed of reading big files

[bv]

You seem to be using pretty structured data (maybe the wrong word here, sorry), yet only one of your regexps is anchored. If you know the position of certain strings (especially /Running|Dump|FromCB|Update/o), you could gain some speedup by anchoring, even if you have to resort to assigning to pos.

Another thing would be to precompile or use the /o modifier on your substitutions, since &filterLog gets called so often.

[korlaz]

Could you explain me the "pos" trick ?

[bv]

From the pos documentation:

pos directly accesses the location used by the regexp engine to store the offset, so assigning to pos will change that offset, and so will also influence the \G zero-width assertion in regular expressions.

So if we know that we have fixed-width data and we only want to match "foo", "ding", or "widget" starting at the 10th character of the string, we can assign to pos($string) to set the "last matched position" to something other than 0, and then use the \G anchor (technically a zero-width assertion, like ^ and $) like so:

while(<DATA>)
{
  pos() = 10;
  print if /\G(?:foo|ding|widget)/o;
}

__DATA__
 3.14 PI  foo
 6.28 2PI ding
   42 LUE answer
    7 LUK superstition
87.32 UNK widget

This code prints lines 1, 2, and 5

[Anonymous Monk]

Its not a trick, its a feature :) Parsing with Regexes and Beyond