Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid
 
PerlMonks  

comment on

( [id://3333]=superdoc: print w/replies, xml ) Need Help??
Hi

I will only sketch an algorithm and leave the programming to you.

I think you should read and process text chunks of size n, e.g. 1024 or 4096 bytes. ²

Whenever you process one chunk you need to append the m first bytes of the next chunk with m=200+l and l the number of characters of your keyword string minus 1, that is 21 for "these are my keywords".

Like this your regex will match all occurrences where at least the first character of the keyword string is still in the chunk.

Of course you need to normalize the chunks and keywords by replacing s/s+/ /g

If your regex is too complicated to be normalized you can still do it by joining two - reasonably big (!)³ successive chunks, but you need either to memorize the match position to exclude duplicated hits or change the regex to only allow matches starting within the first chunk. (e.g. by checking pos)

Cheers Rolf

1) now you could even use index instead of a regex

2) here efficiency depends on the block size of your filesystem. see seek for how to read chunks.

3) a chunk must be bigger than the size of the longest possible match. Now quantifiers like \s+ indicate potentially infinite long matches. Are they really wanted??? Either make a reasonable limit like \s{,20} or you have to normalize your chunks by replacing s/\s+/ /g.


In reply to Re: use regular expressions across multiple lines from a very large input file by LanX
in thread use regular expressions across multiple lines from a very large input file by rizzy

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Are you posting in the right place? Check out Where do I post X? to know for sure.
  • Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
    <code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
  • Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
  • Want more info? How to link or How to display code and escape characters are good places to start.
Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others imbibing at the Monastery: (6)
As of 2024-04-23 14:13 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found