Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot
 
PerlMonks  

Remove RTF Formatting

by GotToBTru (Chaplain)
on Feb 22, 2011 at 19:38 UTC ( #889666=CUFP: print w/ replies, xml ) Need Help??

We have clients sending us (in many ways) documents (in many formats) which we process automatically instead of having somebody key the data in. Applications like this are basically why my group exists at work.

Latest challenge: file is in Rich Text Format. I don't need to parse it, or convert it into HTML. I just want the data, absent any formatting.

Here is my end result. The regexes could no doubt be tightened up.

use strict; use warnings; my ($wholefile,$line); my $outfilename='orderdata.out'; {open INFILE, '<', "orderform.rtf"; local $/; $wholefile=<INFILE>; close(INFILE); $wholefile =~ s/^\s*{.*\n//g; # remove lines starting wi +th braces $wholefile =~ s/\\[\w-]+\b//g; # remove RTF commands $wholefile =~ s/(\n)\s+(\S+)/$1$2/g; # remove extra spaces $wholefile =~ s/([^\\\n]+)\\(\n)/$1$2/g; # get rid of \ at the ends + of lines $wholefile =~ s/\\\n//g; # get rid of lines with a +\ only open OUTFILE,'>',$outfilename or die "Can't open $outfilename!\n"; print OUTFILE $wholefile; close(OUTFILE); }
Update: applied suggestions to regexes #1, #2

Comment on Remove RTF Formatting
Download Code
Re: Remove RTF Formatting
by jdporter (Canon) on Feb 23, 2011 at 14:51 UTC

    Too much overhead!

    s/{.*\n//g; # remove lines starting with braces s/(\\[\w-]+)\b//g; # remove RTF commands s/(\n)\s+(\S+)/$1$2/g; # remove extra spaces s/([^\\\n]+)\\(\n)/$1$2/g; # get rid of \ at the ends of lines s/\\\n//g; # get rid of lines with a \ only

    Then execute with:

    perl -0777 -p remove_rtf_formatting.pl < infile.rtf > outfile.txt
    I reckon we are the only monastery ever to have a dungeon stuffed with 16,000 zombies.
Re: Remove RTF Formatting
by TomDLux (Vicar) on Feb 28, 2011 at 23:35 UTC

    If it's going to remain under 20 lines, I have no problem with using the default $_.

    But I've never known a problem to not develop exceptions, so you wind up with dozens or hundreds of lines of code. If it gets any more complicated, you should split code into subroutines, especially if some of the processing is conditional. In any case, is the verbose form slower than the default form?

    Your first regex comment says remove lines starting with braces, but you don't enforce the starting part. I would personally process the file on a line-by-line basis, after all, you might get gigabyte submissions, and so would stop looking at a line as soon as I detected the curly brace. Note that \A is the beginning of the string, \z is the end of the string.

    LINE: while (my $line = <$INFILE> ) { chomp next LINE if $line =~ m<\A\s*{>; # # or if it was going to be literally # the first char, with no spaces ... # next LINE if q<{> eq substr $line, 0, 1; # # other processing # $text .= $line . $NEWLINE; }

    Lines with only a slash are easy:

    next LINE if length $line == 1 and q{\} eq substr $line, 0, 1;

    In #3, when removing leading spaces ( You don't care about spaces within the line, it seems), instead of copying the newline and the stuff after the space, why not use Look-Around Assertions to specify you only want to remove spaces that come after a newline. No need to say anything about what follows, it will grab all spaces and tabs.

    s/(?<=\n)\s+//; #or, line by line: s/\A\s+//

    Similarly in #4, don't copy the line stuff, just drop trailing spaces

    s/\s+\n//;

    I would predefined a constant, or at least a variable, for the double backslash and newline.

    Readonly my $ESC_BACKSLASH => q{\\}; Readonly my $NEWLINE => q{\n}; $wholefile =~ s/$ESC_BACKSLASH\z//o;

    In #2, don't capture the RTF command you want to discard, it just slows things down. AS for the word boundary \b, it isn't really doing anything useful, because greediness will make the hyphenated word slurp up everything it can.

    Readonly my $RTF_CMD => qr/$ESC_BACKSLASH [\w-]+/xo; s/$RTF_CMD//;

    As Occam said: Entia non sunt multiplicanda praeter necessitatem.

      Good stuff here, thanks for taking the time. I developed these iteratively, and it shows. The parentheses on #2 had a purpose in an earlier version, and I didn't think to remove them when I stopped needing them.

      As I initially understood the project, I needed to strip all the formatting out of the file. In the actual application, I do end up processing line by line because all the data I need is in the first third of the file (these are only 200-300 lines long), and I actually process less than a dozen of the lines I do read. The real benefit of this project has been, as usual, my education.

      if length $line == 1 and q{\} eq substr $line, 0, 1;

      How in the world is that better than

      if $line =~ /^\\$/;

      ?????

      I reckon we are the only monastery ever to have a dungeon stuffed with 16,000 zombies.
      Quoting is tricky.
      next LINE if length $line == 1 and q{\} eq substr $line, 0, 1;
      That's a compile error. I think you want q{\\}
      Readonly my $NEWLINE => q{\n};
      use Readonly; { # I don't think you want this Readonly my $NEWLINE => q{\n}; print ">>>$NEWLINE<<<\n"; } { # Do you really want this? Readonly my $NEWLINE => qq{\n}; print ">>>$NEWLINE<<<\n"; } __END__ >>>\n<<< >>> <<<

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: CUFP [id://889666]
Approved by ww
Front-paged by Arunbear
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chanting in the Monastery: (11)
As of 2014-07-31 12:40 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite superfluous repetitious redundant duplicative phrase is:









    Results (248 votes), past polls