parse file per customized separator / block / metadata

Hello fellow monkers,

I'm trying to solve a simple problem: grep some multiple bigs files having different separator pattern with the same matching rule and output the corresponding block (not only the line). Some sort of grepmail but for any kind of text file.

To be more explicit, the input files could contained multiple blocks like:

case 1:

header1=val1
header2=val2

data
[download]

case 2:

header1: val1
header2: val2

data
[download]

in case 3, like or not previous ones but with a separator line like '^-=+$'

matching rules need to be customized each time. input files could be hundrer with size in Gigabytes. performance needs to be acceptable :)

For now, except manual parsing, the only relevant module that I found is Parse::File::Metadata (1). Has anyone some hints of modules or else to manage this ?

thanks a lot.

Cheers

(1) http://search.cpan.org/~jkeenan/Parse-File-Metadata-0.04/lib/Parse/File/Metadata.pm

Comment on parse file per customized separator / block / metadata Select or Download Code
Re: parse file per customized separator / block / metadata by almut (Canon) on Mar 06, 2010 at 16:43 UTC
Not sure I understand entirely. Could you elaborate on the details, i.e. is there some constant record separator between the multi-line parts you want to extract? Where does the to-be-customized separator occur; is it just the thing between `header` and `val` — in which case you could possibly find a regex that handles all cases in one go while matching against the records. A few more lines of sample input, and some sample search commands might help.	[reply] [d/l] [select]
Re: parse file per customized separator / block / metadata by Utilitarian (Priest) on Mar 06, 2010 at 17:03 UTC
Try using `\Q and \E` to escape the special characters in the split. eg. #!/usr/bin/perl use strict; use warnings; my($first_line,%records,$sep); chomp($first_line =readline (DATA)); seek (DATA,462,0);# use seek(FILEHANDLE,0,0) to return to start of fil +e. print "Please enter the seperator example line follows,:\n\t\"$first_l +ine\"\n"; chomp($sep=<STDIN>); while(<DATA>){ my @record=split/\Q$sep\E/,$_; $records{$record[0]}=$record[1]; } for my $name (sort(keys %records)){ print "$name has the value $records{$name}\n"; } __DATA__ name :-$& value anim :-$& freagra nomme :-$& valeur [download] Works as follows: `~/tmp$ perl tmp.pl Please enter the seperator example line follows,: "name :-$& value" :-$& anim has the value freagra name has the value value nomme has the value valeur` [download] `print "Good ",qw(night morning afternoon evening)[(localtime)[2]/6]," fellow monks."`	[reply] [d/l] [select]
Re: parse file per customized separator / block / metadata by repellent (Priest) on Mar 06, 2010 at 21:51 UTC
Given the lack of information about what you're trying to achieve, seems like Parse::File::Metadata is a good fit so far. Why are you looking for another alternative? Another module that comes to mind is File::Stream, which would help with grepping blocks of data with different separator patterns.	[reply]
Re^2: parse file per customized separator / block / metadata by raiten (Acolyte) on Mar 07, 2010 at 12:12 UTC
Input file could be like `header1=val1 header1b=val1b data1 ================== header2: val2 header2b: val2b data2 =============== header3: val3 header3b: val3b data3 header4= val4 header4b= val4b data4` [download] For these 4 blocks of data, I want to extract the ones matching one or multiple regexp. I could grep them, but I need to reform the data block after, so I'm looking in alternative solutions, module/library or tool. The separator could change in the same file. I quickly check File::Stream (1) and it seems a possible option. about why I search for different solutions, that's a kind of common challenge :). find different views of the problem, differents solutions, more performance, more clean code, more portable and so on ... (1) http://search.cpan.org/~smueller/File-Stream-2.20/lib/File/Stream.pm http://www.justskins.com/forums/file-stream-confusion-80665.html	[reply] [d/l]
Re^3: parse file per customized separator / block / metadata by repellent (Priest) on Mar 07, 2010 at 20:34 UTC
It would really help if there were some `=` equal signs separating `data3` and `header4`. Seems like File::Stream doesn't handle lookaheads that well. Nevertheless, here's an example that may help: use File::Stream; my $lookahead_regex = qr/\w+[=:]/; my ($handler, $stream) = File::Stream->new( DATA, separator => qr/\n=\n$lookahead_regex/, ); my $lookahead = ""; while (my $block = <$stream>) { $block =~ s/($lookahead_regex)$//; $block = $lookahead . $block; $lookahead = $1; print $block; print "-" x 60, "\n"; } __END__ header1=val1 header1b=val1b data1 ================== header2: val2 header2b: val2b data2 =============== header3: val3 header3b: val3b data3 header4= val4 header4b= val4b data4 [download] Output: Read more... (680 Bytes)	[reply] [d/l] [select]
Re^4: parse file per customized separator / block / metadata by raiten (Acolyte) on Mar 25, 2010 at 22:02 UTC
Re^5: parse file per customized separator / block / metadata by repellent (Priest) on Mar 26, 2010 at 05:34 UTC
Re^4: parse file per customized separator / block / metadata by raiten (Acolyte) on Mar 28, 2010 at 14:27 UTC


Clear questions and runnable code get the best and fastest answer
	PerlMonks

parse file per customized separator / block / metadata

How many continents have you visited?