http://www.perlmonks.org?node_id=997645

RedTussock has asked for the wisdom of the Perl Monks concerning the following question:

I have been hand a number of txt files from a datalogger, (which was logging water flow) like this

1=11-07-22,13:59:08 2=+1.304556E+01m3/h 3=+2.458268E-01m/s 4=+0057505E+1m3 5=+0057591E+1m3 6=-0000085E+1m3 1=11-07-22,14:14:05 2=+1.224623E+01m3/h 3=+2.310640E-01m/s 4=+0057505E+1m3 5=+0057591E+1m3 6=-0000085E+1m3 1=11-07-22,14:29:02 2=+1.338452E+01m3/h 3=+2.522141E-01m/s 4=+0057505E+1m3 5=+0057591E+1m3 6=-0000085E+1m3 1=11-07-22,14:43:58 2=+1.195594E+01m3/h 3=+2.252944E-01m/s 4=+0057506E+1m3 5=+0057592E+1m3 6=-0000085E+1m3 1=11-07-22,14:58:55 2=+1.185474E+01m3/h 3=+2.233874E-01m/s 4=+0057506E+1m3 5=+0057592E+1m3 6=-0000085E+1m3 1=11-07-22,15:13:52 2=+1.014633E+01m3/h 3=+1.911945E-01m/s 4=+0057506E+1m3 5=+0057592E+1m3 6=-0000085E+1m3 1=11-07-22,15:28:48 2=+1.131358E+01m3/h 3=+2.131899E-01m/s 4=+0057506E+1m3 5=+0057592E+1m3 6=-0000085E+1m3
A continuous repetitive pattern of 6 lines. IS there a way of parsing the 6 line pattern into 1 csv file ... or just data field format ... which could be parsed or handed to a dat daemon. I.e. point the script at a folder with say 20 files like this in it, and have it process each file into usable field like data, which could be loaded into a database. Clearly some of the 6 lines have propriety numbers perhaps in scientific format in them, but at the moment I would love to get a dump into some sort of usable data field format.

Replies are listed 'Best First'.
Re: Parsing Data logger files ...
by choroba (Cardinal) on Oct 06, 2012 at 22:31 UTC
    Do you mean something like this?
    #!/usr/bin/perl use warnings; use strict; $/ = "\n1="; while (<>) { my @fields = split /\n/; s/^[1-6]=// for @fields; print +(join ',', @fields[0 .. 5]), "\n"; }
    لսႽ† ᥲᥒ⚪⟊Ⴙᘓᖇ Ꮅᘓᖇ⎱ Ⴙᥲ𝇋ƙᘓᖇ
      Sorry the previous comment was a bit naff ... where I am at is
      #!/usr/bin/perl use warnings; use strict; opendir DH, "/a/b/c/perl/dl_parser/down/" or die "Couldn't open the d +irectory +: $!"; while ($_ = readdir (DH)) { $/ = "\n1="; my @fields = split /\n/; s/^[1-6]=// for @fields; print +(join ',', @fields[0 .. 5]), "\n"; } exit
      which gets me a list of files and 5 other lines ... :-) like
      101130.TXT,,,,, Use of uninitialized value in join or string at lineJumper.pl line 11. Use of uninitialized value in join or string at lineJumper.pl line 11. Use of uninitialized value in join or string at lineJumper.pl line 11. Use of uninitialized value in join or string at lineJumper.pl line 11. Use of uninitialized value in join or string at lineJumper.pl line 11. 240429.TXT,,,,, Use of uninitialized value in join or string at lineJumper.pl line 11. Use of uninitialized value in join or string at lineJumper.pl line 11. Use of uninitialized value in join or string at lineJumper.pl line 11. Use of uninitialized value in join or string at lineJumper.pl line 11. Use of uninitialized value in join or string at lineJumper.pl line 11. 110208.TXT,,,,, Use of uninitialized value in join or string at lineJumper.pl line 11. Use of uninitialized value in join or string at lineJumper.pl line 11. Use of uninitialized value in join or string at lineJumper.pl line 11. Use of uninitialized value in join or string at lineJumper.pl line 11. Use of uninitialized value in join or string at lineJumper.pl line 11. 081221.TXT,,,,, Use of uninitialized value in join or string at lineJumper.pl line 11. Use of uninitialized value in join or string at lineJumper.pl line 11. Use of uninitialized value in join or string at lineJumper.pl line 11. Use of uninitialized value in join or string at lineJumper.pl line 11. Use of uninitialized value in join or string at lineJumper.pl line 11. 100807.TXT,,,,, Use of uninitialized value in join or string at lineJumper.pl line 11. Use of uninitialized value in join or string at lineJumper.pl line 11. Use of uninitialized value in join or string at lineJumper.pl line 11. Use of uninitialized value in join or string at lineJumper.pl line 11. Use of uninitialized value in join or string at lineJumper.pl line 11. 081219.TXT,,,,,
      So getting close ... need to read each file ... maybe split into modules one that reads the file names from the directory, and one that process's them?. .. then build some interface to ask for the directory and maybe a file name to print the records too.

        The simplest solution would be to call choroba's script as "munge.pl /some/where/*" and let the shell do the directory stuff. "<>" will automatically try to read everything in @ARGV so you don't have to deal with it yourself.

        Otherwise you're on the right track with readdir but you forgot to actually open each individual file.

      Awesome thanks ... And the open the folder and process all files in it would look something like ..
      opendir XX, "/somefile/somewhere" or die "Couldn't open the directory +: $!"; while ($_ = readdir (XX)) { $/ = "\n1="; while (<>) { my @fields = split /\n/; s/^[1-6]=// for @fields; print +(join ',', @fields[0 .. 5]), "\n"; }
      Sorry my perl scripting is newb in the extreme ... :-)
Re: Parsing Data logger files ...
by BrowserUk (Patriarch) on Oct 07, 2012 at 07:37 UTC

    A one-liner

    C:\test>perl -nE"say join',',map m[^..(\S+)], $_, map scalar <>, 1 .. +5" junk.dat 11-07-22,13:59:08,+1.304556E+01m3/h,+2.458268E-01m/s,+0057505E+1m3,+00 +57591E+1m3,-0000085E+1m3 11-07-22,14:14:05,+1.224623E+01m3/h,+2.310640E-01m/s,+0057505E+1m3,+00 +57591E+1m3,-0000085E+1m3 11-07-22,14:29:02,+1.338452E+01m3/h,+2.522141E-01m/s,+0057505E+1m3,+00 +57591E+1m3,-0000085E+1m3 11-07-22,14:43:58,+1.195594E+01m3/h,+2.252944E-01m/s,+0057506E+1m3,+00 +57592E+1m3,-0000085E+1m3 11-07-22,14:58:55,+1.185474E+01m3/h,+2.233874E-01m/s,+0057506E+1m3,+00 +57592E+1m3,-0000085E+1m3 11-07-22,15:13:52,+1.014633E+01m3/h,+1.911945E-01m/s,+0057506E+1m3,+00 +57592E+1m3,-0000085E+1m3 11-07-22,15:28:48,+1.131358E+01m3/h,+2.131899E-01m/s,+0057506E+1m3,+00 +57592E+1m3,-0000085E+1m3

    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

    RIP Neil Armstrong