Beefy Boxes and Bandwidth Generously Provided by pair Networks
P is for Practical
 
PerlMonks  

Find nonnumeric scalars

by urbs33 (Novice)
on Jan 29, 2015 at 14:59 UTC ( [id://1114920]=perlquestion: print w/replies, xml ) Need Help??

urbs33 has asked for the wisdom of the Perl Monks concerning the following question:

Guys I am working with millions of lines of data that "should" all be in the same format, however I am finding hundreds of lines that are not. A line may have ~10 fields space delimited. Suppose I am splitting the line into scalars that I can work with, and perform math on. Suppose fields 6-8 are supposed to be numeric and available for math. I am getting non-numeric warnings on several of them and just want to write the line out to an "errors" file so that I can resolve the formatting.

How can I do something to test if scalar 6, 7, or 8 is not numeric (or even empty), write the line to a file and move to the next line. I can handle the "else". I'd prefer to use standard perl as it is very difficult at my company to pull in additional packages.

Replies are listed 'Best First'.
Re: Find nonnumeric scalars
by choroba (Cardinal) on Jan 29, 2015 at 15:14 UTC
    Scalar::Util is core since 5.7.3. It features looks_like_number you can use to test whether anything has the format Perl uses for numbers.

    Update: link fixed.

    لսႽ† ᥲᥒ⚪⟊Ⴙᘓᖇ Ꮅᘓᖇ⎱ Ⴙᥲ𝇋ƙᘓᖇ
      looks_like_number you can use to test whether anything has the format Perl uses for numbers

      More specifically, if looks_like_number($x) returns false then perl should issue the "isn't numeric" warning for $x, but if looks_like_number($x) returns true then there should be no "isn't numeric" warning for $x.

      This correlation doesn't hold for recent blead releases (including the current 5.21.8) wrt some 'inf' strings, and perhaps some 'nan' strings too.
      But it should now be restored for 5.21.9.

      That this correlation ought to exist might be no more than *my* opinion, though it seems obvious to me (and matches past behaviour afaik).
      If someone has an example of it not holding in perl-5.20.x, I'd be interested to learn of it.

      Cheers,
      Rob
Re: Find nonnumeric scalars
by BrowserUk (Patriarch) on Jan 29, 2015 at 15:12 UTC

    Use a signal handler?:

    { local $SIG{__WARN__} = sub { print "Bad line: '$_'"; return; }; for( qw[ 1 2 3 d 5 f 6 ] ) { print $_ * 2; } };; 2 4 6 Bad line: 'd' 0 10 Bad line: 'f' 0 12

    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority". I'm with torvalds on this
    In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked
Re: Find nonnumeric scalars
by hotpelmen (Scribe) on Jan 29, 2015 at 16:41 UTC
    Maybe I misunderstand the question but, once you split your line into an array, you can test fields 6, 7 and 8 by doing something like this
    use Scalar::Util (); # as advised by [choroba] my ($line, @arr); open my $fh, "<$ARGV[0]" or die "failed to open file $ARGV[0] for reading: $!\n"; my @check_fields = (6, 7, 8); my $line_counter = 0; LINE: while ($line = <$fh>) { $line_counter++; ### skip line if any of fields 6-8 are not numeric @arr = split /\s+/, $line; foreach my $i ( @check_fields ) { if ( ! Scalar::Util::looks_like_number($arr[$i - 1]) ) { # bad;this checks only for positive integers # (thanks Not_a_Number) # if ($arr[$i - 1] !~ /^\d+$/) { print STDERR "line num $line_counter: " . "1st problem field $arr[$i - 1]: " . "field num $i: " . "$line\n"; next LINE; } } ### the rest of your code }
    This is just to show general approach. Code that does splitting and checking needs to be be extracted into a separate subroutine. With respect to "millions of lines" context of the question, not sure how efficient or inefficient this code is. Caching of test results could be useful, but then caching can take a lot of memory and backfire.

    Update: took into account Not_a_Number's comment. Added comment on "millions of lines" issue.

      Scalars -7, 4.55, and 10e6 are all numeric. Your regex /^\d+$/ tests only for positive integers.

      Update: Minor tpyo

      Update 2: hotpelmen has now completely altered the content of her/his original post, without indicating that (s)he has made any changes (note to hotpelmen: this is considered Bad Form on PM).

        Yeah, you caught me in the middle of my updates. I saved a few times before adding update statement. No preview for updates, that's a problem. Sorry for distress.
        Good point. Fixing.
Re: Find nonnumeric scalars
by AnomalousMonk (Archbishop) on Jan 29, 2015 at 23:01 UTC
Re: Find nonnumeric scalars
by CountZero (Bishop) on Jan 30, 2015 at 10:43 UTC
    Isn't this a good example why one should use perl's taint mode?

    That will (gently) force you to check *all* data that comes from an outside and not fully trusted source.

    Regexp::Common may help you to check for real or integer numbers.

    CountZero

    A program should be light and agile, its subroutines connected like a string of pearls. The spirit and intent of the program should be retained throughout. There should be neither too little or too much, neither needless loops nor useless variables, neither lack of structure nor overwhelming rigidity." - The Tao of Programming, 4.1 - Geoffrey James

    My blog: Imperial Deltronics

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://1114920]
Front-paged by Arunbear
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others avoiding work at the Monastery: (3)
As of 2024-04-19 23:32 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found