I am only able to guess on your problem, as you gave very little detail. Do you want to keep lines that only use ASCII? Do you want to keep lines that are not UTF-8 but are valid Latin-1 or ISO-2022-JP or some other encoding?

If it really is a matter of ASCII or non-ASCII UTF-8, just reject a line if it includes any character above chr(127). Other encodings will present a bit more challenge.

Beware of what you ask for. The following script removes every line that only contains valid UTF-8.

#!/usr/bin/perl
use strict;
use warnings;
use Encode qw( decode );
while (<>) {
   print if !eval { decode('UTF-8', $_, Encode::FB_CROAK); 1 };
}
[download]

Usage:

• From a file: remove_utf8_lines.pl infile > outfile
• From STDIN: remove_utf8_lines.pl < infile > outfile
• In-place: perl -i.bak remove_utf8_lines.pl file

A better solution might be to convert the lines to another encoding.

#!/usr/bin/perl
use strict;
use warnings;
binmode(STDIN,  ':encoding(UTF-8)');
binmode(STDOUT, ':encoding(iso-latin-1)');
print while <>;
[download]

Same usage as the original program.

So, while you sometimes can determine that a line *isn't* UTF-8 (because not every byte sequence is valid UTF-8), you can never be sure a byte sequence is UTF-8 without additional information.

True. So tell me: why on earth does the Unicode standard recommend against putting a BOM at the start of a UTF-8 file? Those guys must really like ambiguous data and the quandary it creates for software developers.