Re^2: UTF-8 text files with Byte Order Mark

[muba]

in reply to Re: UTF-8 text files with Byte Order Mark
in thread UTF-8 text files with Byte Order Mark

Yeah, this works, except that the BOM indeed is a three-bytes thing as said above. So the code, that seems to work, now looks like this:

while (my $line = <$rulesFH>) {

    if ($. == 1) {
        # Remove Byte Order Mark if it's there
        use Encode;
        my $octets = encode("utf8", $line);
        $octets =~ s/^\x{ef}\x{bb}\x{bf}//;
        $line = decode("utf8", $octets);
    }

    # rest...
}

[ikegami]

        my $octets = encode("utf8", $line);
        $octets =~ s/^\x{ef}\x{bb}\x{bf}//;
        $line = decode("utf8", $octets);

is the same thing as

        my $BOM = decode("utf8", "\x{ef}\x{bb}\x{bf}");
        $line =~ s/^$BOM//;

is the same thing as

        my $BOM = chr(0xFEFF);
        $line =~ s/^$BOM//;

is the same thing as

        $line =~ s/^\x{FEFF}//;

which is what I gave you. Much simpler!

[muba]

Meh. Indeed, I didn't realise that. Thank you!

[Anonymous Monk]

Thank you!!! This saved me a lot of trouble. I am also trying to strip out these UTF-8 byte order mark characters, which google docs puts in by default to downloaded text files. By the way I found that \x{FEFF} was not the same as \x{ef}\x{bb}\x{bf}

[ikegami]

By the way I found that \x{FEFF} was not the same as \x{ef}\x{bb}\x{bf}

Yeah, "\x{ef}\x{bb}\x{bf}" is the UTF-8 encoding of the BOM / U+FEFF / "\x{FEFF}".