Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic
 
PerlMonks  

making a regex work with Unicode

by Anonymous Monk
on Feb 25, 2017 at 20:30 UTC ( [id://1182841]=perlquestion: print w/replies, xml ) Need Help??

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

I have a Russian article (Unicode) that I pasted into a plain-text file from a website, meaning each paragraph takes up one "line", and i'd like to be able to read it comfortably in a terminal, i.e. with wrapping on word boundaries instead of the teminal edge. fold does not handle the Unicode characters correctly - word boundaries are ignored and some characters are mangled by the line breaks. I then tried a perl one-liner that works fine with ASCII:

perl -pe's/(.{0,60})\b/$1\n/g' <text-file

This produced output where most lines were not wrapped and the ones that were were before Latin characters, and both before and after numerals. I looked up perldoc regex and tried this:

perl -pe's/(.{1,60}\b)/$1\n/ug' <text-file

This produced output that was wrapped variably between ~33-40 columns, depending on the number of spaces/Latin characters in the line (in other words, the . was couning bytes, not characters). Word boundaries were ignored. I tried many permutations of use utf8 and use feature "unicode_strings" and s///u and s///a and s///aa and \b{wb} but the result is always one of these two cases. What, if not anything I've tried so far, is the correct way to make . and \b work properly with Unicode, and if I am doing the "right thing", why isn't it working?

Replies are listed 'Best First'.
Re: making a regex work with Unicode
by haukex (Archbishop) on Feb 25, 2017 at 20:48 UTC

    As a first step, make sure Perl is reading the files as UTF-8, see the -C switch in perlrun:

    $ perl -CSD -pe's/(.{0,60})\b/$1\n/g' FILE

    If that isn't enough, perhaps using the \X "Unicode extended grapheme cluster" and/or \b{wb} "Unicode Word Boundary" will further help - I've never used them so far, but a quick test on a simple file worked for me:

    $ perl -CSD -pe's/(\X{0,60})\b{wb}/$1\n/g' FILE

    (As I was writing this, poj wrote about Text::Wrap, which also appears to have Unicode support, so that might be easiest.)

Re: making a regex work with Unicode
by poj (Abbot) on Feb 25, 2017 at 20:42 UTC

    Try Text::Wrap

    #!perl use strict; use Text::Wrap; $Text::Wrap::columns = 80; my $text = q!I have a Russian article (Unicode) that I pasted into a p +lain-text file from a website, meaning each paragraph takes up one "l +ine", and i'd like to be able to read it comfortably in a terminal, i +.e. with wrapping on word boundaries instead of the teminal edge. fol +d does not handle the Unicode characters correctly - word boundaries +are ignored and some characters are mangled by the line breaks.!; print wrap('', '', $text);
    poj
Re: making a regex work with Unicode
by kcott (Archbishop) on Feb 26, 2017 at 06:08 UTC

    Short answer - this worked for me:

    s/(.{50,80})(\s+)/$1\n/g

    Longer answer. I generated text containing random Cyrillic characters (simulating words of random length). I then split it into smaller, more easily readable, parts. Here's the test code:

    #!/usr/bin/env perl -l use strict; use warnings; use open OUT => qw{:encoding(utf8) :std}; # From: http://www.unicode.org/charts/PDF/U0400.pdf # Cyrillic: 0400 - 04FF # Basic Russian alphabet: 0410 - 044F my @sample_chars = map { chr } hex '0410' .. hex '044F'; print 'Total characters for random selection: ', 0+@sample_chars; my $string; for (0 .. 50) { for (1 .. 5 + rand 5) { $string .= $sample_chars[rand @sample_chars]; } $string .= ' '; } print 'One line:'; print $string; (my $multiline_string = $string) =~ s/(.{50,80})(\s+)/$1\n/g; print 'Many lines:'; printf '%10s', $_ for 1 .. 8; print ''; print '1234567890' x 8; print $multiline_string

    Here's the output from a sample run:

    Total characters for random selection: 64
    One line:
    дэНштц ЗжывдЖеб АэиТорЫиФ хоИыбЫйси СбГВЧмп шзхендюх жэШЗмВ фЕЙяЛСрЛж виуПхьф жзиНж НщВЬУьы ЪЩОЬБсыон сфИоуМэы ТЬХуйххЯШ дъЬуйъ ЦОзкИ ЧЫйЯф НпГзэц ЭмЭЭВХННС иЧАктЭУЩ сктТфэ ъсязцл ЬОбФцю ХМуик ПбюНшсюЪю ЪнЯцмИТ сШЪьхесвт ейячж тРцеъвР эЙдщчфПИЫ шгдЫщйР йЙДЫлрЮс еХъкДьНюС АТажшкеЙ ПхюЩдйИОх хкваЭ еЙЫЮнФ яСеяеВСШ йюЬижНЧРв СвыЭШАн ЮтвсъХЦа ъРЭШЖВню чЩыоСЕц УУЪуЦЭ эЬЮШвХЧЗ щсущйМ ЩгБхщЛргА фыЯОНдм КзЕХВА фБсиыШ пжАьР 
    Many lines:
             1         2         3         4         5         6         7         8
    12345678901234567890123456789012345678901234567890123456789012345678901234567890
    дэНштц ЗжывдЖеб АэиТорЫиФ хоИыбЫйси СбГВЧмп шзхендюх жэШЗмВ фЕЙяЛСрЛж виуПхьф
    жзиНж НщВЬУьы ЪЩОЬБсыон сфИоуМэы ТЬХуйххЯШ дъЬуйъ ЦОзкИ ЧЫйЯф НпГзэц ЭмЭЭВХННС
    иЧАктЭУЩ сктТфэ ъсязцл ЬОбФцю ХМуик ПбюНшсюЪю ЪнЯцмИТ сШЪьхесвт ейячж тРцеъвР
    эЙдщчфПИЫ шгдЫщйР йЙДЫлрЮс еХъкДьНюС АТажшкеЙ ПхюЩдйИОх хкваЭ еЙЫЮнФ яСеяеВСШ
    йюЬижНЧРв СвыЭШАн ЮтвсъХЦа ъРЭШЖВню чЩыоСЕц УУЪуЦЭ эЬЮШвХЧЗ щсущйМ ЩгБхщЛргА
    фыЯОНдм КзЕХВА фБсиыШ пжАьР 
    

    I ran this several times with similar results. Lines only wrapped when they otherwise would have exceeded 80 characters (not 80 bytes). The 50 in {50,80} was somewhat arbitrary: vaguely based on the longest, non-technical word in English having 29 characters. I don't speak, read or write Russian: you can probably come up with a better number.

    As you can see from the code, the only statement I needed to handle Unicode was for the output:

    use open OUT => qw{:encoding(utf8) :std};

    The problem on your command line examples is that you're not specifying the encoding to Perl: it just does its best and deals with the bytes you're feeding it. If you indicate the encoding, the '\b' assertion works fine and '.' matches characters:

    $ echo "дэНштц ЗжывдЖеб АэиТорЫиФ хоИыбЫйси СбГВЧмп шзхендюх жэШЗмВ фЕЙяЛСрЛж виуПхьф" | perl -C -pe 's/(.{1,20})\b/$1\n/g'
    дэНштц ЗжывдЖеб 
    АэиТорЫиФ хоИыбЫйси 
    СбГВЧмп шзхендюх 
    жэШЗмВ фЕЙяЛСрЛж 
    виуПхьф
    

    — Ken

Re: making a regex work with Unicode
by Anonymous Monk on Feb 25, 2017 at 21:32 UTC

    solved! Text::Wrap had the same effect as using \s instead of \b - the words wrapped but it was still half the width that it was supposed to be and it ate empty lines (I'm guessing there is some option to disable that, but as a quick one-statement stream filter I was looking for a short solution). Using -C however made the regex work exactly as it should have, with the correct widths and everything:

    perl -C -pe's/(.{1,60})\s/$1\n/g' <russian-text

    Using -C with Text::Wrap made the column width correct but it still converted \n\n to \n, which impaired readability. \b also worked properly with the wide characters but it split on periods and other attached punctuation. Thanks for such promt help!

Re: making a regex work with Unicode
by vr (Curate) on Feb 26, 2017 at 10:15 UTC

    Sorry, maybe it's nitpicking or not, but using /b assertion just won't work for the task as stated:

    ...article... plain-text ...each paragraph takes up one "line"

    Punctuation may be torn from preceding words, including (always) at the end of paragraph.

    C:\>perl -e "@a=(qq(This produced output where most lines, alas, were +not wrapped.\n\n))x3; for (@a) { s/(.{0,37})\b/$1\n/g; print }" This produced output where most lines , alas, were not wrapped . This produced output where most lines , alas, were not wrapped . This produced output where most lines , alas, were not wrapped .
Re: making a regex work with Unicode
by Anonymous Monk on Feb 25, 2017 at 20:49 UTC

    small update: I had a "duh" moment and used a literal space instead of the \b in my regex. Now it wraps the way I want it to, but the . still reads bytes instead of characters and I have to use .{0,120} instead of .{0,60}, so I'm still curious how to make the . work properly for any future regexes

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://1182841]
Approved by haukex
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others chilling in the Monastery: (5)
As of 2024-04-23 11:13 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found