Beefy Boxes and Bandwidth Generously Provided by pair Networks
Do you know where your variables are?

Regex for Multibyte Characters

by Anonymous Monk
on Oct 21, 2005 at 14:55 UTC ( #502026=perlquestion: print w/replies, xml ) Need Help??

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Venerable Monks, this is mentioned in the FAQs but I don't seems to be able to get the desired result. I need to edit a file written in Traditional Chinese and replace a version number within it.The version is not always of the same format so I cannot trap this.The start of the specific line is always the same CHT string but I cannot seem to include this in the regex correctly to find the part I need to change. The line is of the format "CHT string" version number "CHT string" (the quotes are for effect!) Any ideas would be greatly appreciated :)

Replies are listed 'Best First'.
Re: Regex for Multibyte Characters
by Roy Johnson (Monsignor) on Oct 21, 2005 at 16:46 UTC
Re: Regex for Multibyte Characters
by ww (Archbishop) on Oct 21, 2005 at 15:05 UTC

    Insufficient detail.

    What have you tried? What is the range within which the 'version number' can fall (eg ( 1 or 2 or 3 ) or ( a#c79foo or f or whatever68 )?

    At first glance the info you have presented suggests s///ing everything between "CHT string" and its repeat.
      But he said that the "version number" has no "format" (otherwise he could just match it).
      His problem is how to match the "CHinese Traditional", multibyte, substrings in that example.

      But you are correct: "Insufficient detail"
      These "Multibyte" characters are Unicode ones ? Or Some-strange-encoding-that-I-don't-know ones ?

      (not that I know something about encodings... ^_^;;; just clarifying things )

      -- 6x9=42
        Clarifying the clarification: Actually, what OP said was "The version is not always of the same format." The request is for sufficient info to have some notion of the range of variant formulae possible.

        If all the possibilities in that range are (readily) amenable to recognition with a regex, then the question of which kind of "'Multibyte' characters" might help in writing a regex.

        If the range would make regex|regexen unacceptable (read, for example, as: 'so complex as to make execution unacceptably slow'), then another approach is required.

        Perhaps the above is simply a more precise reforumulation of your intent in para 1. If so, well done!

Re: Regex for Multibyte Characters
by graff (Chancellor) on Oct 21, 2005 at 23:35 UTC
    Following up on the remark to look up unicode support in Perl, you'll need to know the name of the character encoding used in your data. IIRC, "Traditional Chinese" would refer to some sort of "Big5" encoding, but there might be more than one sort of "Big5", and the difference might matter. (My Perl 5.8.1 on macosx/panther supports "big5-eten" and and "big5-hkscs"; there might also be a "CP???" version.)

    Whatever character set is appropriate, it might be easiest to save a plain text file containing just the "CHT string" (or both the preceding and the following "CHT string", if these are not identical), in the same character set as the original data. Then something like this should do:

    use strict; use Encode; open( I, "string.txt" ); my $string = <I>; chomp $string; # or: my ($pre,$fol) = split ' ',$string; # if file has previous and following strings close I; my $pattern = decode( 'big5-eten', $string ); # you might need a different character-set name (if so, fix it in thre +e places) # also, if you are using $pre and $fol, you need to decode each one se +parately # (e.g. into $pat1 and $pat2) my $newversion = "2.0"; # or whatever... open( I, "<:encoding(big5-eten)", "big_data.txt" ); binmode( STDOUT, ":encoding(big5-eten" ); while ( <I> ) { s/($pattern).*?($pattern)/$1$newversion$2/; # or: s/($pat1).*?($pat2)/$1$newversion$2/; # maybe you also need the "g" modifier too? print; }
    (not tested, of course, but nothing much to it, really)

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://502026]
Approved by ikegami
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others contemplating the Monastery: (2)
As of 2022-01-29 05:00 GMT
Find Nodes?
    Voting Booth?
    In 2022, my preferred method to securely store passwords is:

    Results (74 votes). Check out past polls.