Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"

How to convert a string like this: "\321\201\321\202\321\200\321\203\320\272\321\202\321\203\321\200\320\260" to a proper utf8?

by igoryonya (Pilgrim)
on Apr 28, 2016 at 05:50 UTC ( #1161728=perlquestion: print w/replies, xml ) Need Help??

igoryonya has asked for the wisdom of the Perl Monks concerning the following question:

It contains this string: "структура".

I am lost, can't find how to do it.
I guess, I have to use pack or regex, but I don't know where to start. I don't even know, what this encoding is called, although, I saw it before many times.


I've figured it out. I realized that those 3 digit sequences are not decimal, but octal, so I've tried:
my $string = "\321\201\321\202\321\200\321\203\320\272\321\202\321\203 +\321\200\320\260"; $string =~ s/\\([[:digit:]]{3})/sprintf "%c", oct($1)/ge; print $string;
And it Worked! :)
  • Comment on How to convert a string like this: "\321\201\321\202\321\200\321\203\320\272\321\202\321\203\321\200\320\260" to a proper utf8?
  • Download Code

Replies are listed 'Best First'.
Re: How to convert a string like this: "\321\201\321\202\321\200\321\203\320\272\321\202\321\203\321\200\320\260" to a proper utf8?
by Ovid (Cardinal) on Apr 28, 2016 at 06:04 UTC

    Take a look at Encode::DoubleEncodedUTF8. The following returns your original string:

    #!/usr/bin/env perl use 5.18.0; use warnings; use utf8::all; use Encode; use Encode::DoubleEncodedUTF8; my $bytes = "\321\201\321\202\321\200\321\203\320\272\321\202\321\203\ +321\200\320\260"; my $fixed = decode( "utf-8-de", $bytes ); say $fixed;

    Update: A strong word of caution about the above. Fixing "broken" Unicode data is fraught with error, particularly if you can't guarantee the source of the error. If this bad data is coming from outside your program, then try to contact whoever generates that bad data and see if you can get it fixed there. Otherwise, you may have to trust yourself to be lucky on this.

    However, if the bad data is being generated from within your program, try to fix the underlying bug before you try to "fix" the bad strings in question. It will save you much grief in the long run.

      The output comes from 'git status' command.
      I have some filenames in Russian in my project folder.
      The surprising thing is that git shows all other Russian text correctly, only file names appear encoded.
      I've already seek for answers on git's irc chanel. Nobody could help me there.
      Searched on the internet, but the closest thing that I've found about this issue, is a 2 year old similar to mine question, unresolved.
      So, I thought, that, since the git's community can't help me, I will just write an output reformatter. Something, like this:
      git status -s|perl -nle 's/\\([[:digit:]]{3})/sprintf "%c", oct($1)/ge +; print'
      After seeing, that it works, I've moved perl oneliner to a file, so, now I do:
      git status -s|
        I believe the quoting can be disabled with core.quotepath option. See man git-config for details.

        Also, that reminds me of... :)

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://1161728]
Approved by Ovid
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chanting in the Monastery: (3)
As of 2021-09-18 13:36 GMT
Find Nodes?
    Voting Booth?

    No recent polls found