Beefy Boxes and Bandwidth Generously Provided by pair Networks
Don't ask to ask, just ask
 
PerlMonks  

Re: How to remove wide chars (ex: 年 or 或) from a text file?

by davido (Archbishop)
on Mar 05, 2013 at 20:52 UTC ( #1021903=note: print w/replies, xml ) Need Help??


in reply to How to remove wide chars (ex: 年 or &#x6216) from a text file?

Do you want to remove any code point that is outside the range of normal ASCII text? Do you want to remove any HTML entity that translates to a code point beyond the ASCII range? Both of those could leave big holes in your text.

Or are you asking how to map code points that fall outside of the ASCII range into code points that fall within? If so, what mapping would you prefer? And how would you gracefully downgrade characters that don't have any obvious ASCII approximation?

As you begin to consider and answer (for yourself and for us) those questions, a solution will emerge.


Dave

  • Comment on Re: How to remove wide chars (ex: 年 or &#x6216) from a text file?

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1021903]
help
Chatterbox?
[ambrus]: the directory structure may be 1, 2, or 3 deep, it may have spaces in the filename or not, it has dates in various format, different keywords for the same meanings, and the dates and other keywords are assembled in various ways.
[Discipulus]: no ambrus by specification i mean for example license per core instead of per socket, so fields are different

How do I use this? | Other CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (13)
As of 2017-03-29 12:19 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Should Pluto Get Its Planethood Back?



    Results (351 votes). Check out past polls.