Beefy Boxes and Bandwidth Generously Provided by pair Networks
Syntactic Confectionery Delight

Re^3: unicode version of readdir

by demerphq (Chancellor)
on Sep 17, 2007 at 17:46 UTC ( #639451=note: print w/replies, xml ) Need Help??

in reply to Re^2: unicode version of readdir
in thread unicode version of readdir

Personally i disagree with this. Id like to see this behaviour be automatic. Both VMS and Win32 support filesystems that properly support unicode file names, unlike the hacky approach of the UNIX world. Since we *can* know at a system level how the data is encoded UTF16LE there is no problem mapping that data to and from UTF8. The issue you mentioned elsewhere of codepages doesnt play a role as far as i understand it (which could just mean I misunderstand it :-). Unicode data is unicode data. Codepages only play a role when you want to translate a unicode string into your current "locale", but if you stay in unicode the entire time it shouldnt matter.

The problem here is that the internal interfaces are designed around an exceptionally simple interface due to legacy reasons. However were someone to put in the work to change all of these interfaces to deal with SV's and not char *'s then we would be prepared for the existing OS/filesystems that can handle unicode filenames properly as well as for when some *nix file system does the same thing.

If you check the archives you will see this subject has come up before and that Jan Dubois (of Activestate) has opined on various pathways to resolve it, however they are large scope projects which are not within scope of 5.10, although could easily be in scope for 5.12.


Replies are listed 'Best First'.
Re^4: unicode version of readdir
by dk (Chaplain) on Sep 18, 2007 at 20:34 UTC
    There's no problem when you send a FS request (open,stat etc)-- this can easily be automatic. There's a problem when you receive FS info, that is, OS cannot decide for you what format to provide data in. For example, on win32 you might not want to have results in unicode, -- so OS conveniently maps internal UTF16LE names into, say, Latin-2. Of course, if we (suppose) never use 8-bit OS interface, but only unicode API, we convert utf8<->bytes by using perl's Encode, but then another problem arises: other users of the OS may (externally) supply us with a 8-bit file name, and if system codepage is tricky, then there's no way we can correctly convert the supplied 8-bit filename into proper UTF8 using Encode.

    I actually just 20 minutes ago send a mail to p5p with a proposed patch that does exactly this.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://639451]
[Discipulus]: such tests are intended for modules, as far as i know

How do I use this? | Other CB clients
Other Users?
Others romping around the Monastery: (6)
As of 2017-01-24 12:46 GMT
Find Nodes?
    Voting Booth?
    Do you watch meteor showers?

    Results (204 votes). Check out past polls.