Re^3: unicode version of readdir

[demerphq]

Personally i disagree with this. Id like to see this behaviour be automatic. Both VMS and Win32 support filesystems that properly support unicode file names, unlike the hacky approach of the UNIX world. Since we *can* know at a system level how the data is encoded UTF16LE there is no problem mapping that data to and from UTF8. The issue you mentioned elsewhere of codepages doesnt play a role as far as i understand it (which could just mean I misunderstand it :-). Unicode data is unicode data. Codepages only play a role when you want to translate a unicode string into your current "locale", but if you stay in unicode the entire time it shouldnt matter.

The problem here is that the internal interfaces are designed around an exceptionally simple interface due to legacy reasons. However were someone to put in the work to change all of these interfaces to deal with SV's and not char *'s then we would be prepared for the existing OS/filesystems that can handle unicode filenames properly as well as for when some *nix file system does the same thing.

If you check the archives you will see this subject has come up before and that Jan Dubois (of Activestate) has opined on various pathways to resolve it, however they are large scope projects which are not within scope of 5.10, although could easily be in scope for 5.12.

[dk]

There's no problem when you send a FS request (open,stat etc)-- this can easily be automatic. There's a problem when you receive FS info, that is, OS cannot decide for you what format to provide data in. For example, on win32 you might not want to have results in unicode, -- so OS conveniently maps internal UTF16LE names into, say, Latin-2. Of course, if we (suppose) never use 8-bit OS interface, but only unicode API, we convert utf8<->bytes by using perl's Encode, but then another problem arises: other users of the OS may (externally) supply us with a 8-bit file name, and if system codepage is tricky, then there's no way we can correctly convert the supplied 8-bit filename into proper UTF8 using Encode.

I actually just 20 minutes ago send a mail to p5p with a proposed patch that does exactly this.