I have written three attempts at replies with technical details and but I decided to delete them again. I understand all nuances of the problem fully and I am certain you are, too, so writing out the technical details is pointless. We are both looking at a cone and one from the side sees a triangle and one from the bottom sees a circle.
The difference between our perspectives:
- I chose to live in a world where 'use 5.014; use utf8;' is always in effect.
- I never use the functions internal to the utf8 module because its documentation says so.
- It is irrelevant that Perl says $a eq $b is true. In my mind, they are different types. $a is a buf of octets, $b is a string of characters. I have trained myself to notice that it does not even make sense to test for equality without converting one of them first. The only way to convert between the types is the Encode module.
- Representations are an irrelevant concept. If I have a string, I know it's UTF-8, and system calls on Posix will happily do the right thing.
- By applying the discipline of the best practices rigorously I will not possibly get myself into a situation where the problem you showed can occur.
