Direct access is slower: Dereferencing the hash ref is not a big deal. It's only taking a reference that's slow, for some reason.
Naming the object twice: good point, that's no better than putting the name in the attribute's text.
Not generating an accessor: how do you do it now? I suppose that if you write your own accessor method it will overwrite the generated one, but then you have to make sure it happens in that order, and you have to take pains to zap the ones you don't want.
I think the 5.x attribute mechanism should be extended to get the source name and ref, not just (sometimes) a glob. As is now, attributes are pretty pointless on lexicals because it doesn't provide a way to associate the attribute with the thing it's attached to!