if I do * then I get back all of the elements from the whole tree
Not true. Or at least I do not get it, and you should not get it either:
perl -E '
use XML::LibXML;
my $dom = XML::LibXML->load_xml(string => q{
<library>
<book attr1="zzz">
<title>xxxx</title>
<author>yyyyy</author>
</book>
<book attr1="sssss"> ---- </book>
<book attr1="dddd"> ---- </book>
</library>
});
my ($l) = $dom->findnodes("/library");
say $_->getName for $l->findnodes("*[text()] | text()");'
#text
book
#text
book
#text
book
#text