Your hexadecimal number in
$testString needs to
begin with 0x be properly interpreted as hexadecimal. Then it will work. You can also simplify your white-space removal. All together, here is one way to get where you want to go:
$testString =~ s/\s+//g;
$testString =~ s/^/0x/;
$maskResult = $testString & 400000000000000000000000;
$maskResult = hex( $testString ) & 0x400000000000000000000000;
Updated: Need to use hex and need to have mask as hex string.
Updated 2: I see ikegami has also pointed out my error(s) corrected above.
Updated 3: and ikegami has also noted the need for Math::BigInt here.