Perl uses reference counting as its garbage collection technique. When the last reference to a given object falls out of scope, or is explicitly eradicated (as in your call to delete), the object's reference count drops to zero, and Perl is permitted to clean it up. Memory leaks typically occur when reference counts are prevented from dropping to zero, and that usually happens with self-referential or co-referential objects. If $a = \$a, then when $a falls out of scope the reference count cannot drop to zero. This is what Scalar::Util's weaken() is there to address, or object destructors that are written to break known circular cycles.
A more thorough, and community-refined explanation is available in perlref, under the section Circular References.