boxen that implement the TCP/IP protocol must respond. Even windows machines manage it.
You can catch the return packets with Net::Packet (Update: It's NetPacket, not Net::Packet, available from CPAN, runs under windows with the pcap library), if you have root access. I have a little proggie that does this if you are interested. Actually it sits there and silently matches ethernet addresses to IPs, and identifies gateways and routers from that data.
For extra humour during a boring afternoon, ping flood the broadcast address and watch the network admin go into spasms.
I didn't believe in evil until I dated it.