in reply to Remove duplicate data from text file
If you are on unix and don't care about the order of the output (or want it sorted): sort -u <file> is your friend.
Otherwise you can try:
perl -n -e 'print unless $seen{$_}++;' <file>
In Section
Seekers of Perl Wisdom