|Problems? Is your data what you think it is?|
If it's not broken, don't fix itby RhetTbull (Curate)
|on Dec 21, 2001 at 20:46 UTC||Need Help??|
I was tweaking a rather complicated data processing script of mine last week when I ran accross an output pipe I was using to massage the data:
I said to myself: "self, why are you using two processes (sort and uniq) when one (sort with -u flag) would work? That's silly." So, even though it had nothing to do with what I was tweaking, I changed the output pipe ot be:
and thought to myself "hah! got rid of one extra step."
Now, yesterday I noticed that the script was producing erroneous output. Since the input data often changes (depending on who produced it!) I have no formal spec for the input and I'm often tweaking the processing script to adapt it to new input formats (be liberal in what you accept). I thought that must have happened so I spent a good part of the day trying to track down what had changed. Since the data is several hundred MBs in size this process took a while.
Once I finally exhausted all possiblity that the input data was bad I turned to the script and finally I decided to test to see if "sort -u" was really equivalent to "sort | uniq"
my test data:
Aha! With multiple keys, sort -u does not behave as I had assumed.
The moral of this little story: If it's not broken, then don't mess with it! (Oh, and don't assume either!)
In light of the excellent comments and feedback here, I think I'll modify the moral of this story: