Re: Dirtiest Data

[graff]

Coping with faulty data is one of the larger components of my job description -- all kinds of stuff:

• detecting when an audio file has gone through a "text-mode" unix-to-dos filter;
• fixing nasty anomalies in a database of 10,000+ names/addresses/phone numbers (and then looking for "duplicates", which of course would have been entered in slightly different ways each time);
• spotting and removing all the forms of corruption conceivable in an archive of newswire data that was delivered via modem over a 10-year period (line noise like you couldn't imagine);
• "normalizing" the contents of harvests from diverse web sites in various languages (and multiple encodings per language): get them all into unicode and convert them all to a simple, uniform xml text format;
• ... others too numerous to mention

Throughout it all, the most important thing for me has been to have a good set of diagnostic tools. The one tool I tend to use most often, as a first resort in the widest range of tasks, simply prints out a byte-value histogram, either as a 256-line list or as a nice 8-column x 32-row table, with an optional summary that counts up character categories like "printable ascii", "non-printable ascii", "8-bit", "iso-printable 8-bit", "digits", "whitespace", etc.

With practice, you can figure out quite a lot about any sort of data just by viewing the distribution of byte values this way. If you have a specific expectation of what the data is supposed to be (ulaw audio? pcm? ascii text? text in some given language and encoding?), the byte histogram can tell you right away whether there's anything "out of band" (e.g. text shouldn't contain \x7f or null bytes, among other things), and whether any particular byte values have an unexpectedly high or low frequency of occurrence ("hmmm, too many \x0d bytes in this audio file..." or "this xml file has different quantities for '<' and '>'...")

After that, additional diagnostic tools tend to get more "specialized" (ad hoc). But among these, the next most generic one produces a code-point histogram for wide-character text data in any of several distinct multi-byte encodings, and also reports any encoding errors that it finds -- good for knowing when a utf8 file alleged to be Russian happens to contain Arabic characters, and so on...

[xdg]

The one tool I tend to use most often, as a first resort in the widest range of tasks, simply prints out a byte-value histogram, either as a 256-line list or as a nice 8-column x 32-row table, with an optional summary that counts up character categories like "printable ascii", "non-printable ascii", "8-bit", "iso-printable 8-bit", "digits", "whitespace", etc.

Wow. Too bad I can only ++ once.

Any chance there's a CPAN module to do that? If not, you should definitely write it! Moreover, this sounds like a great talk to give at seminar or conference. Or a great article for perl.com or The Perl Review.

[graff]

Any chance there's a CPAN module to do that? If not, you should definitely write it!

um... well, <confession> the tool I referred to there is one that I actually wrote in C (so long ago, it was before I learned Perl) </confession>. "It ain't broke", so I've had no need to rewrite it. I sincerely apologize if it was inappropriate to discuss it here.

Obviously a good Perl version to do the same thing would be a lot fewer lines of code than my C version, and most likely would not be significantly slower. But for the time being, I'm sorry that I must "leave it as an exercise for the reader..."

(Update: I'm happy to share the C code with anyone who might want to try it out -- you can download it here: ftp://ftp.ldc.upenn.edu/pub/ldc/misc_sw/chist.c -- again, please forgive me for straying off-topic to non-Perl tools, and accept it in the spirit of PerlMonks, as an opportunity to adapt and enhance it in Perl.)