Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris

Re^2: pluggable/dynamic data processing/munging/transforming module?

by rwstauner (Acolyte)
on Nov 17, 2010 at 03:37 UTC ( #871915=note: print w/replies, xml ) Need Help??

in reply to Re: pluggable/dynamic data processing/munging/transforming module?
in thread pluggable/dynamic data processing/munging/transforming module?

Thanks for the suggestion. I agree that fixing the data at the source would be optimal, but it doesn't apply to my current situation.

The entire purpose of my application is to pull data from (various) outside sources and bring it inside to save it in our database.

"Cleaning" the data is a necessary part of the process.

some examples:

  • '1969-12-31 23:59:59' is a dummy value and not an actual date (think -1), so I'd prefer to transform it to NULL before filling my own database with garbage. (But this transformation only applies to one external source.)
  • 20101015 is an integer, not a date. I'd prefer '2010-10-15'. (This example is obviously from a different source.)
  • 'D129', '  D129', 'D129  ' all mean the same thing. I'd prefer the trimmed version.
  • The color 'MAROO' probably means 'Maroon', but looks a little silly.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://871915]
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (2)
As of 2017-02-24 02:35 GMT
Find Nodes?
    Voting Booth?
    Before electricity was invented, what was the Electric Eel called?

    Results (351 votes). Check out past polls.