Beefy Boxes and Bandwidth Generously Provided by pair Networks Cowboy Neal with Hat
Perl: the Markov chain saw
 
PerlMonks  

Re^2: The Eternal "filter.pl"

by Voronich (Hermit)
on Aug 25, 2011 at 16:49 UTC ( #922407=note: print w/ replies, xml ) Need Help??


in reply to Re: The Eternal "filter.pl"
in thread The Eternal "filter.pl"

I hate it. I hate it. I hate it. I hate that answer. "just get a database." Not you! I hate it because it IS the right answer. It's ALWAYS the right answer (for sufficient values of "always".) But it's a near firing offence if I do it as a skunkworks project and they won't allocate it willingly. Even if I were to add "nonclashing table names" into a currently unused dev database I'd get shot. It's the kind of illogic that causes stress fractures to develop in my skull as the steam escapes.

Corion said the same thing re: map reduce. I've heard of it. But I know nothing about it as it seems always to be accompanied with that pie-eyed silver-bullet attitude I've become alergic to.Perhaps it's time to give it a serious look see.

Me


Comment on Re^2: The Eternal "filter.pl"
Re^3: The Eternal "filter.pl"
by chromatic (Archbishop) on Aug 25, 2011 at 17:47 UTC
    ... it's a near firing offence if I do it as a skunkworks project and they won't allocate it willingly.

    Can you use DBD::SQLite?

      Certainly not legitimately. The allergy to external code here is silly. But frankly I need to solve the problem, so that's not going to stop me if I can help it.

      But that's a question best answered by experimentation. The trick with sqlite is getting it installed in the right place.

      Me
        When you have a legitimate need, management is likely to listen; especially when you can put a cost to how much is attributed in manhours/salary to routinely solving similar "recursive" problems.

        I would suggest you discuss having some sort of development area where you could tackle such a task. You have a lot of data, so you probably need a better db engine, but if they aren't willing to give you the tools you need, perhaps you could daisy chain a few JET (MS Access) databases, to have something a little more robust and scalable.
Re^3: The Eternal "filter.pl"
by SuicideJunkie (Priest) on Aug 25, 2011 at 17:55 UTC

    If your desktop box isn't a wimpy laptop or dumb terminal, you should be able to run a temporary database locally, without giving it any access from the network or using external resources. Worst case, would you be allowed to use something like DBM::Deep with a flat file?

    Collect your set of scripts for importing the various types of datafile you commonly see, and run those on demand during the meeting. Once the meeting is over, drop the whole database.

    So when the meeting happens, you fire up the database, doubleclick your "import_feedfile.pl", and then run the first query. Question 2, you change the query. Question 3, you doubleclick "import_downstream_feedfile.pl", and then run your new query. Rinse and repeat.

    IE: Don't have scripts for one-off filtering, only keep scripts for importing the various files you'll see more than once, and do your actual queries with SQL.

      Our desktops are actually little more than X connections with MS Office running. Attempting to install anything that trips microsoft's "installation" system is locked out, as are writes to most of the local disc and any external devices.

      I like the idea of using something like DBM::Deep. But have never used any of those, so it's new stuff. If I can inject custom parse routines into them (the data formats are never quite so simple as plain csv, and certainly not fixed-record files) then it's definitely a candidate.

      Me

        I write this as your friend, and Your Mother, start circulating your résumé. There is nothing wrong with shopping around and the mere fact of having options can make a sub-optimal situation easier to handle.

        Is running anything off a thumb-drive locked out too?

        If not, some "Portable apps" are real life savers, such as a full fledged XAMPP stack. You load your thumb drive on your home machine and sneak it in using this tiny USD drive.

        CountZero

        A program should be light and agile, its subroutines connected like a string of pearls. The spirit and intent of the program should be retained throughout. There should be neither too little or too much, neither needless loops nor useless variables, neither lack of structure nor overwhelming rigidity." - The Tao of Programming, 4.1 - Geoffrey James

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://922407]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others contemplating the Monastery: (10)
As of 2014-04-24 10:02 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    April first is:







    Results (565 votes), past polls