Beefy Boxes and Bandwidth Generously Provided by pair Networks
P is for Practical

Re^2: How to improve this data structure?

by karlgoethebier (Prior)
on May 21, 2013 at 18:30 UTC ( #1034577=note: print w/replies, xml ) Need Help??

in reply to Re: How to improve this data structure?
in thread How to improve this data structure?

I assume you wanted to say something like:

Regards, Karl

«The Crux of the Biscuit is the Apostrophe»

  • Comment on Re^2: How to improve this data structure?

Replies are listed 'Best First'.
Re^3: How to improve this data structure?
by sundialsvc4 (Abbot) on May 21, 2013 at 19:55 UTC

    Yeah, more or less.

    “A million records” is a volume that is “vaguely interesting” to SQLite ... it will take a few minutes’s one-time cost to import the data (and might not require a program).   A couple minutes more to add some indexes.   From that point on, you can use any tool or combination of tools that has a DB-interface (including Perl of course ...) to get the job done, and now the query-engine is the one that’s doing all the heavy lifting.   So long as the “transactions” caveat is carefully adhered-to esp. when doing updates, it’s really quite a remarkable piece of software engineering.   (It’s rare when a piece of software genuinely surprises me blows me away.   Perl/CPAN did it.   So did this.)   I suspect that most of the things that the O.P. is right now “writing programs to do” can probably be reduced to a query (and perhaps a now-trivial program to digest the results).   Furthermore, a huge bonus is that you can put results into a different table, which of course is a self-describing data structure.   (A single SQLite file can contain any number of tables.)

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1034577]
[LanX]: but but but ... the errors are nicely formatted!
[tye]: 1) The python stack traces show you the names of variables, not the values of variables. Almost never informative.
[LanX]: do you have a py job now?
[tye]: Right now I have errors trying to fetch stuff from graphite. It says 'apps not ready, check startup logs'. I eventually find the startup logs and see 'no apps found'. Looking for fixes to this problem...
[LanX]: no idea ... ask py monks! ;-p
[tye]: it turns out that this can be caused by almost anything going wrong. If you want to know what caused the problem, you have to try to load each app by-hand to see what problems it is having, because none of those will be logged.
[tye]: No, I write more Perl than py at work. But I have to deal with plenty of py things.
[tye]: py monks would just be offended.
[Corion]: ;)
[Corion]: Hi tye btw ;)

How do I use this? | Other CB clients
Other Users?
Others lurking in the Monastery: (7)
As of 2017-09-21 20:11 GMT
Find Nodes?
    Voting Booth?
    During the recent solar eclipse, I:

    Results (252 votes). Check out past polls.