Beefy Boxes and Bandwidth Generously Provided by pair Networks
XP is just a number
 
PerlMonks  

comment on

( [id://3333]=superdoc: print w/replies, xml ) Need Help??

When faced with a similar requirement, I shrewdly anticipated that this requirement was the tip of a larger iceberg:   that the client would very soon be asking for the result of other queries against the data.   So, I designed the process from the get-go to use an SQLite database ... which, of course, is just a disk-file.   (There is no “server.”)   It is very easy even with command-line tools (incorporated as a bash-script, say) to import a CSV and make a table from it.   Even gigantic amounts of data can be efficiently processed.   (Sqlite is frankly an awesome piece of work ...)

DBD::Sqlite2 is particularly handy because it includes a copy of Sqlite as part of the package.

And then, you just solve the problem with queries:
 SELECT column1, column2, SUM(column3) FROM mytable GROUP BY column1, column2 ORDER BY column1, column2.

Anything and everything else that they next dream up, can be answered quickly and in the same way.   (You can also remind them that they can access the data and run queries against it using any spreadsheet ...)


The only “gotcha” that you should be aware of with SQLite, particularly when you are updating information but also when you are reading it, is that you should do everything in a “transaction.”   If you do not, you will see that everything runs very slowly, because SQLite is then purposely not using “lazy reads and writes.”


In reply to Re: perl group by and sort from a csv input file by sundialsvc4
in thread perl group by and sort from a csv input file by gowthamvels

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Are you posting in the right place? Check out Where do I post X? to know for sure.
  • Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
    <code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
  • Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
  • Want more info? How to link or How to display code and escape characters are good places to start.
Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others romping around the Monastery: (8)
As of 2024-04-19 08:32 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found