Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW
 
PerlMonks  

Re: Hash lookups, Database lookups, and Scalability

by pg (Canon)
on Oct 31, 2004 at 17:40 UTC ( #404148=note: print w/ replies, xml ) Need Help??


in reply to Hash lookups, Database lookups, and Scalability

First, DBD::SQLite is not a sound database any way. Also there is a big performance boost when you wrap your queries in DBD::SQLite's transaction. I didn't see you use it, and it would be interesting to get another benchmark with that enabled.

In general, database search is less efficient than in-memory ones. First you have more freedom to tailor your algorithm and data structure for in-memory search; secondly for database search, the results are transferred through network communication, two-way communication for each query, which obviously costs extra time. How good is the database's cache ability will also impact the performance.

However this does not mean that database search is not a good viable solution. Database gives you lots of important benefits than just speed, which is OT here.


Comment on Re: Hash lookups, Database lookups, and Scalability
Re^2: Hash lookups, Database lookups, and Scalability
by davido (Archbishop) on Oct 31, 2004 at 17:53 UTC

    I'm curious what you mean by "DBD::SQLite is not a sound database". I know it's a single file, lightweight, minimal-SQL set database, but to me something that isn't sound is buggy. Is that what you're saying? Just curious. (By the way, I just discovered a GPF-producing bug if I switch my code to use selectrow_array() with a pre-prepared statement on Win32, so maybe you've got a point. ;)

    And yes, I fully expected the database solution to be much slower than the in-memory hash solution. I mentioned that in my post. I was attempting to quantify how much slower, and how sensitive the database version is to dataset-size.

    Wrapping in transactions definately helps. Take the example of my building the database initially. With transactions, it takes the blink of an eye, but without, it takes about 45 seconds to build the database on my system. But on the query side, I'm specifically trying to engage in single queries, and didn't want to see transactions influencing the speed. However, I agree it would be interesting to rewrite the test in such a way that it could take advantage of transactions, and I may do that just to see what comes of it.

    Anyway, it's kind of fun to tinker with this stuff. ...hope you agree, even it the database discussion is diverging a little from core-Perl topics.


    Dave

      "Anyway, it's kind of fun to tinker with this stuff. ...hope you agree, even it the database discussion is diverging a little from core-Perl topics. "

      I agree 100 percent. I did notice that recently we are having more and more database related discussions here, which is very possitive, at least to my standard. This indicates to me that more and more people are using Perl for an increased range of application areas, and most likely the average size of the applications is going upwards. This is definitely a positive news.

Re^2: Hash lookups, Database lookups, and Scalability
by Anonymous Monk on Oct 31, 2004 at 18:27 UTC
    there is a big performance boost when you wrap your queries in DBD::SQLite's transaction

    How curious.

    A few days ago, you had a quarrel with another monk when you made the same mistake and were told the same thing.

    Now you are showing off that very fresh knowledge you were criticizing.

      "Now you are showing off that very fresh knowledge you were criticizing."

      No, this means I am learning, just as everybody else here does, why should that surprise you? I would be ashamed of myself, if I am stuck with my non-talent, and refuse to learn and improve myself.

      I downvoted this node. I don't see what the point of it is, and frankly it seems to be a very subtle ad hominim attack on pg. Whats your agenda with pointing this out, what possible benefit does it have for the monastery?

      ---
      demerphq

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://404148]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others romping around the Monastery: (6)
As of 2014-12-20 14:44 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    Is guessing a good strategy for surviving in the IT business?





    Results (96 votes), past polls