Beefy Boxes and Bandwidth Generously Provided by pair Networks
Welcome to the Monastery
 
PerlMonks  

Re: Fetch Mysql huge database

by fisher (Priest)
on Sep 29, 2011 at 06:22 UTC ( #928468=note: print w/replies, xml ) Need Help??


in reply to Fetch Mysql huge database

The main idea behind this database thingie is that you do not need to fetch the whole dataset to client. If you do need it, which is a rare case, it is a bad system design. In most cases, however, you need only a small subset of stored data, and your job is to select it properly.

Second, the mysql server itself can operate too slow in some circumstances -- e.g. no index on 'UserName' field can produce inappropriate time consumption. How many time it takes when you select above dataset from mysql client itself?

Third, you need a space, a memory to store this dataset locally on your client; did you count how much memory do you need to store your 34748806 rows?

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://928468]
help
Chatterbox?
[ambrus]: Dumu: why would you want to read the same file both in binary mode and utf8 mode?
[Corion]: Dumu: Yes, but switching needs to be done with caution, as you might read half an UTF-8 character in binary mode, and then switch to utf8 mode...
[Corion]: Personally, I would do the decoding of data in the program and not in the filehandle
[Corion]: ambrus: Maybe a record-based file where some data is strings encoded as utf8
[Corion]: But again, I would read all data as binary and then decode from utf8
[ambrus]: Corion: that wouldn't be my guess, but I'll wait for Dumu
[Corion]: Meh. I'm now on Firefox 57 ("Quantum") and again have to remove Pocket, and reinstall the adblocker, uMatrix (because NoScript doesn't work) and silence the about:blank page to not download crap from the internet
[Corion]: Maybe I shouldn't update software ...
[ambrus]: Corion: my guess is to first read the start of the file to check what format it is out of two or more possibilities, then rewind it and handle it differently depending on the program
[ambrus]: s/program/format/

How do I use this? | Other CB clients
Other Users?
Others musing on the Monastery: (12)
As of 2017-11-20 17:50 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    In order to be able to say "I know Perl", you must have:













    Results (290 votes). Check out past polls.

    Notices?