Re: PostgreSQL cursors with Perl.

[ctilmes]

in reply to PostgreSQL cursors with Perl.

I used to keep it in plaintext files, but now I'd want to run some simple queries with it. Hence, I need a database.

What type of queries? What type of data structures/schema are you using?

FWIW, I've found BerkelyDB very nice for simple data structures, and very fast local queries. (I also find PostgreSQL very nice for more complicated data structures and queries...)

[atemerev]

I have quotes data for around 15 currency pairs (like 'EUR/USD', 'USD/JPY' etc). For every currency pair there is a separate table (there are no inter-table joins) of a following structure:

id => int primary key,
timestamp => bigint,           # epoch timestamp in milliseconds
bid => decimal,
ask => decimal,
bid_amount => int,
ask_amount => int,
data => varchar                # lots of additional data that 
                               # need not to be indexed nor normalized

The queries look like:

SELECT * FROM table WHERE timestamp > ? AND timestamp < ?

Or:

SELECT MAX(bid), MIN(bid), MAX(ask), MIN(ask) FROM table 
WHERE timestamp > ? AND timestamp < ?

[camlet]

i don't know if sec will work for 15 gig files but sec works very efficiently with monitoring and parsing simultaneously a large amount of log files. hth. http://www.estpak.ee/~risto/sec/

[atemerev]

Thanks for link. Probably I will not use it for this particular task, but this thing definitely is worth looking at.