Re^2: designing a program - your wisdom needed

[SpaceCowboy]

Thank you for your message. I meant an sql join. Like other monks and yourself have suggested, I will look at creating subroutines and get back to the thread. The tables are in few million rows.
So, you are using Redis as staging? I belive Redis is memory hungry and may take a lot of physical memory.
I was unable to understand to fully understand the "simple data store wth perl" link, what I gather is that it connects wtih a remote server and loops into some kind of regex expression? pardon my ignorance, I would love to learn your intention here and how this applies to relational tables.

[bliako]

I meant an sql join.

so, it's better to do that within the DB at least between scripts 1 and 2. (and to be clear on my above writing, doing things into DB is usually much better than retrieving data and transforming it in your Perl script *provided you write the correct SQL* - which I find almost impenetrable, that's why I am always looking for alternative, albeit roundabout and perhaps inefficient, ways. Bottomline: sticking with the DB is better, usually.)

I belive Redis is memory hungry and may take a lot of physical memory.

I have not noticed anything upnormal there. You can always try it out and see. It was quite fast for me. Can't say anything about memory usage.

The "simple data store", I linked, acts as a server which clients (let's say your scripts) contact in order to either store a string (which can be some JSON or XML or with minor modification to be a binary serialised perl hash or array, possibly nested) by key, or retrieve a string by key. The regex you refer to implements the simplest API to do that. That is, it checks if client gave it something like KEY1=VALUE1, in which case it stores it. Alternatively, client can send a KEY1=, in which case it looks in its private hash-store if KEY1 exists and sends back to client the value stored. Warning: all checks for storing and retrieving data done in DBs is absent, e.g. handle race conditions, etc. Also, there is no encryption or password protection, all scripts can see all data provided they know the key.

The "simple data store" does not apply to relational tables directly, it just offers a way for separate, independent scripts or programs (in various languages) to share some temporary data between them. It does not replace a database. It just offers a way to avoid using the DB as a temp data store. One example of use: script1 does data scraping and processing at irregular intervals. It places raw results in the data store. script2 run as a "CGI" checks if results exist in data store and converts them to HTML for viewing. For me script1 was in C, and script2 in Perl. It saved me a lot of trouble to do that (edit: ) instead of (end edit) with temp DB tables (ok, sqlite is a bit easier).

That's my experience which is not industrial. Others here have industry experience.