Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"
 
PerlMonks  

High Transaction Data Persistence Requirement across Different Process / Programs

by norbert (Initiate)
on Jun 28, 2011 at 05:08 UTC ( #911662=perlquestion: print w/ replies, xml ) Need Help??
norbert has asked for the wisdom of the Perl Monks concerning the following question:

Hi,

The question is regarding achieving the following with High Throughput

Have a Perl Process processing 2000 - 5000 Messages / Second. There is a Need for this program to share information of 40 bytes in length , with two different other process. ( Data is of the form {SessionId}="Number-Number-String"

All these 3 process are asynchronous. And process by Process as they fetch the data and Read, they also manipulate the value like {SessionId}="Number-NewNumber-NewString"

And also a need to clean it up when the processing is complete or a sort of expire with a function being triggered Data base Oracle is proving to be too slow , Is there any option of Cache or Memory or any other technique in which such data can be written read modified ?

All / Any help is appreciated

Thanks

Comment on High Transaction Data Persistence Requirement across Different Process / Programs
Re: High Transaction Data Persistence Requirement across Different Process / Programs
by Corion (Pope) on Jun 28, 2011 at 06:09 UTC

    If you don't need the data to be safe, you can look at something like memcached. Perl has modules for accessing memcached.

    If you can tie all three processes onto one machine, maybe it is enough to pipe the data from the first process to the next process and so on? Or does the first process need (write) access to the session ID after it has given the ID to the second process?

      Hi

      All 3 Process are on one Machine itself

      I need to run a Memcache server for the same ?

      Shall look into the URLS

      Thanks
      Hi,

      How stable is memcache ?

      The data we are dealing with is related to charging , so the data is critical

        memcached is not a persistent queue solution. If the memcached process goes down, all information in it is lost.

Re: High Transaction Data Persistence Requirement across Different Process / Programs
by BrowserUk (Pope) on Jun 28, 2011 at 06:38 UTC

    I think you need to clarify the operating environment and operational requirements of your question.

    1. Is this 3 long running concurrent processes? Or with one or more transient (web-server-like) processes?
    2. Is that a single 40-byte string? Or one 40-byte string per id? (If so, how many ids?)
    3. You mention persistence. Is that persistence between connections? Or persistence across server outages?
    4. Are the other two processes also Perl? Or C? Other?
    5. Do all three need read-write access? Do you need coherency?
    6. Many other questions that would be answered or avoided by a much clearer description of the actual purpose.

    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

      Hi let me elaborate a bit MORE to get Better Suggestions from all Perl Monks

      Platform Red Hat Linux 64 Bit 5. 4 with Perl 5.8

      1. There are "A" number of Perl ProcessA running, which read from Linux QUEUE (ID - Amount - Details ) and make a Call to FCGI process B , Collect the response, based on the Response(Response gives Session ID for the ID )....TO 3

      2. There is a FCCI process C , which is invoked by External systems for Session ID to provide value Success or Failure ..., which is read and Mapped to the DB with Session ID - Success/Failure

      3 . The FCGI B after getting the response, based on the Session ID will query DB for the Session ID and Write in to a Linux Queue

      All of these 3 Process are in Synchronous mode dependent on one another, we are trying to make this Asynchronous and achievable at high throughput. So to make the same , we need to share data across the 3 Process to remove the Database interaction

      Coming to the Questions asked

      1.These are Non stop running process , running in multiple instances

      2.Mapping is required to be like Id- SessionID - Amount - Unit - Success/Failure - Number

      3.Persistence - In terms of Data persistence across these processes on single server

      4.All the 3 are perl Process

      5. Yes worst case scenario need all the 3 to read write

      6. Sorry for not being clear in my first description

        Sorry, I don't think I can help. I've read and re-read your description of the 3 processes, but I cannot make sense of the flow of initiation at all. Hopefully someone with more experience of clustered/cloud-based CC processing will be able to help.

        Specifically I don't understand your nomenclature "process A ... make a s call to FCGI process B"? Are you saying that Process A makes an HTML connection to process B?

        Nor do I understand "FCGI process C , which is invoked by External systems".

        Basically, I'm out of my knowledge zone.


        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
Re: High Transaction Data Persistence Requirement across Different Process / Programs
by zek152 (Pilgrim) on Jun 28, 2011 at 12:44 UTC

    I would suggest doing some reading on IPC (interprocess communication). (see Perl Doc perlipc) There are several immediate options that I can think of at the moment.

    1) Use FIFOs (aka named pipes). These are unidirectional so you would need 4 FIFOs (M->A,A->M,M->B,B->M).

    2) Use Unix-domain sockets. This will allow bidirectional communication so you will need 2 connections. (M<>A,M<>B)

    3) Use TCP sockets. This is probably overkill unless you plan on the processes being on different computers.

    Assuming that the 40bytes is 1/message you have a need of about 5000*40=200,000 or 200kB/s. All of the proposed methods should be able to reach that throughput. I have seen benchmarks that suggest that 93mB/s should be obtainable with local TCP sockets. 93mB/s was the lowest benchmark I saw.

    Hope this helps.

      Hi , i have put some more explanations on the process and the goals to achieve . DO you still feel we should go by IPC ?

        Yes. From your requirements you have different processes and you want them to communicate with each other. I still believe that interprocess communication is the way to go.

        I think that the unix-named sockets are a good starting point. You should have no trouble meeting the required throughput on the communication side of the system.

Re: High Transaction Data Persistence Requirement across Different Process / Programs
by tokpela (Chaplain) on Jun 29, 2011 at 18:02 UTC

    This sounds like a job for a queue - something like RabbitMQ. It would give you the persistence and allow you to scale even farther out if necessary across multiple servers or even instances of your processes.

    I would also consider using a NoSQL db such as CouchDB to hold your data if passing your data via the message body in the queue is not enough.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://911662]
Approved by davido
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chilling in the Monastery: (3)
As of 2014-09-18 03:51 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (106 votes), past polls