Re: threads::shared vs IPC::Shareable

[Anonymous Monk]

What is the purpose of the hash here? Do you need to build some lookup index? Sort or categorize?

Natural fit for a preprocessor is to function as a filter. Keep state but don't slurp the data. You can pipe it and stream it. But if complicated queries are required, then a database model might be more suitable.

[Nar]

Id like it in multiple processes due to size of logs. As each preprocesser is processing GBs of logs having each executing in its own process space would be important to take advantage of multiple cores.

It uses hashs so as multiple preprocessers write, the processer can execute in order of datetime.

Each preprocesser would write to the same hash and the engine would process that hash.

[BrowserUk]

Id like it in multiple processes due to size of logs. As each preprocesser is processing GBs of logs having each executing in its own process space would be important to take advantage of multiple cores.

Where/what type of storage do these logs files exist on? Because running multiple concurrent file readers can slow things down rather than speed them up if you aren't very careful.

[Nar]

They are coming in by syslog, I have a perl syslog server listening on a socket. Each preprocesser will access a shared hash to access the logs. As most logs are trash and preprocessed out and the other logs dropped this should keep IO low. ---- I don't throw the logs out in syslog as they are used to preprocess the other logs, but still not relevant for the engine.