I am not sure that it would help you with your specific issues (although I don't think that threading would be a problem), but Log::Log4perl is perhaps worth looking at.
loris
"It took Loris ten minutes to eat a satsuma . . . twenty minutes to get from one end of his branch to the other . . . and an hour to scratch his bottom. But Slow Loris didn't care. He had a secret . . ." (from "Slow Loris" by Alexis Deacon)
| [reply] |
First of all, I'd say that 2K per second is _not_ a lot of data. If your program is not keeping up with the input streams, post your code, and I'm sure we can figure out what's holding it up.
Also, I think that most logging systems will flush at the end of each line since the purpose of a logging module is to save the log message to permanent storage especially in the event that the program crashes shortly afterwards. Still, flushing should not significantly slow your program down. Again, if it is, show us what your code is like, and we can suggest how to improve it.
| [reply] |
Are you sure that logging is the problem? You should profile your code to find out where it's really is spending its time before you worry about optimizing some part of it. It's possible that logging is not a big enough part of your execution time to be worth optimizing.
| [reply] |
You seem have contradictory requirements. You want it to work with multiple threads. But you also want buffered output. But buffered output means that one thread will write out a block, ending with part of a line, then other threads will write out data before the first thread finishes its line. That will result in corrupted output.
As others have said, try it and see what performance looks like. My bet is that performance will be just fine. After all flushing after a print just means that a block of data is sent to the filesystem. It does not mean that a write happens immediately to disk. Sure, one will happen when the filesystem gets around to it. But pages are generally left dirty in memory for a bit. And as I noted above, the flushes are necessary anyways if you want to be able to avoid having multiple threads cause you headaches. | [reply] |
Unfortunately the performance is not fine. I gave also a wrong number for the data per second - in fact we are talking about up to 100 KByte per second with a base load of maybe 20 KByte/second.
I guess I will need to move the logging into a seperate thread - so that the application will keep its speed.
| [reply] |
I moved the logging into a seperate thread.
It was way too slow using the logfile write - I had only about 600.000 lines of constant back log.
Now I have moved to a standard file and the performance is ok.
| [reply] |
