Re: Reading and writting data in a text file with several scripts simultaneously time synchronization problem

[Monk::Thomas]

Can you give an explanation what you are trying to do? (an explanation for the actual problem, not the implemented solution) Maybe there's a totally different solution?

[thanos1983]

To: Monk::Thomas

I tried to include a short description on the previous comment, but for simplicity reasons I will pasted also here.

The whole story is that I am creating a main test script that I need it to produce random data and store them into a file for simplicity reasons. The secondary scripts I want them to retrieve the data from the folder, process them and also store their output on separate files, for comparison reasons after. I want to compare how many data were able to process and how fast!

In conclusion even the secondary scripts are test scripts I will choose only one of them to apply on my real goal. I am testing them to see performance under pressure in order to choose the best one.

This is the reason that either I need to store the data to a folder for later retrieving them or into a database. The problem is also that on my real goal several scripts will need to access this folder to retrieve the data.

So I am trying to make my experiments as realistic as possible.

I need to compare the process times. Initially with the folders in different formations and possible create a short script with MySQL also just to observe if there is a huge difference on process time.

P.S. Sorry for the long answer I tried to include everything.

[Monk::Thomas]

Maybe a totally different solution to your problem is to pipe the data. Let the main script generate the test input and then write it to 3 different pipes (or whatever the number of your testclients is). The test client would simply listen on StdIn and process any incoming data packet.

This would neatly sidestep the whole issue of file locking and concurrence. However I really can't tell if this is a suitable solution for your problem even after reading your provided description.

simplified code

open my $fh1, '-|', 'bin/testclient1';
open my $fh2, '-|', 'bin/testclient2';
open my $fh3, '-|', 'bin/testclient3';

my $time1 = 0;
my $time2 = 0;
my $time3 = 0;

while (<condition>) {
  my $testdata = make_foo();

  my $time_a = time;
  print {$fh1} $testdata;
  my $time_b = time;
  print {$fh2} $testdata;
  my $time_c = time;
  print {$fh3} $testdata;
  my $time_d = time;

  $time1 += $time_b - $time_a;
  $time2 += $time_c - $time_b;
  $time3 += $time_d - $time_c;
}

print "method 1: $time1 seconds\n";
print "method 2: $time2 seconds\n";
print "method 3: $time3 seconds\n";

exit;

P.S.: I always get '-|' and '|-' wrong. You definitely need to check the syntax.

[thanos1983]

To: Monk::Thomas,

Nice, I never thought to have a process like this one. Because I am not familiar with it I will try to apply it and see what happens.

Thank you for your time and effort, it is great to learn new things.