Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic
 
PerlMonks  

The most precise second (timer)

by tukusejssirs (Sexton)
on Nov 26, 2019 at 14:53 UTC ( #11109264=perlquestion: print w/replies, xml ) Need Help??

tukusejssirs has asked for the wisdom of the Perl Monks concerning the following question:

We have monitoring scripts that need to be run every second. The scripts execution takes longer than a second, but that is not a problem.

So we have need a function that would run the script every second. The interval must be as precise as possible. I know that the preciseness depends on CPU, RAM, CMOS etc. I donít care about these.

Currently, we use a script, letís call it negative_timer. It works quite precisely, but the Time::HiRes::sleep from time to time complains that negative time not invented yet. A second using this script is in average about 0.000000676723442 less than a real second.

The problem is that the negative_timer substracts a second from a $max value. Sometimes the NTP kicks in and it makes the trouble. We donít case if a second would be longer/shorter when the NTP updates the time.

Actually, we donít need any $max value, as we run the monitoring script indefinitely (i.e. until seldom reboot or maintaince).

Could anyone help me out in this?

Thank you in advance.

The negative_timer script:

use strict; use warnings; use Time::HiRes qw/time sleep/; sub negative_timer { my $max = $_[0]; my @time_test; my $start = time(); for (my $i = 0; $i < $max; $i++) { $start += 1; sleep $start - time(); $time_test[$i] = time(); } return @time_test; }

Replies are listed 'Best First'.
Re: The most precise second (timer)
by Fletch (Chancellor) on Nov 26, 2019 at 18:04 UTC

    Perhaps rather than reinventing wheels use AnyEvent and EV to setup a timer event to be posted every second.

    $ perl -MTime::HiRes=time -MAnyEvent -E '$quit = AnyEvent->condvar;$w += AnyEvent->timer( after => 1, interval => 1, cb => sub { state $ctr += 1; say "@{[Time::HiRes::time()]}: FOOP! $ctr"; $quit->send if 10 == + $ctr++; } );say qq{Starting loop};$quit->recv;' Starting loop 1574791289.26471: FOOP! 1 1574791290.26702: FOOP! 2 1574791291.26394: FOOP! 3 1574791292.26511: FOOP! 4 1574791293.26735: FOOP! 5 1574791294.26826: FOOP! 6 1574791295.26665: FOOP! 7 1574791296.265: FOOP! 8 1574791297.26659: FOOP! 9 1574791298.26699: FOOP! 10 $

    The cake is a lie.
    The cake is a lie.
    The cake is a lie.

      Thanks, but using low-level commands is faster. For us speed (and thus accuracy) is very important.

      I have tested your code on my computer and the average difference from a second is +0.00006222222222222222, which again is more by two decimal places than the difference of the negative_timer (which is -0.000000676723442).

        EV is an interface to the lower-level libev, and AFAIK it's fairly performant. Its "periodic" function claims to work with wall clock time, so you shouldn't have any drift relative to the system clock. That the system clock can drift relative to real time, and that it can be adjusted in jumps, are whole different issues. I'm not sure if the default ntpd supports smooth adjustments to the system clock (Update: apparently, it does), butand I believe chrony does. (I have several systems with a GPS receiver that supports the PPS signal, and I feed that into chronyd to provide a system time that should be quite accurate. I've published some info on that here.)

        use warnings; use strict; use Math::BigFloat; use Time::HiRes qw/gettimeofday/; use EV; my $w1 = EV::signal( 'INT', sub { EV::break } ); my ($cnt, $avg, $prev) = (0); my $w2 = EV::periodic( 0, 1, 0, sub { my $cur = Math::BigFloat->new( sprintf("%d.%06d",gettimeofday) ); if ( defined $prev ) { my $delta = $cur - $prev; $avg = defined $avg ? ( $avg + $delta ) / 2.0 : $delta->copy; $cnt++; } $prev = $cur->copy; } ); EV::run; print "\nAverage over $cnt deltas: $avg\n";

        Sample output from different runs on my system:

        Average over 67 deltas: 1.000005597916695584862263807609389365538 Average over 76 deltas: 1.00012161307435464691799175142220832736 Average over 137 deltas: 0.9997675784785176200109895891945901313775 Average over 442 deltas: 0.9999437250604076389815434390305220372265
Re: The most precise second (timer)
by GrandFather (Sage) on Nov 27, 2019 at 01:53 UTC

    Why is sub micro-second time important and if it is why are you using Perl which implies a non-real time OS? Even using dedicated hardware for time critical applications with that degree of time precision can be challenging so the implication of your post is that you have made a major challenge for yourself!

    Optimising for fewest key strokes only makes sense transmitting to Pluto or beyond

      Agreed. Even if you, say, used an Arduino with an external crystal for triggering interrupts you will have trouble hitting the exact timing. That is, unless you spend quite a lot on a very precise, temperature controlled and calibrated timing source.

      If you want really precise, reproducable timing, you probably have to put your time critical stuff in an FPGA (or even better: ASIC) and hook that up to a very, very precise timing source, both for chip clock cycles and the external triggering interrupt. There are chip scale atomic clocks and things like that available for that purpose.

      To make the data logging even more time stable, you also need to make sure your time critical code path is always the same, otherwise you introduce clock cycle count jitter. That means no conditionals from the start of the interrupt to having gathered all time critical data values in memory.

      ...basically, the more precision you want, the more specialized engineering you have to do. This can get expensive really, really fast. And there will always be compromises involved. It's the old mantra of "good/fast/cheap - pick any two of those".

      perl -e 'use Crypt::Digest::SHA256 qw[sha256_hex]; print substr(sha256_hex("the Answer To Life, The Universe And Everything"), 6, 2), "\n";'
Re: The most precise second (timer)
by Don Coyote (Friar) on Nov 26, 2019 at 19:20 UTC

    For finer resolution you could try select rbits,wbits,ebits,timeout, though I don't think this is what you are really after.

    select(undef,undef,undef, $start - time()); #...4962.38324 #...4963.39886 #...4964.41448

    If you just plug in your calculated difference that gets closer. But again, I don't think thats really what you are after.

    select(undef,undef,undef, 0.9993); #output #....5554.40476 #....5555.40476 #....5556.40476

    I was wondering if there was a solution that could be derived from using the complements of the numbers, as I have recently been looking at these. I got as close as 0.00129 difference from a second, on the first iteration. Whether its truly to spec, or could be refined, I am not sure, though it looks like something might be there.

Re: The most precise second (timer)
by haukex (Chancellor) on Nov 27, 2019 at 12:13 UTC
    The interval must be as precise as possible.

    I think that this is causing some confusion in this thread, and we should take a moment to define the problem better. See e.g. Accuracy and precision - as several people have commented, if you want precise measurement of time, then really the only way to go about it is a real-time OS and/or other custom circuitry connected to an atomic clock, high-quality GPS receiver, etc. However:

    We have monitoring scripts that need to be run every second. The scripts execution takes longer than a second, but that is not a problem. ... I know that the preciseness depends on CPU, RAM, CMOS etc. I donít care about these.

    This, along with the code loop you showed, leads me to believe that precision isn't the central point of your question, it seems to me more like you're worried about a systematic and cumulative error, which would mean you're actually more worried about trueness.

    In other words: You want something like for (1..100) { sleep( 1 + $correction_factor ) } to take as close to 100 wall clock seconds as possible. And at the same time, perhaps it's not so important to you if each delay has e.g. Ī0.1 seconds precision, as long as they even out, in other words, it's ok if 10 of the delays take 1.1 seconds, as long as 10 other delays take 0.9 seconds.

    Is this interpretation of your problem correct? In that case, IMHO sleep is not the right tool for the job, as I showed you'll need something that works with the computer's wall clock time.

    And keeping the computer's wall clock time in sync with real-world time is a whole different can of worms! But in general that's what NTP is for, and if you're having trouble with that, as you seem to indicate, then perhaps it's best to look at that as a separate topic. (Update: Actually, on a second reading, apparently it's the corrections made to the system clock by NTP that seem to be giving you the trouble? That seems strange, if you're really running chronyd and no other NTP client on this machine. But in any case, everything above still stands.)

      haukex, youíre the closest to understanding my problem and possibly to give me a solution!

      Well, on the chronyd stuff: I am running chronyd service, but not running ntpd (which is not even installed on particular system), see bellow.

      $ systemctl status chronyd &#9679; chronyd.service - NTP client/server Loaded: loaded (/usr/lib/systemd/system/chronyd.service; enabled; v +endor preset: enabled) Active: active (running) since Wed 2019-10-23 13:38:15 CEST; 1 mont +hs 4 days ago Docs: man:chronyd(8) man:chrony.conf(5) Main PID: 8102 (chronyd) CGroup: /system.slice/chronyd.service &#9492;&#9472;8102 /usr/sbin/chronyd Warning: Journal has been rotated since unit was started. Log output i +s incomplete or unavailable. $ systemctl status ntpd Unit ntpd.service could not be found.

      I still need to delve into the chronyd docs.

      Is this interpretation of your problem correct?

      Well, kind of. Weíd like to monitor some stuff each and every second. In the perfect world, this would mean that each and every real second a script would run, get data and insert in into the DB. As we donít want/need to use atomic clock (etc), therefore by the real second we mean a second as precice accurate as possible. Ö In the end, you are probably right about the interpretation. ;)

      So if sleep is not the right tool for the job, what would it be? And why does the EV (my $w2 = EV::periodic( 0, 1, 0, sub { print Math::BigFloat->new( sprintf("%d.%06d",gettimeofday) ), "\n";} );) seconds diff is greater than the original code (below is simplified version of the original code; on my computer it has diff bellow 0.0004, sometimes the avg is 0.0002)?

      use strict; use warnings; use Time::HiRes qw/time sleep/; sub negative_timer { my $max = $_[0]; my @time_test; my $start = time(); for (my $i = 0; $i < $max; $i++) { $start += 1; sleep $start - time(); $time_test[$i] = time(); } return @time_test; }
        I am running chronyd service, but not running ntpd

        Yes, AFAIK the two are mutually exclusive. I updated my node here to suggest some of the chrony configuration directives to look at that might be relevant for clock jumps.

        So if sleep is not the right tool for the job, what would it be?

        I checked, and interestingly, libev uses nanosleep(2) (or select(2)), which is apparently quite similar to what Time::HiRes's usleep does - so the method actually isn't as fundamentally different from your approach as I initially thought (but Perl's built-in sleep is of course too inaccurate). However, in libev, the calculations for the next event time happen at the C level, so I think it could be a little bit more accurate than if you do the same on the Perl level.

        And why does the EV ... seconds diff is greater than the original code (below is simplified version of the original code; on my computer it has diff bellow 0.0004, sometimes the avg is 0.0002)?

        The difference might just be from inaccuracies in the calculation of the sleep time, or the time period over which you measured (e.g. there might have been different jitter during the two). I ran a test where I ran my code from here over almost an hour, with the addition that I wrote out the timestamp for each time the event fired. Although my gettimeofday on each event was on average 0.0017s after the whole second (max 0.0021), I could see no signs of a cumulative error - perhaps it would be best if you run your own tests over a longer period of time (like a day) to see if it suits your needs.

        Update: Oh, and Average over 3312 deltas: 1.00001844328777194271613548903407496361, I'd hope you'd see this number shrink over an even longer test. Also minor edits.

Re: The most precise second (timer)
by duelafn (Vicar) on Nov 26, 2019 at 15:22 UTC

    Just save the value and test before sleeping.

    use strict; use warnings; use Time::HiRes qw/time sleep/; sub negative_timer { my $max = $_[0]; my @time_test; my $start = time(); for (my $i = 0; $i < $max; $i++) { $start += 1; my $dt = $start - time(); if ($dt > 0) { sleep $dt; $time_test[$i] = time(); } } return @time_test; }

    Good Day,
        Dean

      Thanks, Dean! I thought about doing that, but that would slow the calculation a bit, and thus the calculated second would differ a bit more.

      Isnít there a way to accomplish this by addition (incrementing the time by a second) instead of substraction (decreamenting the time by a second)?

      Update

      As I have said, after some tests, the average difference from a real second is -0.000003595544834, which is about 0.000003513663706 more than the average difference in the original code.

      I know this is still somewhat irrelevant difference, but we need as precise second as we can get, and since the original version of the code is more precise (by that much), this new code does not solve our problem.

Re: The most precise second (timer)
by jcb (Deacon) on Nov 27, 2019 at 23:14 UTC

    Instead of time(), have you tried clock_gettime(CLOCK_MONOTONIC) from Time::HiRes? The monotonic clock isn't adjusted by NTP and simply counts from some unspecified time in the past.

Re: The most precise second (timer)
by Anonymous Monk on Nov 26, 2019 at 16:06 UTC
    Check the actual duration of a sleep second on your system:
    perl -MTime::HiRes=sleep -E 'say sleep 1'

      Here you go. This is even worse.

      $ time perl -MTime::HiRes=sleep -E 'say sleep 1' 1.000147 real 0m1.139s user 0m0.006s sys 0m0.008s

        You miss the point of this test. Anonymous Monk's point is not to suggest calling out to an external perl. The point is that sleep() itself has inherently imprecise. You requested sleep(1) but actually got 1.000147 seconds of sleep. Therefore fussing over 0.00000XXXX seconds of "error" is futile.

        Good Day,
            Dean

Re: The most precise second (timer)
by Anonymous Monk on Nov 27, 2019 at 00:55 UTC
    You are probably at this point asking for a "real-time operating system" which you don't have. If a difference of 0.000000676723442 actually matters to you, a Perl interpreter running in a conventional operating system might not be able to deliver, period.
Re: The most precise second (timer)
by GrandFather (Sage) on Dec 03, 2019 at 03:58 UTC

    If the concern is first ensuring errors don't accumulate and second an emphasis on minimum variation in period then the following code should help:

    use strict; use warnings; use Time::HiRes qw/time sleep/; my $nextTick = int (time()); while (1) { $nextTick += 1.0; # We want to stuff again in 1 second my $wait = 0.99 - ($nextTick - time()); # allow 10ms slop in sleep + period sleep $wait if $wait > 0.0; 1 while time() < $nextTick; # spin until time is up RunStuff(); } sub RunStuff { # kick off required sub tasks here print time(), "\n"; }

    The key is to calculate when the next interesting epoch is, then sleep for most of the time until then. The 1 while ... loop is a busy wait until the next epoch time has arrived.

    Optimising for fewest key strokes only makes sense transmitting to Pluto or beyond
Re: The most precise second (timer)
by misc (Friar) on Nov 30, 2019 at 17:30 UTC
    You do not explain, why the script has to be run exactly every second.

    Anyways, you should probably keep in mind, that you're about to write a fork bomb.

    When the server for whatever reason get's busy, and the spawned script already takes longer than a second for execution,
    as you write, -> the script will take even longer -> more and more scripts are spawned, burden the server even more -> voila, k'boom.

    Beside of this, I guess, best way for really exact timing would be spawning the script,
    which now wait's within a spinlock for the current second to change.

    -> the delay of spawning the script is gone, it's already in action, cpu scaling won't matter, and so on.

    You also don't write, which platform you're on.
    So which timer you are able to use is another question.
    afaik, without looking it up, the most exact "timer" is the cpu tick counter.
    So you probably have to write the spinlock in C.
      Reading again, and guessing...

      The problem is the cumulative error,
      NOT the exact timing.

      So, forget my hint at the cpu tick counter.
      (Which would be the most exact "timer" at the average system..)

      The solution is easy, just do (Pseudocode)
      $oldtimestamp = timestamp(); nanosleep(0.9seconds); while ( $oldtimestamp == timestamp() ) { nop; } spawnmyself(); dothemysterious();

      This way in, say, 1 hour exactly 60*60 instances are going to be spawned.
      good luck
        Anyways, you should probably keep in mind, that you're about to write a fork bomb.

        I see what you mean. Original code (not published here) had a check for that (see below) which is executed in the loop as launch(@ARGV). And if a Bash script should run periodically, we have a special Bash script to check if only one instance of a particular script is running. Although I must say, we are currently in the middle of porting of the Bash scripts to Perl.

        sub launch { return if fork; exec @_; die "Couldn't exec"; } $SIG{CHLD} = 'IGNORE';

        For Perl scripts, we use the Highlander function to eliminate the forks. This, however, could effectively kill the processes we want to run each second (n seconds), when one instance is running longer that the period.

        And generally, we need n * second intervals. Some scripts run each second, others every 5 seconds. I have created this question to find a way to run the scripts every n seconds most accurately.

        Beside of this, I guess, best way for really exact timing would be spawning the script, which now wait's within a spinlock for the current second to change.

        This is interesting to me, but I am not a Perl guru (not even a pro programmer), therefore Iíd like to ask you for more info what exactly is the spawning and the spinlock (although for this I should open another question I think). Thanks. :)

        You also don't write, which platform you're on.

        We run the scripts (and Perl) on CentOS exclusively. No need for portability.

Re: The most precise second (timer)
by harangzsolt33 (Pilgrim) on Dec 04, 2019 at 16:08 UTC
    If you're on Windows, you could write a JavaScript application that calls perl every second.

    Save this as "something.hta" on your desktop and open it in Notepad. You will have to edit the location of the perl interpreter and the location of your perl script within the code. And then simply double-click on the JavaScript program and click on Start. This can also display the exact millisec when perl was executed, but I've noticed that it's not exactly 1 second. It's always 10 ms more. But you can adjust for that by calling the script every 990ms instead of 1000.

    <HTML> <BODY onLoad="Init();"> <INPUT TYPE=BUTTON VALUE="START" onClick="Start();"> <INPUT TYPE=BUTTON VALUE="STOP" onClick="Stop();"> <SCRIPT> // I have tested this. It works. function Init() { WScript = new ActiveXObject("WScript.Shell"); RUNNING = 0; document.title = "INACTIVE"; } function RunPerl() { /* var T = new Date(); document.title = T.getTime(); // SHOW MILLISECONDS IN THE WINDOW TI +TLE BAR */ WScript.Run('C:\\BIN\\PERL\\tinyperl.exe K:\\DESKTOP\\myscript.pl'); } function Start() { if (!RUNNING) RRR = setInterval(RunPerl, 1000); // 1000 ms delay RUNNING = 1; document.title = "RUNNING"; } function Stop() { if (RUNNING) clearInterval(RRR); RUNNING = 0; document.title = "STOPPED"; } </SCRIPT>

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://11109264]
Front-paged by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others taking refuge in the Monastery: (6)
As of 2019-12-05 22:18 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Strict and warnings: which comes first?



    Results (152 votes). Check out past polls.

    Notices?