http://www.perlmonks.org?node_id=808830

whale2 has asked for the wisdom of the Perl Monks concerning the following question:

Hello monks.

I'm looking for any standard way to convert a timestamp from one timezone to another inside perl thread.

I have no problem if this happens in main thread, but if I try to change $ENV{TZ} in any other thread, this gives nothing.

Script to demonstrate the problem:

#!/usr/bin/perl use strict; use warnings; use threads; use POSIX qw(tzset); my $t=localtime(); print "local time: ",$t,"\n"; my ($tr) = threads->create(\&thr_proc); $tr->join; $ENV{TZ}="Europe/Paris"; $t=localtime(); print "main: ",$t,"\n"; $ENV{TZ}="Europe/Moscow"; $t=localtime(); print "main: ",$t,"\n"; sub thr_proc { $ENV{TZ} = 'Europe/Paris'; $t = localtime; print "in thread: ",$t,"\n"; $ENV{TZ} = 'Europe/Moscow'; $t = localtime; print "in thread: ",$t,"\n"; }
Output:
$ ./mt-demo.pl local time: Mon Nov 23 15:14:06 2009 in thread: Mon Nov 23 15:14:06 2009 in thread: Mon Nov 23 15:14:06 2009 main: Mon Nov 23 16:14:06 2009 main: Mon Nov 23 18:14:06 2009
I tried adding of tzset() or some other tricks but with no luck. Here is my perl:
$ perl -V Summary of my perl5 (revision 5 version 10 subversion 0) configuration +: Platform: osname=linux, osvers=2.6.18-128.7.1.el5xen, archname=x86_64-linux- +thread-multi uname='linux testbox-blizzard.didit.com 2.6.18-128.7.1.el5xen #1 s +mp mon aug 24 09:14:33 edt 2009 x86_64 x86_64 x86_64 gnulinux ' config_args='-Duseshrplib -Duseithreads -d' hint=recommended, useposix=true, d_sigaction=define useithreads=define, usemultiplicity=define useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=und +ef use64bitint=define, use64bitall=define, uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing + -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=6 +4', optimize='-O2', cppflags='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -I +/usr/local/include' ccversion='', gccversion='4.1.2 20080704 (Red Hat 4.1.2-46)', gcco +sandvers='' intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=1 +6 ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', + lseeksize=8 alignbytes=8, prototype=define Linker and Libraries: ld='cc', ldflags =' -L/usr/local/lib' libpth=/usr/local/lib /lib /usr/lib /lib64 /usr/lib64 /usr/local/l +ib64 libs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc perllibs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc libc=/lib/libc-2.5.so, so=so, useshrplib=true, libperl=libperl.so gnulibc_version='2.5' Dynamic Linking: dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E - +Wl,-rpath,/usr/local/lib/perl5/5.10.0/x86_64-linux-thread-multi/CORE' cccdlflags='-fPIC', lddlflags='-shared -O2 -L/usr/local/lib' Characteristics of this binary (from libperl): Compile-time options: MULTIPLICITY PERL_DONT_CREATE_GVSV PERL_IMPLICIT_CONTEXT PERL_MALLOC_WRAP USE_64_ +BIT_ALL USE_64_BIT_INT USE_ITHREADS USE_LARGE_FILES USE_PERLIO USE_REENTRANT_API Built under linux Compiled at Nov 16 2009 16:44:01 @INC: /usr/local/lib/perl5/5.10.0/x86_64-linux-thread-multi /usr/local/lib/perl5/5.10.0 /usr/local/lib/perl5/site_perl/5.10.0/x86_64-linux-thread-multi /usr/local/lib/perl5/site_perl/5.10.0
I'm ready to write some thread-safe TZ manipulation module, but maybe I'm missing something obvious. Thanks.

Replies are listed 'Best First'.
Re: Standard way to convert timezone in multithreaded script
by BrowserUk (Patriarch) on Nov 23, 2009 at 17:43 UTC

    This is a case of POSIX standard hysteresis.

    Your CRTs idea of the current TZ setting is determine at startup and then buffered. Changing the value of TZ in your process' (or thread's) copy of the environment won't change the CRT's buffered notion of the timezone directly. After the change you need to request that the CRT re-evaluate it. The POSIX tzset() call is defined to do that. It may work for you on *nix.


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
Re: Standard way to convert timezone in multithreaded script
by Fletch (Bishop) on Nov 23, 2009 at 15:50 UTC

    Perhaps use DateTime (rather than time_t values and localtime) to represent your times and use its zone conversion capabilities?

    The cake is a lie.
    The cake is a lie.
    The cake is a lie.

      Thanks for your suggestion, but I found some time ago that using DateTime for parsing and manipulating timestamps is not very fast. My tests shown that it is tens times slower than combination of split() and POSIX::mktime.

      I'm writing my things multithreaded to gain some speed by using several cores of CPU, so returning to DateTime would be serious step back from the performance point of view.

        I'm writing my things multithreaded to gain some speed by using several cores of CPU,

        ....i don't see exactly what your problem is with the timestamp....it prints out....what do you expect it to do?

        ...as far as using Perl threads to gain a performance benefit, you may be barking up the wrong tree...... Perl threads are not as solid as c pthreads, and you may actually find that threading slows you down.... if you search the nodes here, or google it, you will find that Perl threads don't give you speed thru concurrency..... in Perl the main thread gets a priority, and the time slice gets divided amoung the it's threads..... no concurrency..... just a thought before you start banging your head on the wall over thread speed ... :-)

        ...additionally, i don't think Perl has any capability to tell the system to utilize different cpu's, for each thread, that's the kernel's job


        I'm not really a human, but I play one on earth.
        Old Perl Programmer Haiku
        You've already shown that POSIX isn't thread safe in that respect. I don't know what you're expecting from us if you're dismissing the alternatives.
Re: Standard way to convert timezone in multithreaded script
by zentara (Archbishop) on Nov 23, 2009 at 20:19 UTC
    .... regardless of BrowserUk's belief that threads, especially Perl threads, will give you some sort of performance boost ..... try it yourself and see, .....huge overheads

    ....maybe they do in MSWindows....but not under linux...and frankly, i would not use an OS that let a process running on one core of a multi-core system, alter the goings on in a neighboring core..... that is what you seem to be suggesting...... i could start a perl script, start x number of threads, and be assured that each thread (code block) will be assigned to a separate core..... with cross-core variable sharing thru threads::shared? .....bad design......

    on linux, each thread of a spawned thread set shares the same pid, and if you call exit from any thread, it takes down all threads under the master thread, including the master thread.... what more evidence do you need that they share the same pid?... and probably have their nice level set as a whole based on the master thread's pid

    ...so even though in theory, BrowserUk's assertion that Perl threads will give you concurrent processing... in all likelihood, they won't..... i would bet for security, a pid gets spawned into 1 core, and since threads on linux share the same master pid, all threads probably will be assigned to the same core.... otherwise it would be chaos as programs try to spawn threads into other cores..... maybe it is done in sloppy OS's ?..... but not to abandon the intent of my reply..... threads are good if you want easy realtime sharing of data between threads..... but they are not for speed

    ...what you want to look at is forking and using shared memory segments for the best speed.... read "perldoc perlipc" and look for the section SysV IPC.... it is the fastest

    .... and that is no baloney...i'm a vegetarian ;-)


    I'm not really a human, but I play one on earth.
    Old Perl Programmer Haiku
      i would bet for security, a pid gets spawned into 1 core, and since threads on linux share the same master pid, all threads probably will be assigned to the same core

      Can I name an amount? Because I'll take that bet :-). Sorry, but this is completely wrong, and if you think about it for a minute you'll realize that it would be completely brainless and negate the usefulness of kernel threads entirely. Yes, threads are spawned under a single PID, but they can very well use more than a single core. This is simple to demonstrate (WARNING, following code will hog your CPUs, don't run it on a production system!):

      perl -Mthreads -e'$n=3;for (1..$n){$t[$_]=threads->create(sub{my $r; while(1){$r++}});}; sleep 10;threads->exit()'

      Set $n to 1 less than the number of cores your machine has (in this case 3 for a quad-core). Then run the command and observe its behaviour in the system monitor of your choice (top will do). You'll see that it runs on the number of CPU cores specified, despite retaining a single PID. So your assumption is wrong.

      As for the usefulness/performance benefit of updating a shared variable from threads running on several cores, I'm staying firmly out of that debate. My own gut feeling would be that it is indeed not worth it on Linux, context switching Perl threads is more expensive than doing the same with processes (or at least it was the last time I measured), and a short test with the above command line using threads::shared seems to support the feeling. But the true answer will invariably depend (almost exclusively) on the concrete problem the OP is trying to solve, so he should benchmark competing solutions to that concrete problem and find out for himself.


      All dogma is stupid.
        perl -Mthreads -e'$n=3;for (1..$n){$t[$_]=threads->create(sub{my $r; while(1){$r++}});}; sleep 10;threads->exit()'

        FWIW, I can attest that setting n=2 pegs both cores on my amd 3800+ running Arch Linux. However, n=1 (recommended setting) only pegs one core.

        ... i don't have a multi-cpu system to test on, but i accept your explanation that a single process can be running things across core boundaries.... but that seems quite insecure and it would seem to be a good kernel option to have ( like in the NSA-security kernel )...why?.... because according to your explanation, any process can start injjecting code into all the other cores of a multicore machine, just by starting a few threads..... how does this affect the nice value of the parent thread... does it increase according to thread count?....... what does it mean then to give a nice value to a script?... if the script can start spawning an infinite number of threads..... or what is to stop core hijacking by me spawning a thread until it gets into the desired core.... then executing something that halts the core, or other malicious intent...oO( maybe promoting threading is good for national security ;-) )

        ... my original intent was to point out that threads are not the fastest IPC out there, and continuing in that vein, there is Why use threads over processes, or why use processes over threads? and Threads versus fork, which to use? and Using forks and threads ...then choose the IPC right for you

        ..... by the way.... you would have no such POSIX module thread-safety in a fork-and-exec


        I'm not really a human, but I play one on earth.
        Old Perl Programmer Haiku
      I has already achieved required speed boost in my script using threads. Everything is clear and pretty straightforward here. The problem I ran into is converting timestamps using POSIX routines. Now I see that this will not work in multithreaded app. But I tried to do this directly from .xs and it works. Thanks everyone for your attention :)

      I'm afraid that this post just demonstrates how little you understand about how your favorite OS kernel works. Don't feel too bad, you're far from alone in that.

        likewise,

        i know how my kernel assigns pids, and i thank God that my os dosn't confuse fork-and-exec with threading, as does your favorite ...... at least my destroyer won't be totally shut down because the gun turret is stuck


        I'm not really a human, but I play one on earth.
        Old Perl Programmer Haiku
Re: Standard way to convert timezone in multithreaded script
by whale2 (Novice) on Nov 24, 2009 at 23:16 UTC

    I finally solved my problem by writing little .xs module. I don't know could this be useful for anyone or not, but I'll post my code just in case. Also, this will look like I'm not just crying for help but providing something in exchange :)

    XS:
    #include "EXTERN.h" #include "perl.h" #include "XSUB.h" #include "ppport.h" #include <time.h> #include <stdio.h> #include <stdlib.h> MODULE = tztime PACKAGE = tztime long tz_mktime(sec, min, hour, mday, mon, year, tz_name) int sec int min int hour int mday int mon int year char *tz_name CODE: long epoch; struct tm tz_tm; char *old_tz; char *tz_p; bool restore = FALSE; tz_tm.tm_sec = sec; tz_tm.tm_min = min; tz_tm.tm_hour = hour; tz_tm.tm_mday = mday; tz_tm.tm_mon = mon; tz_tm.tm_year = year; tz_tm.tm_wday = 0; tz_tm.tm_yday = 0; tz_tm.tm_isdst = -1; tz_p = getenv("TZ"); if(tz_p != NULL) { restore = TRUE; old_tz = malloc(strlen(tz_p) - 2); // 'TZ=' is -3, '\0' is +1, + total is -2 if(old_tz == NULL) { XSRETURN_UNDEF; } sscanf(tz_p,"TZ = %s",old_tz); } setenv("TZ",tz_name,1); tzset(); epoch = (long)mktime(&tz_tm); if(restore) { setenv("TZ",old_tz,1); free(old_tz); } else { unsetenv("TZ"); } tzset(); RETVAL = epoch; OUTPUT: RETVAL void tz_gettime(epoch, tz_name) long epoch char *tz_name; PPCODE: struct tm tz_tm; char *old_tz; char *tz_p; bool restore = FALSE; tz_p = getenv("TZ"); if(tz_p != NULL) { restore = TRUE; old_tz = malloc(strlen(tz_p) - 2); // 'TZ=' is -3, '\0' is +1, + total is -2 if(old_tz == NULL) { XPUSHs(sv_2mortal(newSVnv(errno))); } sscanf(tz_p,"TZ = %s",old_tz); } setenv("TZ",tz_name,1); tzset(); localtime_r(&epoch,&tz_tm); if(restore) { setenv("TZ",old_tz,1); free(old_tz); } else { unsetenv("TZ"); } tzset(); XPUSHs(sv_2mortal(newSVnv(tz_tm.tm_sec))); XPUSHs(sv_2mortal(newSVnv(tz_tm.tm_min))); XPUSHs(sv_2mortal(newSVnv(tz_tm.tm_hour))); XPUSHs(sv_2mortal(newSVnv(tz_tm.tm_mday))); XPUSHs(sv_2mortal(newSVnv(tz_tm.tm_mon))); XPUSHs(sv_2mortal(newSVnv(tz_tm.tm_year))); XPUSHs(sv_2mortal(newSVnv(tz_tm.tm_wday))); XPUSHs(sv_2mortal(newSVnv(tz_tm.tm_yday))); XPUSHs(sv_2mortal(newSVnv(tz_tm.tm_isdst)));
    Calling script:
    #!/usr/bin/perl use strict; use warnings; use threads; use tztime; use Thread::Semaphore; my $sem = Thread::Semaphore->new; my ($tr) = threads->create(\&thr_proc); my ($tr2) = threads->create(\&thr_proc); $tr->join; $tr2->join; $ENV{TZ}="Europe/Paris"; my $t=localtime(); print "main (Europe/Paris): ",$t,"\n"; $ENV{TZ}="Europe/Moscow"; $t=localtime(); print "main (Europe/Moscow): ",$t,"\n"; sub thr_proc { $sem->down(); my @tms = tztime::tz_gettime(time,"Europe/Paris"); $sem->up(); $tms[5] += 1900; $tms[4] ++; $tms[$_] =~ s/^(.)$/0$1/ for (0..2) ; print "in thread (Europe/Paris): $tms[5]-$tms[4]-$tms[3] $tms[2]:$ +tms[1]:$tms[0]\n"; $sem->down(); @tms = tztime::tz_gettime(time,"Europe/Moscow"); $sem->up(); $tms[5] += 1900; $tms[4] ++; $tms[$_] =~ s/^(.)$/0$1/ for (0..2) ; print "in thread (Europe/Moscow): $tms[5]-$tms[4]-$tms[2] $tms[2]: +$tms[1]:$tms[0]\n"; }
    Results:
    $ ./mt-demo.pl in thread (Europe/Paris): 2009-11-24 23:27:28 in thread (Europe/Moscow): 2009-11-25 01:27:28 in thread (Europe/Paris): 2009-11-24 23:27:28 in thread (Europe/Moscow): 2009-11-25 01:27:28 main (Europe/Paris): Tue Nov 24 23:27:28 2009 main (Europe/Moscow): Wed Nov 25 01:27:28 2009
    so, now I can convert timestamps from one timezone to another quite fast and thread-safe :) Thanks to everyone for sharing your wisdom.

      Nice. And thanks for sharing it.

      FWIW: I think the reason why POSIX::tzset() fails to work correctly in threads is because when threads are spawned, they are given their own copy of the process' environment block. But the tzset() wrapper in POSIX and the underlying CRT tzset() know nothing of those copies, hence do not honour changes made to them.

      It would be possible to make the POSIX tzset() wrapper recognise that it is being called within a thread and adjust the process env block to match the calling threads prior to calling the CRT, but unless the CRT also maintained a per-thread notion of the current timezone it would be messy. One day the POSIX definition will be revamped to take account of threading.


      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
        I can't see where my XS code is aware of threads, except of calling localtime_r() instead of localtime(), but perl manuals says somewhere that if you compile perl with 'useithreads', it will automatically use those *_r() functions.