Beefy Boxes and Bandwidth Generously Provided by pair Networks
Syntactic Confectionery Delight
 
PerlMonks  

Re: Re: Re: Missing (i)threads Features? (Some ranting involved:)

by BrowserUk (Pope)
on Aug 21, 2003 at 20:12 UTC ( #285576=note: print w/ replies, xml ) Need Help??


in reply to Re: Re: Missing (i)threads Features?
in thread Missing (i)threads Features?

Periods determined by what? Time? An event? (Un)Availability of a resource?

The way this works is:

If a thread is designated as being in a 'dynamic priority class' and 'priority boosting' is enabled, then when the thread is 'woken' after a wait state (asynchronous IO completion, timer expiry or the like), then the OS temporarially boosts it's priority.

The effect is that when whatever thread (system wide rather than process wide) that was running when the asynchronous event occured, completes it's timeslot, the thread that was waiting on that event has a higher (but not exclusive) chance of being selected as the next thread to be run by the scheduler. The threads priority returns to its unboosted level the next time it is run.

This provides a mechanism for the application to increase the probability that time-critical threads run as soon as possible once they have something to do without risking overriding higher priority threads elsewhere in the system as would be the case if this was done in a completely deterministic way.

That's a poor destription, but may be good enough.

That's easily done if you're able to determine reliably that yield() is a no-op on your platform.

I guess what I am saying here is that if a platform has a sensible alternative to a native yield(), then it would be better if that alternative was provided by the threads module by default rather than requiring an extra piece of code that every user needs to use to determine if they need to add another piece of code to work around that fact that it isn't available by default, and then force everyone to come up with their own alternative implementation.

Commensurate with that, it would be better to have the yield() function raise an exception on those platforms that do not implement it rather than needing to add on a module to detect that it is a no-op.

... on Linux with Thread::Signal...

The problem with that is that it won't work on platforms that don't support signals. It is also adding layers on top of the threads API in a way that makes it difficult to code cross-platform. The extra layers also make for very complicated debugging.

I'm not saying that it isn't the right way to do it on Linux, only that at the architectural level, if it is going to be possible to write cross-platform threaded applications in perl, then the API's available need to be integrated at higher level than can really be done by add-ons written at the perl level.

In this case, on Win32, SuspendThread and ResumeThread translate directly to native APIs. Then only thing missing is access to the native thread handle. It's actually quite easy to obtain this handle as there is a native API that will return a usable handle for the currently running thread, which may or may not be the same as the native handle stored within the threads module.

The problem comes to trying to devine what effect using this to suspend a thread externally to the threads module will have upon the perls management of that thread? Maybe if the thread is suspended, nothing untoward will happen, but maybe Perl will take umbrage at having one of it's threads rendered un-runnable without it being consulted:)

With regard to your many Thread::* modules. The problem (from my Win32 platform perspective) is that many of them rely entirely upon platform-dependant idioms like Signals. I can see how to implement most, if not all of them on the Win32 platform, but not in a way that would make them compatible with the modules you have already written. This would make life very tough for anyone trying to utilise them for cross-platform development.

This is where trying to find a platform independant iThreads API set would be useful. If (we?) could take a step back from specific implementations and try and look at what features would be useful to have and try to encapsulate those features within an API that would enable them to implemented reasonably efficiently on all (or at least most) platforms, then it might be possible to come up with something that would be truely usable.

...many tests depend on the format of die, warn and Carp...

Perhaps the solution here is for the patch(es) to die, warn and Carp to detect whether threads were being used and only use the modified form if they were? That would leave existing non-threaded testcases unaffected.

Patches welcome ;-). But not for 5.8.1...

Rather than patching the existing threads module directly, I was thinking more of coming up with a single threads::util module that would have access to the internal structures (native thread handles etc.) that encapsulated all of the new APIs. Whether that was later adopted or integrated could be left for powers-that-be to decide later on.

on what system?

Win32. I tried MingW, Borland and LCC so far. I get furthest with Borland, but its STDIO libraries don't have huge file (>4GB) support.

Theoretically, using PerlIO ought to work around this, but as currently implemented (for Win32 at least), PerlIO still relies on STDIO calls for fseek(), fstat() and ftell(). I ought to be able to work around this by modifying the Win32_... verions of the three APIs in win32.c to use the native APi equivalents rather than the C-library versions, but this is not trivial as they all require access to the

FILE *
handle that is their first parameter and this is not exposed by the C-libraries.

I've searched the web hoping that Borland or someone else would have already released extensions to accomodate huge files, without success. I've also looked to see if the source code for the libraries was available to no avail. I'm currently trying to reverse engineer the FILE * structure, but as many of the fields are bit-fields, and the structure is used ubiquitously through out the libraries and these libraries are in turn used extensively when building perl, reverse engineering it and all the constants required to access and manipulate them is a distinctly non-trivial process.

The bigggest barrier to making changes to the perl build process is the incredibly conveluted and incestuous nature of the build mechanisms.

(D|N)MAKE is used to build mini-perl, which is used to process header files into a config which is used to create makfiles which use perl to create header files which are processed using perl to .....

I think I got lost somewhere in a maze of dark twisty passages. Combine that with the conditional compilation directives in every source module to account for all the disperate platform differences and it is amazing that perl manages to build anywhere. That it is seccessful in doing so on so many platforms is truely remarkable and a testimony to losts of hard work by lots if people. The problem is that the hands of all those individuals show and the while thing is now so complex that it a huge investment of time and expertise to even begin to make inroads into the process.

I have had lots of time, and whilst my skill levels may not be upto the standards of those that precede me, I'm not exactly clueless. I've spent several months trying to get a handle on the while thing and still it defeats me.

Even where I have made so progress and have seen things that I might like to do, I am reluctant to try because I feel that making further piecemeal changes is likely to simply perpectuate and add to the problems I see. And each time someone, somewhere makes changes and additions that add to the complexity, it further removes the whole process from the 'ordinary man or woman', by which I mean thise that don't have the luxury of unlimited time and high levels of expertise to devote.

Step by step, this means that fewer and fewer people will ever be able to contribute to the perl codebase and leave more and more people dependant upon the time, skills and motivation of fewer and fewer experts. From my persepctive, this is tantamount to the death-knell for an open source project.

I think I understand why Perl 6 is going for a clean sweep rather than trying to further extend the existing. I think that the greatest service that could be performed for the Perl 5 source is for someone to discard the existing build process and start again. I've had 2 or 3 attempts at starting this process, but you need to be a computer to be able to resolve all the paths and inter-dependancies. Trying to unwind them is nearly impossible for a human being--at least it is for this human being. I'm afraid I've had to give up on that approach.

Please excuse any typos, but hopefully everything is understandable. Life is too short to pander to anal-retentive pedants who specialise in braying about the syntax instead of concentrating in the symantics.


Examine what is said, not who speaks.
"Efficiency is intelligent laziness." -David Dunham
"When I'm working on a problem, I never think about beauty. I think only how to solve the problem. But when I have finished, if the solution is not beautiful, I know it is wrong." -Richard Buckminster Fuller
If I understand your problem, I can solve it! Of course, the same can be said for you.


Comment on Re: Re: Re: Missing (i)threads Features? (Some ranting involved:)
Select or Download Code
Re: Re: Re: Re: Missing (i)threads Features? (Some more ranting:)
by liz (Monsignor) on Aug 22, 2003 at 16:06 UTC
    ...that event has a higher (but not exclusive) chance of being selected as the next thread to be run by the scheduler. The threads priority returns to its unboosted level the next time it is run.

    Makes sense. Seems like a nice feature to have (and be able to support from Perl).

    ...if that alternative was provided by the threads module by default rather than requiring an extra piece of code that every user...

    I agree. But I'm trying to do the next best thing, namely doing it in such a way that _if_ it would get fixed in the threads module in the future, my Thread::xxx modules would effectively become no-ops.

    The problem with that is that it won't work on platforms that don't support signals.

    If we could get something similar going on those other platforms, that would fine also. Fake signal delivery on Win32 ;-).

    t is also adding layers on top of the threads API in a way that makes it difficult to code cross-platform. The extra layers also make for very complicated debugging.

    If it were possible another way, I would be for that. But debugging threaded applications is complicated anyway. A Heisenbug is very easily made.

    In this case, on Win32, SuspendThread and ResumeThread translate directly to native APIs. Then only thing missing is access to the native thread handle.

    But do SuspendThread and ResumeThread know about Perl? There's SIGSTOP and SIGCONT on *nix, but I didn't use that for Thread::Suspend because I was afraid that Perl would get stuck somewhere in the middle of something vital.

    It's actually quite easy to obtain this handle as there is a native API that will return a usable handle for the currently running thread, which may or may not be the same as the native handle stored within the threads module.

    Would this value somehow be shareable?

    The problem comes to trying to devine what effect using this to suspend a thread externally to the threads module will have upon the perls management of that thread? Maybe if the thread is suspended, nothing untoward will happen, but maybe Perl will take umbrage at having one of it's threads rendered un-runnable without it being consulted:)

    Indeed. That's why I'm not using SIGSTOP and SIGCONT on *nix.

    With regard to your many Thread::* modules. The problem (from my Win32 platform perspective) is that many of them rely entirely upon platform-dependant idioms like Signals.

    The only real platform dependent one (right now) is Thread::Signal. But the input to Thread::Signal->signal are a thread ID (which is also available with threads on Win32) and a string. Surely this should be portable to Win32 somehow?

    I can see how to implement most, if not all of them on the Win32 platform, but not in a way that would make them compatible with the modules you have already written. This would make life very tough for anyone trying to utilise them for cross-platform development.

    I actually think it would be much easier than we think. The most important thing is to get Thread::Signal (or something similar) going on a thread independent manner.

    This is where trying to find a platform independant iThreads API set would be useful. If (we?) could take a step back from specific implementations and try and look at what features would be useful to have and try to encapsulate those features within an API that would enable them to implemented reasonably efficiently on all (or at least most) platforms, then it might be possible to come up with something that would be truely usable.

    idea++

    Perhaps the solution here is for the patch(es) to die ,warn and Carp to detect whether threads were being used and only use the modified form if they were?

    That sounds easier than it is, I'm afraid ;-(

    Rather than patching the existing threads module directly, I was thinking more of coming up with a single threads::util module that would have access to the internal structures (native thread handles etc.) that encapsulated all of the new APIs. Whether that was later adopted or integrated could be left for powers-that-be to decide later on.

    One very important reason for me to make so many seperate modules, is that you can mix and match easily. And that's very important in the current Perl threads implementation, as all of the data-structures are copied between threads (and stashes are data-structures ;-).

    ...Win32. I tried MingW, Borland and LCC so far. I get furthest with Borland, but its STDIO libraries don't have huge file (>4GB) support.

    Have you taken this up with p5p?

    The biggest barrier to making changes to the perl build process is the incredibly conveluted and incestuous nature of the build mechanisms.

    I think no one will disagree with that. Maximum portability comes at a price. Patches are welcome. You might want to contact H. Merijn Brand (configure pumpking and fellow Amsterdam PerlMonger) if you have questions about the build process. He is in currently streamlining the configure process.

    I have had lots of time, and whilst my skill levels may not be upto the standards of those that precede me, I'm not exactly clueless. I've spent several months trying to get a handle on the while thing and still it defeats me.

    Have you asked questions on p5p? There are a lot of people there that might be able to help you.

    Step by step, this means that fewer and fewer people will ever be able to contribute to the perl codebase and leave more and more people dependant upon the time, skills and motivation of fewer and fewer experts. From my persepctive, this is tantamount to the death-knell for an open source project.

    That is why things are being done differently for Parrot, Ponie and Perl6. That is more a community effort from the beginning as opposed to a personal project with community support bolted on afterwards.

    ...someone to discard the existing build process and start again. I've had 2 or 3 attempts at starting this process, but you need to be a computer to be able to resolve all the paths and inter-dependancies. Trying to unwind them is nearly impossible for a human being--at least it is for this human being. I'm afraid I've had to give up on that approach.

    I think you really should contact H. Merijn Brand... ;-)

    Please excuse any typos, but hopefully everything is understandable. Life is too short to pander to anal-retentive pedants who specialise in braying about the syntax instead of concentrating in the symantics.

    No problem about that from my end ;-)

    Liz

      We are taking further discussion of this elsewhere. If anyone is interested in following or contributing to the discussion, please contact one of us and we will include you.


      Examine what is said, not who speaks.
      "Efficiency is intelligent laziness." -David Dunham
      "When I'm working on a problem, I never think about beauty. I think only how to solve the problem. But when I have finished, if the solution is not beautiful, I know it is wrong." -Richard Buckminster Fuller
      If I understand your problem, I can solve it! Of course, the same can be said for you.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://285576]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others meditating upon the Monastery: (3)
As of 2014-09-21 06:25 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (166 votes), past polls