Beefy Boxes and Bandwidth Generously Provided by pair Networks
Problems? Is your data what you think it is?
 
PerlMonks  

Controlling a non-perl GUI

by dbae (Beadle)
on Mar 28, 2010 at 13:17 UTC ( [id://831453]=perlquestion: print w/replies, xml ) Need Help??

dbae has asked for the wisdom of the Perl Monks concerning the following question:

My perl expertise is limited---I've never used a perl module before, for example. So I would welcome advice in the form of pointers to online documentation, or to books, to help me with the following task.

An XP computer runs a program that I will call MetaRobot.To run a robotically controlled microscope, one needs to persuade MetaRobot to "read" a program that I will call AssemblyRobot, because writing it is like writing in assembly language. On firing up, MetaRobot produces a GUI with boxes and menus and other GUI stuff. This GUI is used first to construct (write) AssemblyRobot, and then to run AssemblyRobot. Neither MetaRobot nor AssemblyRobot have anything to do with perl. All interaction with MetaRobot is via standard GUI operations---mouse clicks in certain boxes, menu selection. keyboard strokes---and the manufacturers intend it for human use (Brave New World's delta class). The only interaction with AssemblyRobot is writing it and executing it.

The problem is that AssemblyRobot is at a very low level. One needs thousands of lines to tell the robot to do something fairly simple. Writing AssemblyRobot is error-prone and extremely time-consuming. The manufacturer of the robot is prepared to make things easier for us, but at an expense that we can't meet, and in an impossibly long time-frame. Furthermore, I would expect low quality software from the manufacturer and a refusal to let us see the source code.

I would like to write a perl meta-program Robot.pl that interacts with MetaRobot to produce AssemblyRobot. It seems that this has to be done by persuading the XP machine that certain mouse operations have been performed and certain keys on the XP machine depressed, this as a substitute for human actions. I plan to make low level parametrized functions, and build up to more complex functions that call the lower level functions. My idea is to test from the bottom level up, as the lower level functions are written. I hope that what I write can be put to use immediately as it's written, because it seems that the whole perl project will be quite lengthy, particularly in view of my other commitments.

My questions are: is this feasible and is Perl a reasonable way to go? Which perl modules are relevant, and how can I conveniently learn to use these modules? I also have to learn how to download modules, and the modules on which they depend. Can you direct me to documentation that is preferably at my level? I have written quite a bit of perl over the years, but "contact with the outside world" has been limited to getting data from files and writing to files.

Replies are listed 'Best First'.
Re: Controlling a non-perl GUI
by Corion (Patriarch) on Mar 28, 2010 at 13:43 UTC

    The preferrable way of controlling another GUI application is through Win32::GuiTest, which can send mouse clicks and, more recommended, keystrokes to any application, and also read the screen content of the target application.

    Personally, I would try first to synthesize AssemblyRobot programs by starting off with some simple primitives like "reset robot position", "plot straight line 1 unit in length", "plot straight line 10 units in length", "plot spline", "plot circle", and then looking at the files generated for these programs trying to concatenate these programs. But it might also be suitable to (ab)use the "IDE" to create AssemblyRobot programs.

    Installing Win32::GuiTest is fairly easy. Either you use the ppm command to install it (preferred way for ActiveState Perl), or you use the cpan command to install it (preferred way for Strawberry Perl).

      Can I do this: Make a short AssemblyRobot program by using clicks and keystrokes etc, while some perl module is recording my actions in Win32:GuiTest format? That would seem like an easy, lazy way to do the programming. Then I could see where the parameters occur, and easily edit the record to make a perl subroutine with variables. If not with Win32:GuiTest, could I work with some other perl module?
        Win32::GuiTest includes a recorder program, Recorder/Win32GuiTest.exe for that purpose
Re: Controlling a non-perl GUI
by moritz (Cardinal) on Mar 28, 2010 at 13:43 UTC
Re: Controlling a non-perl GUI
by BrowserUk (Patriarch) on Mar 28, 2010 at 17:45 UTC

    Once you've contructed a program for your microscope with MetaRobot, you can save it somewhere so that you can re-run it on another day? I mean, you don't have to re-create every program from scratch every time you want to use it do you?

    If so, then I would urge you to consider reverse engineering the file format in which the programs are saved.

    In the general case, it will be far easier to write programs to that storage format, than to automate the gui in a consistant and reliable way.


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

      I need to explore MetaRobot more comprehensively. One can certainly save an AssemblyRobot program, and currently our MetaRobot lists about 20 such AssemblyRobot programs. Using XP search facilities, I have looked for files on the system whose names contain the names of AssemblyRobot programs listed by MetaRobot. Result: zilch. So I'm not sure where and how MetaRobot is hiding these programs. To find out what is in the programs, one descends a menu hierarchy,noting what's there at each stage. But even if all this information was neatly written out in a textfile, one would be infinitely far from having something that would drive the robot.

      As far as I know, the only ways to make the robot do something are 1) select a pre-existing, saved AssemblyRobot program and tell MetaRobot to execute that or 2) to use clicks and keystrokes to construct an AssemblyRobot program from scratch or 3) copy an existing AssemblyRobot program and alter it using clicks and keystrokes.

      It doesn't seem possible to find out the format in which the programs are saved, or to obtain any examples. I suspect the manufacturers think they can force people to pay for something better, and, in the meantime, are keeping everything as secret as they can.

      However, as I type this, I'm thinking that maybe I can create a new very short AssemblyRobot program, and then asking XP to find all files created in the past 10 minutes. My strong suspicion is that, if the new AssemblyRobot program has been saved in a separate file, then it will be in binary. I have no idea how I would reverse engineer a binary file. Is it possible? I think that would be harder than getting the perl module to do the job.

      But I will bear in mind what you say and make another fruitless attempt to get cooperation from the manufacturer.

        Were I attempting this, my starting point would be to install ProcessExplorer. This is a TaskManager replacement that has a built-in facility for monitoring what files are being accessed by running processes. By enabling "Show lower pane" and selecting "Lower pane->view handles" (both from the View menu), you can click on the running appication and see what files it is accessing. Realise that it might not put the programs into separate files. It might store them in a single file, or in the registry somewhere.

        It won't be easy to reverse engineer, but if you can locate where the programs are stored and then: start with something simple, take a copy; add one thing, take another copy; and one more thing and take another copy. it should be reasonably easy to build up a picture of the way the programs are stored, and build up a library of routines for generating the required code for each type of action.

        The problem with driving GUIs is that the windows and dialogs involved have a tendancy to move around the screen each time they are displayed. And that makes automating complex interactions, laborious and fragile. If you do have to go this route, then you might consider trying something like AutoIt which is designed for exactly this type of task.


        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
Re: Controlling a non-perl GUI
by kikuchiyo (Hermit) on Mar 29, 2010 at 09:52 UTC
    Trying to attack the problem from a different direction:

    What kind of interface does the microscope use to connect to the PC?

    If it is serial (standard RS-232 or some bastardization of it), chances are that it uses simple ASCII commands to drive the microscope. You could capture and reverse-engineer those commands more easily than a proprietary binary file format.

    Even if it is USB, those are sometimes realized with a serial-to-USB converter inside or outside the unit, and even if it is a proprietary USB-based protocol, you might have some chance of deciphering it.


    Remote controlling a GUI that was not designed to be remote controlled seems like a really desperate last ditch effort to me (i.e. resort to this only if all else fails).

      I'm taking what you and BrowserUK say very seriously, and I find what you say interesting and helpful, potentially saving me many days (weeks?) of wasted labour. My next step will be to install ProcessExplorer, as recommended by BrowserUK, and hope that what I find is comprehensible.

      I'm not sure that capturing the output from the computer to the robot will be that helpful. Maybe what the robot sends back is also important. It sounds foolhardy to try to control $300K worth of equipment without knowing what I'm doing. Even if I don't destroy the machine, I'm sure I would destroy the guarantee. So, if I can't get anywhere with ProcessExplorer, I will try what you call the last ditch approach, unless a better suggestion arises. Nevertheless, I find your lateral thinking very interesting and stimulating.

      Do your odds on my success with remote controlling the GUI improve when I report that the GUI screen covers the whole computer screen, so there is no chance of it moving or wobbling?

        Do your odds on my success with remote controlling the GUI improve when I report that the GUI screen covers the whole computer screen, so there is no chance of it moving or wobbling?

        Does it ever pop-up dialogues whilst you are programming it? If you make mistakes?

        Are the menus contextual? Ie. Do the options available on the menubar, or within the dropdowns change at different points within the process of programming?


        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
        The point of not wanting to mess aroung with equipment worth $300k is a valid one, I can understand your sentiment.

        In that case, further questions: does this AssemblyRobot language have a clear text representation that is displayed in a box somewhere in the GUI? Does this box - or the program you call MetaRobot in general - accept text via copy/paste? I guess it doesn't or you would have had a solution already. But even if it doesn't, having such a text representation would make the situation easier, as you could assemble the necessary text sequences outside this horrid GUI and then feed it - keystroke by keystroke if needed - via some of the automation of GUI remote control methods mentioned earlier in the thread.

        I can't help but say that I find the attitude of your microscope vendor rather petty and close-minded. Instead of making their interface, file formats etc. versatile, open and well-documented thus more useful to the advanced user, they adamantly try to close off everything that does not conform to their "intended" modes of usage. Hiding so desperately behind the veil of closed source hints at ineptitude - that is their code is so horrible that they don't dare to show it to anyone.
        I'm not sure that capturing the output from the computer to the robot will be that helpful

        Not necessarily. A colleague programmed two types of microscopes equipped with moving stages (Nikon and Mac2000), long time ago; both had more-or-less comprehensible command set. I'm not sure though if the set was grabbed through I/O dumping or he found manual somewhere (probably the latter), but f.ex. setting movement speed was as easy as sending "SPEED x=100\r" down the line.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://831453]
Approved by moritz
Front-paged by ww
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others avoiding work at the Monastery: (4)
As of 2024-03-29 06:03 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found