Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl: the Markov chain saw

javascript links and robots

by dannoura (Pilgrim)
on Feb 19, 2004 at 15:18 UTC ( #330225=perlquestion: print w/replies, xml ) Need Help??
dannoura has asked for the wisdom of the Perl Monks concerning the following question:


Does anyone know if there are any robots out there which can spider through javascript links? Going through the discussion boards the general consensus seems to be "not right now but probably sometime in the future".

If a robot were required to do this what command could be issued?

p.s. I'm aware of the Javascript and Javascript::Toolbox modules, but since the documentation is so sparse (and I've so far received no reply from the author) I can't figure out if they are the answer.

Replies are listed 'Best First'.
Re: javascript links and robots
by tilly (Archbishop) on Feb 21, 2004 at 02:25 UTC
    It is possible to do this, but it is impossible to do this perfectly.

    The problem is that JavaScript is a Turing-complete language. Therefore without actually executing all JavaScript that you run into, you can't always tell what it is going to do.

    That said, you can guess. One guess is that if you see anything that looks like a URL, like, then that is probably going to be reachable somehow. Likewise the same thing URI encoded is also likely to be reachable somehow. With a few heuristics like that, you'll be able to figure out things enough to be useful, but not enough to be really reliable. As you improve the reliability of the heuristics (remember, heuristic is a fancy way of saying "it doesn't work"), the problem rapidly gets harder.

    If you want more detailed advice, I'd suggest contacting cogent. He faced almost exactly your problem in a past job and had a mostly working solution (albeit he went through nightmares to get there). He has also been working on trying to parse JavaScript well enough to do better than the simple heuristic that I gave above, but I don't think that he is making enough progress. (Nudge him a little...)

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://330225]
Approved by b10m
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others browsing the Monastery: (2)
As of 2018-05-24 23:50 GMT
Find Nodes?
    Voting Booth?