Beefy Boxes and Bandwidth Generously Provided by pair Networks
Syntactic Confectionery Delight
 
PerlMonks  

javascript links and robots

by dannoura (Pilgrim)
on Feb 19, 2004 at 15:18 UTC ( #330225=perlquestion: print w/ replies, xml ) Need Help??
dannoura has asked for the wisdom of the Perl Monks concerning the following question:

hi,

Does anyone know if there are any robots out there which can spider through javascript links? Going through the discussion boards the general consensus seems to be "not right now but probably sometime in the future".

If a robot were required to do this what command could be issued?

p.s. I'm aware of the Javascript and Javascript::Toolbox modules, but since the documentation is so sparse (and I've so far received no reply from the author) I can't figure out if they are the answer.

Comment on javascript links and robots
Re: javascript links and robots
by tilly (Archbishop) on Feb 21, 2004 at 02:25 UTC
    It is possible to do this, but it is impossible to do this perfectly.

    The problem is that JavaScript is a Turing-complete language. Therefore without actually executing all JavaScript that you run into, you can't always tell what it is going to do.

    That said, you can guess. One guess is that if you see anything that looks like a URL, like http://some.domain.com/stuff, then that is probably going to be reachable somehow. Likewise the same thing URI encoded is also likely to be reachable somehow. With a few heuristics like that, you'll be able to figure out things enough to be useful, but not enough to be really reliable. As you improve the reliability of the heuristics (remember, heuristic is a fancy way of saying "it doesn't work"), the problem rapidly gets harder.

    If you want more detailed advice, I'd suggest contacting cogent. He faced almost exactly your problem in a past job and had a mostly working solution (albeit he went through nightmares to get there). He has also been working on trying to parse JavaScript well enough to do better than the simple heuristic that I gave above, but I don't think that he is making enough progress. (Nudge him a little...)

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://330225]
Approved by b10m
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others pondering the Monastery: (10)
As of 2014-12-21 16:15 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    Is guessing a good strategy for surviving in the IT business?





    Results (106 votes), past polls