Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot
 
PerlMonks  

LWP::UserAgent lengthy download

by jonnyfolk (Vicar)
on Feb 17, 2010 at 06:15 UTC ( #823624=perlquestion: print w/ replies, xml ) Need Help??
jonnyfolk has asked for the wisdom of the Perl Monks concerning the following question:

I'm using LWP::UserAgent to interact with a third party search engine. The full list of searchable items is about 1500. If I restrict the search to a small number of results the delay is manageable. However the more items the more lengthy the delay.

If I go to the page in my browser the search is actually much more rapid, the results are printed but each result has an image and it takes a long time for the page to complete loading.

I'm wondering if the delay on printing the results via LWP is because the server waits for activity to stop before it prints the results. In which case is it possible to stop the script prematurely and print any output to screen - I could then try to balance the times to achieve a happy medium.

Or if there are other things that might be tried...

Comment on LWP::UserAgent lengthy download
Re: LWP::UserAgent lengthy download
by Anonymous Monk on Feb 17, 2010 at 07:08 UTC
    Browsers have this ability to avoid downloading images ....
      But is this available in LWP::UserAgent - I can't find reference to it.

        If you don't make an explicit request for an image (URL), LWP::UserAgent won't download the image. That's how HTTP works.

Re: LWP::UserAgent lengthy download
by BrowserUk (Pope) on Feb 17, 2010 at 07:19 UTC
    . If I restrict the search to a small number of results the delay is manageable. However the more items the more lengthy the delay.

    If you can restrict the search, can you not split the required search into several restricted searches and do them serially? That'd allow you to display the intermediate results as you get them, and so reduce the pregnant pause up front. Possibly at the cost of taking slightly longer overall.


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

      That's an interesting idea, thank you. How would you display further results? One way might be to print the first split to screen, and the rest to a file, putting a link on the screen results. Did you have something else in mind?

        I assume from your question that you are representing the data you're fetching with LWP to a browser.

        I've made substantial use of LWP, but done very little with cgi, so I'm not the right person to answer that question.

        I have vague memories of something called multi-part data--and wikipedia kicks up this which may be related--but I know little about it.


        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://823624]
Approved by planetscape
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others romping around the Monastery: (13)
As of 2014-07-23 14:30 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite superfluous repetitious redundant duplicative phrase is:









    Results (144 votes), past polls