I'm using LWP::UserAgent to interact with a third party search engine. The full list of searchable items is about 1500. If I restrict the search to a small number of results the delay is manageable. However the more items the more lengthy the delay.
If I go to the page in my browser the search is actually much more rapid, the results are printed but each result has an image and it takes a long time for the page to complete loading.
I'm wondering if the delay on printing the results via LWP is because the server waits for activity to stop before it prints the results. In which case is it possible to stop the script prematurely and print any output to screen - I could then try to balance the times to achieve a happy medium.
Or if there are other things that might be tried...