I am working on scraping a website. My script is able to get to google.com but not to that particular site. I get 403 forbidden error and it doesnt generate any error logs. I have checked with the site policies and they dont restrict web scraping. My guess is i am not passing all the required parameters to the site.
Any help will be highly appreciated.
my $cookie_jar = HTTP::Cookies->new;
my $ua = LWP::UserAgent->new;
$ua->agent('Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)');
my $response = $ua->get("http://www.mywebsite.com");