in reply to Re^3: POE Based web crawler or web spider ?
in thread POE Based web crawler or web spider ?
I am just exploring Gungho in order to build the search engine from the scratch .
I have the following questions.
- 1. I don't have any place to look at all the possible options used for Gungho . Can you please suggest me a powerful crawler config file (.yml) so that I can reuse it.
- 2. Is Gungho parse sitemap.xml as well as robots.txt operation ? If not, suggest me how can I do that ?
- 3. Is it crawel multi level ( kind of nested sites ) ?
Any suggestion on this ?