Re^4: POE Based web crawler or web spider ?

I am just exploring Gungho in order to build the search engine from the scratch .
I have the following questions.

1. I don't have any place to look at all the possible options used for Gungho . Can you please suggest me a powerful crawler config file (.yml) so that I can reuse it.
2. Is Gungho parse sitemap.xml as well as robots.txt operation ? If not, suggest me how can I do that ?
3. Is it crawel multi level ( kind of nested sites ) ?
Any suggestion on this ?
Comment on Re^4: POE Based web crawler or web spider ?

Replies are listed 'Best First'.
Re^5: POE Based web crawler or web spider ? by marto (Cardinal) on Dec 05, 2012 at 19:37 UTC
This reads like a "do it for me" type post. Please do some research since this is your task. Basic research will answer your questions.	[reply]
A reply falls below the community's threshold of quality. You may see it by logging in.