Hi perlmonkey2

You are bringing up some very interesting points. However, I think you should include some links providing background information to better illustrate what you were talking about. In any case...

In the field of web based information retrieval, a common problem encountered is discerning the interesting text in a web page. A human decides this based on the formatting, location, and content of the text.

Yes, we tend to analyse the content to determine what concepts are described and how those concepts relate to our previous experiences. In that sense, marking a text as interesting will always depend on the person doing the evaluation. One research area that could help you deal with concepts is Granular Computing. Granular Computing will also allow you to deal with different levels of abstraction helping you control the level of accuracy required for each leaf in the DOM tree. One paper that could interest you is: Information Granulation for Web based Information Retrieval Support Systems. Finally, it is important that you consider different ways of grouping the data (or creating information granules in Granular Computing terms) to see which one is more suitable for your particular application. For that, I will recommend you to have a look at clustering methods like those in Algorithm::Cluster or in Re: module for cluster analysis.

perlmonkey2, good luck with this project. Please keep us posted on how it progresses

Cheers,