You are bringing up some very interesting points. However, I think you should include some links providing background information to better illustrate what you were talking about. In any case...
In the field of web based information retrieval, a common problem encountered is discerning the interesting text in a web page. A human decides this based on the formatting, location, and content of the text.
Yes, we tend to analyse the content to determine what concepts are described and how those concepts relate to our previous experiences. In that sense, marking a text as interesting will always depend on the person doing the evaluation. One research area that could help you deal with concepts is Granular Computing. Granular Computing will also allow you to deal with different levels of abstraction helping you control the level of accuracy required for each leaf in the DOM tree. One paper that could interest you is: Information Granulation for Web based Information Retrieval Support Systems. Finally, it is important that you consider different ways of grouping the data (or creating information granules in Granular Computing terms) to see which one is more suitable for your particular application. For that, I will recommend you to have a look at clustering methods like those in Algorithm::Cluster or in Re: module for cluster analysis.
perlmonkey2, good luck with this project. Please keep us posted on how it progresses
Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
Read Where should I post X? if you're not absolutely sure you're posting in the right place.
Please read these before you post! —
Posts may use any of the Perl Monks Approved HTML tags:
You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
- a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
Link using PerlMonks shortcuts! What shortcuts can I use for linking?
See Writeup Formatting Tips and other pages linked from there for more info.
| & || & |
| < || < |
| > || > |
| [ || [ |
| ] || ] ||