Is is possible to write a script that when fed a
file containing properly formed Unicode text, it will count the
number of words and sentences it contains? No!
Languages of the world are way too complex. Unicode deals with text
at the character and grapheme level, which is hard enough. It is
silent on what constitutes a word or sentence. It is certainly
possible in many cases to define "words" and "sentences" in a way
appropriate to some particular expected text format in some known
language, but even then there are usually exceptions. Take
choroba's code which satisfies a given spec. Is Sports.ru
one word or two? Is какое-то two words, as the code
determines, or just one as Russian linguists would probably contend?
Do all other languages handle hyphenated text similarly? Almost certainly not,
as a general rule. The more text considered, the more edge cases
and ambiguities arise, even within a single language. I am
slowly but steadily working to handle Thai and Lao text in Perl.
For these languages, sentences are generally delimited by
whitespace, and individual words are not delimited at all in the
text, but instead are delimited by syntactic rules. Code can and
has been written to count individual Thai words, but it is
considerably different and more complicated than counting the
number of character strings between spaces.
-
Are you posting in the right place? Check out Where do I post X? to know for sure.
-
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big>
<blockquote> <br /> <dd>
<dl> <dt> <em> <font>
<h1> <h2> <h3> <h4>
<h5> <h6> <hr /> <i>
<li> <nbsp> <ol> <p>
<small> <strike> <strong>
<sub> <sup> <table>
<td> <th> <tr> <tt>
<u> <ul>
-
Snippets of code should be wrapped in
<code> tags not
<pre> tags. In fact, <pre>
tags should generally be avoided. If they must
be used, extreme care should be
taken to ensure that their contents do not
have long lines (<70 chars), in order to prevent
horizontal scrolling (and possible janitor
intervention).
-
Want more info? How to link
or How to display code and escape characters
are good places to start.
|
