Is is possible to write a script that when fed a
file containing properly formed Unicode text, it will count the
number of words and sentences it contains?
Languages of the world are way too complex. Unicode deals with text
at the character and grapheme level, which is hard enough. It is
silent on what constitutes a word or sentence. It is certainly
possible in many cases to define "words" and "sentences" in a way
appropriate to some particular expected text format in some known
language, but even then there are usually exceptions. Take
which satisfies a given spec. Is Sports.ru
one word or two? Is какое-то
two words, as the code
determines, or just one as Russian linguists would probably contend?
Do all other languages handle hyphenated text similarly? Almost certainly not,
as a general rule. The more text considered, the more edge cases
and ambiguities arise, even within a single language.
slowly but steadily working to handle Thai and Lao text in Perl.
For these languages, sentences are generally delimited by
whitespace, and individual words are not delimited at all in the
text, but instead are delimited by syntactic rules. Code can and
has been written to count individual Thai words, but it is
considerably different and more complicated than counting the
number of character strings between spaces.
Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
Read Where should I post X? if you're not absolutely sure you're posting in the right place.
Please read these before you post! —
Posts may use any of the Perl Monks Approved HTML tags:
You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
- a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
Link using PerlMonks shortcuts! What shortcuts can I use for linking?
See Writeup Formatting Tips and other pages linked from there for more info.
| & || & |
| < || < |
| > || > |
| [ || [ |
| ] || ] ||