Casual Communication, AI, and Text Processing

[one4k4]

Tis been a while since I've posted, especially in SoPW. I thought up an idea while thinking about the smarterchild AIM bot. Currently this bot has somewhat "canned" responses (imho) to your statements. I thought about that bot, and the everything project and wondered if I could create a bot that would chat with it's users and ask for help in expanding it's knowledge base. Somewhat of a dictionary, with definitions submitted by the chatters.

This good and all, but what about responces to it's questions, and processing the actual words in the statements? This node (AI -- Artificial Intelligence) helped me quite a bit, but I'm not sure where to start.

It's almost as if I want to have a set of "control" words, and take words in question and compare them to the control words. A table of synonyms, antonyms (spelling mistakes..) and rate the possible control words against the results.

So if you said "My dog is a real pain" it would take "dog" and compare that to things like "animal" and then it would know that the subject of the sentence is animals, in reality "my dog". Basically, applying english language rules to text processing with perl. Yeah, that about summs it up.

Knowing "dog" is the noun and subject of the sentence, if it does not know what dog is, it would ask..

Am I crazy? Am I thinking about something that is too far fetched? Am I entering a good part of the perl community? I know this would at bare minimum help with my OOP and the like..

Welp, that about summs it up friends. Feel free to shoot me in the foot if I haven't already, but any help or insight is appreciated.

I'm only thinking about this because jmcnamara told me I couldn't help him with Spreadsheet::WriteExcel until he gets back on track. ;)

_14k4 - [e-mail] (www.poorheart.com)

[dragonchild]

Basically, you're trying to quick-parse sentences. My cousin did her master's thesis on comparing two algorithms for natural language parsing. Just two, and she got a Masters.

The problem is that the rules for parsing a natural language are context-driven. In your example, "my dog" is the subject because it precedes "is", not because it's a noun. (And, just to make things more complicated, "dog" is considered a noun solely due to placement. There is a verb "dog", too.)

I'll give you another example. "Pole, you Pole!" That is a perfectly good imperative sentence in english. It breaks down as "Verb (command), Object". However, if I change it to "Pole, you pole!" ... That breaks down to "Object, Verb (command)". The only difference was capitalization. But, I suspect all the english-speakers could understand what I was saying. How would you encapsulate those rules?

I'm not trying to say it can't be done. In fact, I'm working on an algorithm myself, in my spare time. However, I certainly do not think that Perl is the place to do it. The data structures are too large and the algorithms too computationally-intensive for Perl to be a good language.

------
We are the carpenters and bricklayers of the Information Age.

Don't go borrowing trouble. For programmers, this means Worry only about what you need to implement.

[one4k4]

Hrm, I like your commentary. Very interesting indeed. I was thinking of maybe something similar to the concept behind soundex in the SQL langauge. Even doing the "soundex" like calculations on a table level? But my interest is indeed sparked, and I shall go from here.

I did, however, create a very basic sourceforge project for this idea located at sourceforge.net/projects/ieokb. Expect things to change. And change often. Sourceforge is basically there for CVS and portability.

Thank you so much for your thoughts, they're a *big* help.

_14k4 - [e-mail] (www.poorheart.com)

[lachoy]

Not only is it possible, but folks are working on it. Ken Williams gave a presentation at YAPC 2001 on his AI::Categoroize module, and IIRC he's using it right now to route questions sent to the Math Forum. You might also look into the work Lenzo's done with IRC bots.

Chris
M-x auto-bs-mode

[trs80]

Most anyone attempting to create AI has to be crazy. The complexity surrounding language and its use is a mind numbing thing to consider. There was a very good article in Harper's April 2001 called "Tense Present". I think what you are trying to accomplish really needs to be throughly broken down into the parts that make up languages, the more the better, and discern the commonality between them.

There are good articles on Perl for language parsing in TPJ. Fall 2000 and Summer 2000 jump to my mind immediately.

Take a look at the Lingua related modules to see what others have done.

[Hanamaki]

Take a look at the Lingua related modules to see what others have done.

Probably Lingua::LinkParser could be a good starting point. There is also a TPJ article by Dan Brian about that Module.

Hanamaki

[one4k4]

I just wanted to say, that your link and quick comment deserves more ++'s than I can give. That module will help me a lot. If anything to keep my mind busy for the day. :) Thank you!

_14k4 - [e-mail] (www.poorheart.com)

[RMGir]

A quick google search turned up BotSpot, which looks like a good site for investigating this kind of question.
--
Mike