Re: Re: Re: Choosing a data structure for AI applications

[PhiRatE]

I post here all the time but I never remember to log in and I keep killing all my cookies :)

The problem thus far with the discussion is that everyone is talking like using a database costs you something, as if its slower than a perl hash or some more complex in-ram data structure.

The truth is that, for a dataset that can fit reasonably in ram, perl is quicker. But once you talk about anything running into the millions of facts, a real DBMS will be unbelievably helpful. Indexing is a huge win for one, they do their own ram-based caching as well.

So, let us consider that, short of re-inventing a specialist database engine with many of the features that a DBMS already provides, a DBMS would be the most effective fact store for any reasonably sized factset.

Your next problem is how to effectively generate database schema in such a way that you get the most out of indexes, ordering and joins within the database without having to replicate data all over the place in different forms (an option, but an unhappy one).

The first thing is to understand that you want a powerful RDBMS. The object databases, PostgreSQL included, have many benefits, and its possible that one could be appropriate, but in my opinion the area of relational systems has been the most examined and they have the best combination of performance and flexibility (postgresql isn't shabby though, so as a free backend it may well be worth it).

From this you would then want to read up on reasonably advanced SQL, notably views, indexes, subselects and joins, and the performance parameters inherent in them. Many features of SQL will map directly to features of prolog with the relevant schema, for example a table of facts could represent, in two columns, the IS A relationship and retrieving the facts is a set of trivial SQL statements.

Unfortunately, I don't know enough prolog to take the analysis further, but it would seem to me that, if you're serious about doing big performance-intensive stuff, which is where expert-system AIs become really valuable, that the RDBMS backend is the only real option.