The reason I think that SQLite would be useful is that if we want to separate the spider from indexer, finding the articles to update in the index is as simple as
SELECT * FROM ARTICLES WHERE LAST_UPDATED > $LAST_TIME_I_RAN
instead of searching the filesystem. Stored on the filesystem, we will need code to
- store, and
the documents. SQLite provides all of that for free. Want to move to a different machine? -- The database is a single file. Plus, who knows what other useful things SQLite's flexibility will allow us to do?