Perl version of 'diff'?

[ninja_byte]

I have a database filled with about 200,000+ records. Each one has information - domain name, username, host.
This is *not* a dynamic database - I have to manually go through each one of the hosts, extract a master domain list, and parse it from there...
The initial challenge was to get all of the information into a database - that has been handled(albeit crudely).

Now I'm faced with the task of keeping it updated every few days/weeks...
So lets say I have 'host01-original.txt', and I can get 'host01-updated.txt' at any given time. Any suggestions as to the best way of finding new additions to the file only?

The brute force method of a huge 'grep' loop seems a bit distasteful at present.

The file is currently in the format:
domainname.com:username

Thanks!

[tachyon]

If you move to a real DB (or even with a slight modification to the current format) there is a useful trick - add an update time field. Unix epoch time is fine. Thus your master database looks like:

data_field1 data_field2 ... update_time

With the update time to hand it is a simple matter to select all the records with an update time > some_value and thus generate a record set than encapsulates all the latest changes, valid from any fixed time point.

It looks to me as though your data storage format is going to give you issues but this depends on what you are actually doing. There are essentially 3 types of data changes you may need to deal with:

1. INSERTS - adding a brand new entry
2. DELETIONS - removing dross
3. UPDATES - ie what if [e-mail] changes their username? You either need to UPDATE their record or delete the old record and insert the new one.

[PERLscienceman]

Greetings Fellow Monk!
You may want to have a look at the following module on CPAN: Text::Diff . It's description states that it 'performs diffs and files and record sets......providing a basic set of services akin to GNU diff'.
I believe that this may be the very nearest thing to a "Perl Version of 'diff'".

[Joost]

Another suggestion: Algorithm::Diff.

[borisz]

I would put the data into a real database. Like postgres, perhaps SQLite is enough.

[runrig]

If both files are sorted, and you just want the new lines, then just:

comm -13 host01-original.txt host01-updated.txt

[cLive ;-)]

How about I add a timestamp field to the relevant table and show you the SOAP interface to the DB :)

cLive ;-)

Err, for those of you wondering, Ninja_Byte is sitting about 10yds away from me :)

[ninja_byte]

That'd be cool, but it's supposed to be a master list spanning all types of servers(both *nixes, windows, jails). I have to use a variety of methods to get the list, then grep and awk my way into a standardized text file format.

From there it's another deal to get it into sql format.

from there, the updates.. It's supposed to be a bit overlapping, with the same person showing up on multiple hosts. I'm going to play with the aforementioned modules and see if I can avoid reinventing the wheel..

"efficiency via the work of others..."