Re^2: [OT] Re: Get unique fields from file

[erix]

That's nice, too, but it is essentially different from my example: your code actually /imports/ data, whereas mine only configures a table interface on top of an underlying text file, so that you can run any SQL against that underlying text table data (one could also read a zipped text file in this way).

Let's just say SQL too has more than one way to do it ;)

[Marshall]

Well I guess there is also DBD::CSV but once I got onto SQLite, I haven't looked back. SQLite is the most used DB in the world (geez, it is on every cell phone and in almost all browsers...), well debugged and very efficient for what it does.

Let's not quibble too much about the details.

The main point is that using SQL especially in conjunction with a real DB is definitely "on the table" as a reasonable solution for the OP's problem. So I don't think that is OT at all. This approach will scale well into much, much larger files. Functionality like "build histogram" and "print only the first time" are heavily optimized and a real DB is pretty smart about how it uses the memory available to it.

I don't think we have heard back from the OP about requirement questions, so this is really just an exercise for our own edification and enjoyment.

[erix]

so this is really just an exercise for our own edification and enjoyment.

So let me just enjoy some more light quibbling:

Hipp, SQLite-man extraordinaire:

"SQLite was a spin-off of postgres, a conceptual fork"

"We continue to look up at postgres as our mentor"

(which makes sense, why invent the wheel? I saw (just a few days ago) that the SQLIte devteam/Hipp are implementing some JSON operators, taking the example from postgres. Good!)

More fun: a youtube talk by Hipp (admittedly talking to a postgres-audience, at PGcon 2014):

PGcon 2014 talk by Hipp

[Marshall]

Well....I don't know about the postgres foundation for SQLite...check this interesting story out from Richard Hipp: Origin of SQLite

My SQL-Foo is weak. OMG this can get complicated!
As an example, I present jaro-winkler.sql - this is so far above my SQL pay grade, I get a nose bleed just thinking about it!:

Read more... (4 kB)

I have a tutorial on postgres that I am slowly working my way through. SQL is a complex subject. Admin of a DB is a complex subject. I have certain tasks that I want to learn how to do in SQL. But this is very much a "work in progress". I am now able to do simple SQL tasks and I have found that the combo of Perl + SQL is super powerful. SQLite is my "go to" DB because there is no admin or installation hassle.

I like this Hipp quote: "Fortunately, I didn’t know any experts — and so I did it anyway.”