Re: Joint Database Technology

[marinersk]

I've done this. In fact, most DBMS systems grew out of this kind of good engineering and have taken it to the next level -- many times, in fact, to the point of being orders of magnitude ahead of these excellent fundamentals.

These days, free, multiplatformic, SQL-complaint RDBMSs offer lots of pre-built functionality that I don't have to maintain. But for a few really exceptional cases anymore, they've proven to be as fast -- sometimes faster -- than home-grown solutions when applied to large datasets, and eliminate most of the development related to the data engine.

This not only reduces the cost and speed of development, but also the risks inherent to any software development activity.

I'm not saying an RDBMS is always the right answer -- far from it. It is my considered opinion that far too many people jump toward a database solution far too quickly, when a faster, more efficient solution is right in front of them (often involving the use of Perl features like hashes and such).

But once you've decided to go with a DBMS, most -- not all, but most -- of the time, a pre-built RDBMS offers so many advantages that a home-grown, rudimentary DBMS such as you have described simply cannot compete with.

I'm fundamentally a data and algorithms guy -- always have been. I've designed and refined data storage, retrieval, and indexing routines for most of my career. And I was slow to adopt SQL in any form, but where they've gone with the technology is amazing even to me -- and it's hard to impress me in the data/algorithm space.

What you've done is impressive, efficient, elegant, and represents the very best in applying engineering principles to solving a fairly general-use case. I fervently hope and pray this kind of talent is not lost in our profession, and it is heart-warming to see it thrives yet.

But the world of general-purpose data storage has moved far past these fundamentals, and most of your stated advantages reflect a lack of understanding of where the state of the art has gone whilst you weren't looking.

I painfully admit that I once stood where you now stand -- rightfully proud of excellent engineering efforts and what it has produced, and all the while having no idea what I was about to find had been going on out in the rest of the world which would render my efforts nearly irrelevant.

I wish for you a gentler, and more importantly, speedier, recovery than I experienced. The sooner we accept the world for what it is, the more likely we can get back to doing things both great and small to continue to improve it.

[sundialsvc4]

Acknowledging all of your points, Mariners, here’s a good, every-day example of a distinction that I might draw:   “given the requirement to ‘store large images,’ do you store those images in your RDBMS, or do you instead store filenames?”

There’s no bright-line rule.   It depends on many things, including how well your particular database implementation handles “large BLOBs.”   Also to be considered is the fact that any file-system is also “a database of sorts,” which is extremely adept at storing variable-sized objects using a single key otherwise known as a “file name.”   Both approaches are defensible and we have all seen both of them done.

If the files which contain the data are enormous, and/or if they are already being used for other purposes by other existing applications, then it might well not be appropriate to move these data into an RDBMS, but, instead, to use the RDBMS as an index.   There is still a lot of high-volume data processing which requires the use of sorted files, with sorted updates being applied to those always-sorted remains-sorted files, specifically to avoid random-access.   But an RDBMS might still be useful for quickly and directly finding the location (exact, or approximate) of whatever data is being sought, thereby allowing the records to be queried randomly.   It is an admitted compromise, to be judiciously employed if and when the case arises.

[marinersk]

Agreed on all points.

Man, that's twice in one year. We should check and see if Armaggedon is happening or something.

[sundialsvc4]

Don’t worry, the stars must be out-of-alignment or something . . .

[Anonymous Monk]

In response to "Do you store images in your RDBMS, or do you instead store filenames?” In my tandem: Perl SDBM(random access indexing)/Flat File Databases(text data storage, fixed-length records) I store Bitmaps as text. Then, on the fly, I convert those text representations of images back into real images to display them to the end-user. This is a great security feature. This can be done on Windows O/S systems with Win32 Perl and the InLineBitMap conversion utility that comes with the Win32-GUI module.