Wysardry has asked for the wisdom of the Perl Monks concerning the following question: ⭐ (database programming)
I've seen it mentioned time and time again that flat text databases become very inefficient as the size of the file approachs 1Mb.
Does anybody know why that is and, more importantly, any way to lessen that effect?
It can't really all be down to the file system, as an RDBMS uses files on the same system in many cases. For instance, I have Perl and MySQL installed on the same drive.
Is it the indexing, or is it just down to the queries being controlled by a specialised compiled program instead of Perl? Is that really enough to allow MySQL to handle tables 5000 times the size available for a flat text version?
Has anyone experimented with indexing flat file systems in Perl and using seek to query smaller blocks at a time?
Originally posted as a Categorized Question.