in reply to Re: Re: Re: Re: Re: Re: writing to the top of a file
in thread writing to the top of a file

Yep, this is a susupect test. The files will still be hot in memory, confounding your attempts to measure them.

Plus, any professional strength backup solution will cope with small files just fine.

That's why people have written things like database systems to optimize the storage and retrieval of data from disks.

People wrote database systems so they could efficiently store and search relational data. Programmers abused databases because they were faster than the shitty filesystem drivers most vendors shipped. This is no longer the case, but I am continuously deluged by sweaty little morlocks who tell me that the solution to all data storage problems is a relational database.

In general, they just use it as a hash table, which is something that filesystems are much better at today.

Your objections are based on using consumer grade hardware. Run your own example, but look at the disk I/O meter. You will see that your benchmark is I/O bound - something that can only be improved by better hardware.

If you are just running a site for a few friends, my solution will be great. If you are running a big site with lots of hits, you'll have I/O channels that work at bus speeds, and my solution will still be great.

All your post says is that you have shitty hardware, and you're generalising that to 'the world has shitty hardware'. And the votes are going the way thay are because there are more readers who are deluded that their P4-3Ghz is the best computer in the world, and have no idea that a $100k 1Ghz Sun server will beat the pants off it in every test that counts... like handling small files.

____________________
Jeremy
I didn't believe in evil until I dated it.

  • Comment on Re: Re: Re: Re: Re: Re: Re: writing to the top of a file

Replies are listed 'Best First'.
Re: Re: Re: Re: Re: Re: Re: Re: writing to the top of a file
by Dr. Mu (Hermit) on May 22, 2004 at 04:15 UTC
    Jeremy,

    I feel compelled to step in here with a mild, but no less heartfelt, rebuke.

    One of the things that makes Perl Monks unique among internet discussion groups is our collegiality. Even when we disagree, we accord each other a certain degree of respect. In this sanctuary of civility, questions and their solutions can be debated in a well-mannered fashion, without snide remarks or personal attacks. You said,

    "And the votes are going the way thay are because there are more readers who are deluded that their P4-3Ghz is the best computer in the world, and have no idea that a $100k 1Ghz Sun server will beat the pants off it in every test that counts... like handling small files."
    I'd like to propose an alternate explanation. Many of your technical points may well have some validity. But I suspect that the majority of any negative votes you've accumulated in this thread are due to the arrogant, dismissive attitude that comes through in your posts. None of us, neither saint nor accolyte, is God's gift to programming. We're all here to learn from each other. And statements like, "You're telling me how to suck eggs.", or "So what are we supposed to use it for? Tracking icecream?", are not constructive contributions to the discussion. It's obvious you have a unique perspective and a lot of great ideas to contribute. But more people will pay attention if you demonstrate a little more regard for your intended audience.

    Thanks for listening,
    Phil

      If you really do believe in well mannered discussion I suggest you zap over to Compnay hackes through my Perl's Website Securtity hole and back me up against self righteous twats who are busy applying the boot to a very foolish person.

      Whenever this sort of(filesystem) discussion comes up, the efficiency experts always charge in to slap down anyone who doesn't do it the right way. I understood your point, I even knew it before the conversation started. But if you are going to make these arguments, you owe it to yourself to be up on the state of the art.

      Now, as a complete coincidence, I had just spent the day going through Hans Reiser's filesystem site, so I was brushed up on the state of the art. I posted a solution that would work OK most of the time, and well if the user were up to date on hir filesystems. It solved all the posters requests, including concurrency ( few others even addressed that).

      But what do I get? Requests for links? Asked for more details? Nope. Just "my 80's filesystem screws your idea, so obviously you must be misinformed, here, let me explain what a filesystem is to you".

      So when we talk about comments contributing to a discussion, I was annoyed at yours because it misleads naive readers into believing that they shouldn't do something due to efficiency problems that no longer exist. And that results in me having to nstall and maintain over-complicated and fragile code that was written to work around problems that don't exist.

      And that is why I got cranky.

      ____________________
      Jeremy
      I didn't believe in evil until I dated it.

Re^nth: writing to the top of a file
by graff (Chancellor) on May 22, 2004 at 04:47 UTC
    OK fine, if you have a few extra $100K's to throw around and cover up a sub-optimal process design, go ahead and spend your money that way. Keep those wheels of commerce rolling!

    I work at a university (i.e. with a non-commercial academic budget -- in linguistics, mind you), so I've had to put up with lower-grade Sun Enterprise hardware (including the multi-terabyte tape robot for backup) for the last several years. And by God golly, when one of my programmers installed a directory tree with a few million little files on one of the raids, everybody felt the pain, including the poor sysadmin trying to work around the level-0 backups that either took a week to finish or simply failed. I had to tell my programmer not to do things that way, because I can't just go out and buy bigger iron.

    Mercifully, we are finally migrating to FreeBSD (and running it on a variety of hardware from the more "lean and hungry" vendors). Things are looking up.

    Got any more flame bait? (as if I had to ask)

    (updated to chill on the hyperbole. Thnx, jepri.)

      ...cover up a sub-optimal process design...

      You mean like the fact we're using Perl and not hand-tuned assembler?

      Got any more flame bait?

      Sure, here's some. Learn some statistics skills. You've managed to generalise one incident, that involved one programmer, once, to be a general rule that applies to all situations and all people, everywhere, all the time.

      You constantly refer to vague, unspecified things (like, "the backups"), and expect that to be a convincing argument. You don't even bother to take into account things like journalling of disks, normal load, the program's access patterns, the system tuning you've done (or haven't done). Anyone who was taking system tuning seriously would at least mention those factors. and certainly nobody with experience in admin would generalise from a single case like that.

      And saying 'by God' in a post tends to mean you've left the realm of facts and have taken off into hyperbole. "... a few million little files...". That's the closest thing to a statistic I've heard from you in the entire debate, and it is laughably fuzzy.

      ____________________
      Jeremy
      I didn't believe in evil until I dated it.