Thanks for the condescension, but it's actually not impossible to get away from NFS. Most of my work is done with application-level clustering.
I've done NFS-hosted home directories (and in fact /etc and /usr too; also used other networked file systems like AFS for that) before. It does grant some flexibility but it is far from the only way to do things. Maybe you should look at Ceph, GlusterFS, or GFS2 if you really want shared files.
If you trust yourself to write a fully-fledged database more than the PostgreSQL folks then go right ahead. I'd personally rather let the DB folks take care of anything more complex than a simple key/value store or object store. I like to focus my efforts elsewhere in the stack where we have problems that aren't already solved. PostgreSQL can be replicated very reliably and application scaling can be accomplished a number of ways other than putting everything on one big central file store.
We actually have a home-grown distributed object store with eventual consistency and failover to the original storage server for things not yet fully replicated. It's not what we use for a database. It's used as an object store. The database (often poor little MySQL, but sometimes PostgreSQL) is used as the database, complete with replication and read/write spreading with failover.
