So why would anyone do this? The standard response is "scalability." The theory here is that it will be helpful to be able to add more machines to your application server tier without adding any to your web servers, because your application servers need the extra power and your web servers don't. If you have both of these in a single physical tier, you can't separate them like that. When you think about it though, what exactly is the problem with your web servers getting more power than they need? Having these in the same process makes you use fewer resources (because of the missing RPC overhead), not more. I don't think you could reasonably argue that it takes more machines to do the same work because they are together. It will probably take fewer machines. You also get to use standard approaches for load-balancing, like hardware HTTP load-balancers.
I'll try and refute this. Imagine a site that keeps it simple and they need to keep the fast parts to stay fast. Well, things that take up more speed need more resources. Yes, throwing more hardware, complete webservers, does solve something, but in the long run, the entire site will slow down again as it gets more popular.
So fine, now you create a seperate farm for one part of your site. Tons of links to change over etc etc..
It'd a big pain in the ass. So why not keep the web interface running uber fast, the actual display layer, but move the bottle necks to a seperate area. This of course, as we both know, is how n-tiered would start to evolve. The advantage is, that if I hit one interface that is dog slow, it may not mean the others will slow down. That might mean tweaking the aplication layer or db layer.
As for your RPC argument, you are right, but then again, the same problems happen in PHP or Perl. That's why you have connection pools and the likes. You keep everything on high speed connections, and things will run well. You and i both know that sending 1k of data over RJ45 is very quick to move that type of stuff over. And we all know you don't do repeditive rpc calls to get all your data.. .just one fell-swoop with one biz objects.
Also, doing things via proxy server would be fast, but as the scenario gets more complex with a url proxy, things slow down in a way you can't speed back up. Load balancing a bunch of proxy servers is just ugly. 'cause then you have something that is just as bad as a regualr web farm. As things slow down, the entire service slows down and not just the parts that are costly.