Can you use multiple physical tiers? Yes. You can use XML-RPC or SOAP to do it, although neither of those really offer object-oriented access (but neither does EJB, when used with popular approaches like "session facade"). You could probably get better performance by using straight HTTP and Storable for serialization, but then you'd have to write it yourself.

More importantly, should you? As you already realized, there is no need to have multiple physical tiers just to get the abstraction benefits. Adding physical tiers will add tons of overhead in the form of RPC calls. Don't let anyone tell you that this overhead is insignifcant. On a fast platform like mod_perl, this sort of RPC stuff is likely to be your biggest bottleneck. It could easilly cut your performance in half, or worse, and would probably force you to change your nice fine-grained OO interfaces into ugly corase-grained ones just to reduce the number of calls it takes to do something (J2EE people do this all the time with the aforementioned session facade stuff).

So why would anyone do this? The standard response is "scalability." The theory here is that it will be helpful to be able to add more machines to your application server tier without adding any to your web servers, because your application servers need the extra power and your web servers don't. If you have both of these in a single physical tier, you can't separate them like that. When you think about it though, what exactly is the problem with your web servers getting more power than they need? Having these in the same process makes you use fewer resources (because of the missing RPC overhead), not more. I don't think you could reasonably argue that it takes more machines to do the same work because they are together. It will probably take fewer machines. You also get to use standard approaches for load-balancing, like hardware HTTP load-balancers.

Some people will say "What about when you have really uneven business objects, and a few of them take 10 times as long to run as others? That will mess up your HTTP load-balancing." Well, most load-balancers can keep track of how many requests they have open to each web server and use that to send requests to the ones that have the lightest load, but it is possible that you could have a component that did some really heavy calculation and sucked up lots of CPU, making it hard to load-balance because it is so unlike the other requests. You can deal with this either by using asynchronous processing (a queue), or by dedicating some specific servers to handle these requests. This avoids penalizing every single request with RPC overhead. It's easy to route requests based on URL when you are using a front-end proxy or a good load-balancer.

The real joke here is that the people who shout about this loudest are typically the EJB vendors, and they are selling snake oil. EJB performance is so bad when remote calls are involved that the EJB containers were all written to make the calls local whenever they can. That's right: the remote calls are actually turned into locals calls into the same Java process whenever possible. In EJB 2, an explicit local interface was added so that developers can force this behavior on the few EJB containers that weren't already doing it!

There is one thing that is good to separate, and that's your static web content. No need to tie up a big mod_perl process serving your JPEGs. There are several ways you can handle this, discussed here.

You might also be interested in the article I wrote about how we used multiple logical tiers and in some cases physical tiers (for separate search servers) at eToys.com. That article is here.