Close, but not quite. It depends on the ratio of "sensible" vs. "nonsensical" phrases, which in turn depends on how much leeway you're willing to give the program. If you want to mimic a rational human, then yes, you're right. But if you're willing to take something less stringent - say, your average irrational person, or worse, a suit - then you might be able to pull it off this way.
I did something like this with a music-composing program. I randomly generated tones, intervals, and durations, and filtered out the obvious dissonances and silly combinations. What was left was OK - not exactly music, but close to Musak. It would have passed muster in an elevator, but not at a concert.
