It's not dumb to ask, and it's easy to see why it's not easy to see why.
The basic reason is that we cannot treat conductors, as well as their terminations, as though they have no resistance. When carrying current, the voltage at one end of a conductor will not match that at the other end.
In other words, with a proper, four-wire feeder, there will be a voltage difference between the grounded (neutral) and grounding (EGC) conductors (unless there's a shared neutral and the lines' currents just happen to match).
With a three-wire feeder, that could cause energized what-should-be grounded surfaces. The same applies to the no-longer-permitted practice of allowing the neutral of a major-appliance circuit to do the grounding duties.
In a nutshell, if there are no other conductive pathways between buildings, and you're not yet under the 2008 (or later) NEC, you can still use a three-wire feeder, but there are few here who would recommend it; I wouldn't.