I assume you are talking about a feeder panel here. If it is/were a service panel, the minimum size is per 250.66, and a neutral is required to be pulled to the service disconnecting means....
If you had a 3 phase load with no neutral as you mention above then you don't need to pull a neutral at all. If you had a panel with rtu and other loads then you would use T. 250.122 for the lowest possible conductor for the neutral. It cannot be smaller then the egc.