Voltage ratings have to do with spacings between live parts, like bus bars and terminals. There are basically only 2 levels of spacings, 600V and 300V. There is no such thing as 480-600V 1 phase service (that you would use this way) so basically all 1 phase service panels and load centers are built to the 300V standard in that aspect.
But when you get into 3 phase panels under 300V, there are differences in the construction between a 208Y120V panel and a 120/240V 3P4W panel. That's because you have to be restricted from plugging in 1 pole breakers to the red (high) leg on the 120/240V 3P4W systems. So it's not really about the voltage, its about the bus arrangements.