I think the answer is; we just don't know, but it COULD have lead to a disaster and it was not worth taking the chance.
Most panelboard manufacturers do NOT make two different classes of bus systems, so they will make only the higher class, which is perfectly fine to use on a lower voltage. The real difference is in the breakers, as has been discussed already. There ARE significant differences in the breaker construction and for this reason, sometimes the manufacturer will install physical barriers in a panelboard to prevent the installation of lower voltage rated breakers into one LABELED for a higher voltage, i.e. the interior will not fit, clips that will not stab in etc..
The reason is, the destructive part of a fault as far as the circuit breaker goes is fault ENERGY, not just fault current. At a higher voltage then, the energy in a fault of equal current is drastically higher. Let's say for instance that you have a breaker with a "slash" rating, i.e. 480/277V. This means that you CANNOT use that breaker on a 480V 3 wire service, it has not passed the necessary tests at full line voltage because if there is a grounded fault, the voltage could be 480V, not 277V. In other words, by displaying the "slash" rating, it means that to pass the UL489 tests it relied upon the fact that there would never be a ground fault in excess of 277V, which limits the amount of fault energy in the circuit even if the current is the same.
Now apply this principal to the above situation. We don't know, but it is possible that the breakers were only rated for 240V max. Since the panel was originally labeled for 208V 4 wire, it would not have been a problem to use breakers like that, there would have been no physical barrier to prevent installing them. So when the dufus changed the service to 480V, he was overcoming any safeguards that would have been in place. It is possible that there was no problem because the breakers were rated for 480V anyway, again we just don't know.
You were right to change it out.