The maximum you can load this transformer @ 208 volts is 96 amps.
S0, secondary breaker would be of 100 amps, you have mentioned that your present load is only 50 amps ( Is that motor load?) if yes, what is the size of the biggest motor?
The primar circuit breaker (@480volts) can be of 40 amps ( with inrush current delay feature).
The primary inrush current of any transformer can go upto 12 times the primary full load current for few milliseconds and hence they have to have the inrush delay in them.