Instantaneous trip levels can be a trick to coordinate for a transformer inrush, an engineer should be consulted with to determine the ITL's of all these breakers that have had their settings modified, the last is the rating of the 100 amp breaker for this transformer, I think it may have a low ITL and needs to be changed to one with a higher ITL, or one thats not instantaneous trip, and or a higher rating, question is did this set up ever work ok at one time? if your secondary has OCPD for it, you can go as high as 250% this would have to be determined along with the other breaker settings.
One question, has anyone ever tried to manually transfer or just turn off the main then turn it back on to see if the same happens without the generator running? that would eliminate the zero sequence problem some have brought up?
As was said table 450.3(B) is the max allowed but not required, (my mistake) and if you do have over current protection see also note 3 to the table above.
If this has been a problem since this transformer was first installed, then maybe no coordinated study was ever done, in which one should be done by an electrical engineer who understand these types of studies. AS Zog has pointed out a very big concern, their maintenance personnel should have not just started turning up the ITL's to get the breakers to stop tripping as they could be putting conductors and equipment in jeopardy if not sized for this new setting.
I'm not by any means any kind of expert on these types of problems, and would love to learn more about it one day, but I have run into very similar types of problems with trying to fire up a cold transformer, with the same results, but in our cases we consulted with the manufactures of the transformers engineers who knew the particular transformer, which might be another road to take also.
In one case we had a 750 kva 480 D X 208/120 Y that was tripping a 900 amp on the 480 every time we had a power failure and power was restored, the transformer manufacture sent out an engineer who had us change the breaker ITL setting to .9 from the .4 it was set at and trip time to 1 second from the .6 it was set for, on a GE series 2k MDP but where those numbers came from I have no idea, well I have a dangerous inkling
