UPDATE:
finally got into the issue today, here is what we found...
Generator service company was onsite and scoped the system, synchronization is correct.
Megged the transformer - no problems there.
megged the feeds - no problems there
found CB feeding transformer to be 'burnt' on A phase - 100A Siemens NGK
customer had new CB on site, we replaced it after testing...
Additionally:
1) no ground from primary panel to transformer, they used the EMT
2) no bond between XO and ground
3) ground from secondary panel was a #6 connected to a bonding bushing in the secondary panel.
4) building steel was brought to the ground bar in the secondary panel - not connected to transformer.
5) originally, there was a bond in the secondary panel from the ground to the neutral bar, this was removed by a previous electrician
6) loose neutral lug in secondary panel
7) loose lug on phase A in transfer switch on generator feed and on feed to generator panel (primary panel)
8) utility voltage of 499V L-L and 288V L-N
9) generator voltage of 480V L-L and 278V L-N
after verifying the transformer was ok, tightening the loose connections and grounding the transformer, we put it too a test....
load was approx 13A primary and 40-50A secondary (fluctuation I believe was due to 2 small UPS systems fed from the secondary panel)
when utility power was sent to transformer, inrush current was approximately 25A
when switching from utility to generator, under load (we did not shut down building) inrush current varied from 30-45A
when switching from generator back to utility, under load, inrush current varied from 75A to 150A (not sure why the variation)
the transfer occurred without a problem, no more tripping 100A CB...so the IT guy from the company decided to test it again...
again, no problem...
then a 3rd time....and the breaker tripped going from generator to utility...inrush was approx 100A when it tripped.
he tried a 4th time, same problem...
now, the 4 attempts were all within about 15 mins...could the time have an impact on the test? i.e. thermal buildup?
we switched everything back to utility, let the system run for 30 mins...switched to generator - no problem...back to utility - no problem...
so, where it stands is that we need to investigate more...
any thoughts?