When performing an instantaneous trip test on a 480 V circuit breaker. What is the max amperage difference I would want to see between each phase of the breaker?
You're not comparing phases to each other, you're comparing each phase to the published trip curve for that breaker plus the error tolerance for that curve.
I don't test many MCCBs so I wouldn't be the person to ask there, but for power circuit breakers the tolerances I often see are 10%, though it does vary.