Let's parse this out a bit to get more clarity.
I have an existing 120/240V, 1 phase system being served from a utility company. The client is adding some 480V equipment and has requested the utility provide 480/277V service to the facility.
So first off, is the change to 480V 3 phase for only NEW equipment? Or is some of the new equipment REPLACING some of the existing single phase equipment?
The problem is the existing main circuit breaker for the facilty is 600A, 240V 1 phase. The max daily kWH usage for the facilty in a 13 month period is 872 kWH. The facility is on a 10 hours work day which equates to 87.2kW.
This is an assumption that you are making that is not actually indicative of the real load demand, as pointed out by others. Without measuring the real peak load demands, you have no idea what your service capacity NEEDS to be. One can safely assume only that the reason they had a 600A service is because they NEEDED a 600A service, at least at some point. This is why knowing if you are REPLACING existing loads or just ADDING is so important too.
The utility says they have a near unity PF at .98.
Interesting. Can we assume the existing loads are basically lighting and heating? Or is there Power Factor Correction taking place.
I would like to reuse the existing main circuit breaker and feed this from a 480-120/240V transformer.
No problem, as long as you are not expecting that secondary breaker to be the protection for an under rated transformer, see below.
A 100kVA transformer will not meet the existing load and a 150kVA is larger then the existing circuit breaker. Can the client use the 150kVA transformer and reuse the existing 120/240V 1 phase, 600A circuit breaker?
There are no standard 150kVA single phase transformers, 150kVA is a 3 phase transformer. Single phase transformers would be either 100kVA, capable of 416A, 125kVA, good for 520A, or 167.5kVA, good for 698A. I would opt for the 167.5kVA if the load really is that high.
You can technically use a 3 phase transformer to give you a 240/120 3 phase 4 wire output, and not use anything on the high leg, effectively unbalancing that transformer. But the capacity of the transformer with nothing but single phase loads will be severely curtailed. You will NOT get 600A single phase out of that transformer, you will be limited to 5% capacity in each 120V leg (I think, maybe 5% total? Never tried to do that), so at BEST, 15kVA available from that and if I'm wrong, 7-1/2 kVA. No matter what, total waste of time.
The entire facility, except the new equipment to be added, is being served from the 600A circuit breaker. This facility has (2) large buildings being served from the existing 600A circuit breaker.
Superfluous info.
I was going to add a 600A 480/277V main panelboard and feed the new transformer with a new 400A 2-pole circuit breaker. Would I need to lower the circuit breaker current rating in order to limit the current to the existing circuit breaker?
A 400A 2 pole breaker to feed what? Your 150kVA 3 phase transformer (that you actually can't use anyway)? Then no, you can't do that.
If you get a 125kVA, the 480V side would need a 300A breaker max., if you go with using the 600A main on the secondary side, because you would have to consider the secondary as "unprotected" since the transformer is not capable of putting out 600A, so that breaker would not trip before the transformer is overloaded. So 450-3(b)(2) says 125% of the transformer full load amps with no secondary protection and on 125kVA, the FLA on the 480V side will be 260A, so 125% is 325A. You might get away with rounding up to 350A though.
If you go with a 167.5kVA, the 400A primary would be OK, because the 600A Main you have now is within 125% of the transformer secondary FLA. Per 430-3 then the primary side can be up to 250% of the transformer FLA, which in that case is 349A, so the primary can be anywhere from 400 to 800A, maybe 900A. 400A would be OK, but might not hold in on inrush whenever the power is turned on, you might want to consider higher than that.
So...
Start by straightening out your facts on the transformer, something isn't adding up right so far, then determine (if you haven't already) whether the 480V loads are all new or replacing some of the existing, then measure the ACTUAL demand and go from there.