277/480 WYE 4-Wire Woes

[bluescarecrow]

I was called out to mop up a considerable mess. Client has 900 Crypto Currency Mining Machines which run at 200-240 @2200 Watts. The electrical contractor had Navopache Electric Co-Op Inc. install 277/480 WYE 4 Wire along with 4 giant transformers for a total of 4 megawatts. When the machines power supplies fail, they take the machines out of service. At almost 10 amps each, several machines constitute a considerable phase imbalance. They were cooking the Eaton 400A breakers (visibly smoking) burning the insulation off of the 400A Feeder. I have since solved that problem, by instructing the technicians, that if they turn off one machine on the brown phase, they will have to turn off a working machine on the orange and yellow phase to keep the phases in balance. I even color-coded the 4x4 electrical boxes and identified each box with the circuit number in bold vinyl lettering. So far so good. Further I have instructed them to purchase a 3 phase monitoring system that will give them email and text alerts should a phase imbalance threshold be exceeded.

Now for the 277 volt issue. There is a possibly that the transformers can be re-tapped to lower the voltage to 240/440 WYE 4-Wire, but I've never heard of that. To move to a 208 WYE or a 240 Delta scheme, I'd have to change out all of the neutral wiring... big job. If they re-tap the main transformers, the sub-transformers that supply the 120vac to the network switches would have to be re-tapped also.

The original electricians were fired. They were pulling the 400A (1,000 kcmil?) feeders with their car. They scraped some of the insulation off of one of the feeders. The fireworks started when they turned the power on. Concerned residents immediately called the police to report a sonic boom (kidding). Why they used a wire so big is beyond me. They cut off a bunch of strands just to get it into the breaker.

I did my apprenticeship with the Illinois Central Railroad and became a Journeyman Electrician when I was 19. My Dad was also IBEW 134. My graduation certificate was a Reverser Lever for the Locomotives. LOL I've worked with most everything, low voltage, single phase, WYE, Delta and even 1,200 vdc on the electric trains, but this is a first. I've never seen such a huge screw-up.

Maybe I should just walk away from all this, but I do like a challenge.

Any help would be greatly appreciated.

Thanks!


Best,
Mike

 

[tortuga]

What about installing some real time ammeters on the panels so the miner techs can see in real time there phase loads?

 

[augie47]

Why does disconnecting "x" machines cause the problem with the 400 amp breakers ??

 

[hillbilly1]

Kinda confused why insulation is burning off the 400 amp feeders if they are 1000 kcmil? The breaker would trip way before any insulation damage would happen for that size of wire.

 

[bluescarecrow]

What about installing some real time ammeters on the panels so the miner techs can see in real time there phase loads?

Miner techs don't visit the machines every day. It's so cold in there. They're too busy repairing the smoked power supplies fed by the 277vac . Monitor will send text messages to them. I think I've got them trained now. Thanks for writing!!!

 

[bluescarecrow]

Kinda confused why insulation is burning off the 400 amp feeders if they are 1000 kcmil? The breaker would trip way before any insulation damage would happen for that size of wire.

I guess I kinda misled. Several of the Eaton breakers are fed with twin 2/0 feeders. It's a phase imbalance not an amperage overload... or it might have been a loose connection. That problem was addressed before I got there, but I sure heard about it. I have found breakers where the Allen set screws weren't even finger tight. When I first started, I found a brown phase at 140 amps, with the orange and the yellow at 170. They shouldn't be more than a few amps off. Thanks for writing

 

[bluescarecrow]

Why does disconnecting "x" machines cause the problem with the 400 amp breakers ??

When I first started, I found a brown phase at 140 amps, with the orange and the yellow at 170. They shouldn't be more than a few amps off. When I worked as an electrician on the movie "Silent Witness", I saw the guy operating the generator sitting there reading the newspaper. He had a big light hooked to the generator. I thought to myself... that's an awfully big light for reading. Later I learned the generator was 208 WYE 4-Wire and he had to load up the 3rd phase so as to not ruin the generator. Amazing machines... you can stand right next to them and they hardly make any noise.

Thanks for writing!

 

[winnie]

If everything is properly sized, unbalanced loading should not cause excessive heating.

Unbalanced loading on a generator might cause torque pulsation and problems but this shouldn't be a problem for utility supplied power.

Jon

 

[bluescarecrow]

Here are some pic from Bay One.

​

 

[bluescarecrow]

Here are the 4 Megawatt transformers out back.

 

[bluescarecrow]

Unbalanced loading on a generator might cause torque pulsation and problems but this shouldn't be a problem for utility supplied power.

Truthfully, I have never dealt with unbalanced phases before, but from what I read, the phases shouldn't be more than 3% out of balance. Maybe it has to do time and the unbalanced phase is trying to stay synced and can't. From my experience so far at this location, I've replaced 3 400A Eaton Breakers. one visibly smoking. We measured the heat at 145 degrees on one breaker and after I changed it and balanced the load, there were no further problems. So I have to assume I'm on the right track.

Thanks for writing.

 

[Hv&Lv]

You got 200- 240v machines running on 277/480??

 

[winnie]

Truthfully, I have never dealt with unbalanced phases before, but from what I read, the phases shouldn't be more than 3% out of balance.

That is the sort of requirement you will see for voltage to a motor, and it is real. But it doesn't apply to the current going to L-N loads.

I've replaced 3 400A Eaton Breakers. one visibly smoking. We measured the heat at 145 degrees on one breaker and after I changed it and balanced the load, there were no further problems. So I have to assume I'm on the right .

Thanks for writing.

I suspect that the problem was the loose connection issue you found on other breakers. You fixed the real problem (the loose connection) and at the same time did something unnecessary (implemented load balancing procedures.)

In any case, the real problem is 240V power supplies at 277V, and you are on the right track to fix that.

Jon

 

[tortuga]

In any case, the real problem is 240V power supplies at 277V, and you are on the right track to fix that.
Jon

Right

There is a possibly that the transformers can be re-tapped to lower the voltage to 240/440 WYE 4-Wire

Are the transformers utility owned?
440 Wye would still be too high @ 253 not 240.
The most 'standard' wye would be 400Y230.
415Y240 would be close enough, but I doubt the transformers could tap that low?
If you can pull that off all your gear can stay the same.

EDIT if there are three 5% taps down that would get you there.

 

[bluescarecrow]

W... Thanks for writing. All you guys are great!!!

I believe the transformers are utility owned.

Monday I'll be in a conference call with Navopache Electric Co-Op. The 277 thing is a real problem. From what I've heard they are going to be hard to deal with. They're telling my client that any change here will affect users hooked to the Navopache feed. Don't know how that works, unless they are talking about changing the feed voltage. I think Navopache purchased these transformers for a client who went belly-up and then they sat in Navopache's yard collecting dust until my guy came along and then Navopache told him these would be a perfect fit. My last name is O'Connor and I was instrumental in getting Jim O'Connor way more signatures needed to get on the ballot and elected to the Corporation Commission. Hope that will carry some weight. Never dealt with Electric Companies before. This ought to be fun. The Corporation Commission is a great ally.

 

[bluescarecrow]

You got 200- 240v machines running on 277/480??

I don't but the client does.

He thinks he's saving money by running at a higher voltage. Lets see...

2200W divided by 240 = 9.1 amps (standard operating voltage).
2700W divided by 277 = 9.7 amps.

Collateral damage includes:

Cost of repairing computer power supplies.
Cost to replace 400A 3-Phase Eaton Breakers.
Cost of additional 3-Phase fans to cool the place.
Cost of electricity to run the 3-Phase fans.

He's got these 400A panels maxed out. Each 20A breaker serves two machines; that's 19.4A x 20 breakers.

I read somewhere where these Switch-Mode Power Supplies will generate harmonic currents from 60 cycle ac, ie: 120 (1st harmonic),
180 (2nd harmonic). Haven't even gotten into that yet.

 

[paulengr]

I guess I kinda misled. Several of the Eaton breakers are fed with twin 2/0 feeders. It's a phase imbalance not an amperage overload... or it might have been a loose connection. That problem was addressed before I got there, but I sure heard about it. I have found breakers where the Allen set screws weren't even finger tight. When I first started, I found a brown phase at 140 amps, with the orange and the yellow at 170. They shouldn't be more than a few amps off. Thanks for writing

When a mechanical lug overheats the metal stretches permanently and loses its temper. The lugs will feel “loose” but the discoloration is your clue to replace them. Maybe they were loose but junk now. Crimp lugs have the same problem on their bolts but since the steel fastener isn’t really much of the circuit it tends to hold up to this abuse better.

When you have loose connections or cut strands you get something called constriction resistance. Resistance at the joint increases with current. The bulk wire is OK but you have much higher resistance at the joint so I-squared R applies and temperatures at the joints are much higher. Typical resistance across a well made joint (DC, not AC or constriction) is under 1 milliohm (0.001 ohms). I’ve seen 2-3 ohms on bad connections. I have a Ductor to measure it. Heating would be thousands of times higher. But voltage drop might only be say 5-10 V and it can still easily be under ampacity so the breaker never trips. Other than having the lugs destroyed and the heat can even damage internal parts like melt the contacts together, the breaker is unharmed and won’t trip.

Current imbalance is harmless except for two issues. First off three phase balanced loads like motors can’t tolerate unbalanced voltage and if VD causes more than 2-3% voltage unbalance you can have a problem. Where I usually see this is either loose wiring, worn out contacts, or utilities with wye-wye transformers with say most of their trailer parks on one of the three feeders so a local small industrial plant burns up their motors every Friday afternoon on a hot summer day. Single phase fans obviously would be immune. Second issue is that ampacity limits have to be respected. That means the neutral has to carry the imbalance, the highest phase has to have enough ampacity, and the transformer kVA per phase (divide by 3) has to be respected. With this type of load harmonics are an issue at the limits (ampacity, transformer). Use a full size neutral, don’t under size.

 

[winnie]

2200W divided by 240 = 9.1 amps (standard operating voltage).
2700W divided by 277 = 9.7 amps.

He's got these 400A panels maxed out. Each 20A breaker serves two machines; that's 19.4A x 20 breakers.

I read somewhere where these Switch-Mode Power Supplies will generate harmonic currents from 60 cycle ac, ie: 120 (1st harmonic),
180 (2nd harmonic). Haven't even gotten into that yet.

Ok, several points:

1) your math is wrong on current consumption, for a constant power load current goes down as voltage goes up.

2) As you know, normal breakers should not be loaded on a continuous basis to more than 80% of their trip rating. So your breakers may be frying simply because they are overloaded.

3) switching power supplies can create harmonics if they are not properly designed. These harmonics can cause more current to flow on the neutral than on the phase conductors, and can also overheat transformers. They would not have a large effect on a breaker, but the slight effect in this case might push an overloaded breaker over the edge.

Jon

 

[tortuga]

Jon beat me to it.
I second that.
Sounds like you should get an EE involved or a budget to do proper load calcs.
Not considering harmonics, you probably should not have more than 100 miners on a 400A panel, and thats balanced loads.
80% of 400A = 320 = 266kW per panel.
Then there is the branch circuits..

Just did a quick google on the miner power supplies and yeah you need a 240V system.

Amazon.com: 2600W | 2200W Power Supply for ASIC Miner Compatible with BITMAIN Antminer (2600W Power Supply): Computers & Accessories

Buy 2600W | 2200W Power Supply for ASIC Miner Compatible with BITMAIN Antminer (2600W Power Supply): Internal Power Supplies - Amazon.com ✓ FREE DELIVERY possible on eligible purchases

www.amazon.com

900 blown power supplies = $117000.00

There are usually 5 position tap changers in those Easton pad mount transformers,
but yeah you'll need to re-do your 120V transformers and if there are other customers on those they wont do it.
Byt you could probably get custom 480 to 415/240 step down's.
Keep us posted
Cheers

 

[bluescarecrow]

.

Here is the collateral damage station. 4 breakers so far; and these babies aren't cheap. You can see the discoloration in the lug extender up front. That was replaced.

The covers on the right are color coded to help the techs locate the right circuit. The first B is the second main panel outside. The 3 represents the 3rd panel in the first bay. The second B is a sub-panel. Just below that will be the breaker/circuit number. I tried painting the covers along with stenciled numbers, but I found that covering the cover with vinyl and using vinyl lettering is much faster and a lot more readable.

View attachment 2555961 View attachment 2555962

I was hired and arrived last Saturday from Phoenix; this job is up in Lakeside, AZ, where it is a little cooler. Another problem is that 3 phase fans are hooked up to the last sub-panel "D" on breakers 4, 10 & 12. They were too lazy to put a 3 phase breaker in the panel. These guys were real schlock artists.