Current limiting in 3 phase 240v computer application

[quantboy5]

Hi,
I come from the commercial/residential electrical field, but was recently presented with an industrial problem. I'm hoping someone here can give their 2 cents on it.

I have a client that has (60) -- 1200w computers, that require 240 VAC.

Currently, at the location, there is a 3 Phase, 240VAC, 2000A dead front end switchgear service, with several 200 and 400 amp breakers (not used by anything) I believe it is a 240V 3 phase bastard leg system.

What is the safest way, and best way, to power these 60 units?

I calculated so far:
1200w / 240v = 5A / unit
5A / 0.80(%) continuous load = 6.25 A

the 60 units will be placed onto a rack that holds 30 units each.

This gives us 187.5A / rack of 240V requirements.

There were a couple of options I was considering....

OPTION A)
Ignoring the 3 phase, and using just the center tapped legs...
Taking a 400A breaker off of the 2000A servie
Bringing it to a box and bugging off to (2) 200A disconnects...
each disconnect would go with SE cable to a 200A main lug SqD QO box w (10) 2 pole, 20A breakers...
every 3 devices would be connected to a 20A breaker.
* is this more expensive if I'm pulling on only two sides of the 3 phase system?


OPTION B) using a 3 phase SqD QO-B panel -- with a phase balanced load, of 3 machines per 20 A breaker....
* Is this proper?

OPTION C) using DIN rail mounted 60A breakers per 10 machines, and 10 wires going to C-13 endes for each 10 machines of 1200w each
*DOES THIS PROVIDE ADEQUATE LOAD PROTECTION IF ALL 10 machines are always running???

ANY thoughts would be appreciated.

Customer wants to have 60 machines powered by December 31st... 300 machines by feburary 31st (2000A @240V 1 phase)... and 1000 machines by August 31, 2018 (6000A @ 240v 1 phase or 2000A @240V/3phase)

 

[Ingenieur]

1000 machines at 6.25/240/3 is 3600 A

1200 w may be 1330 va due to pf

1000 x 1330 / (240 x sqrt3) x 1.25 = 4000 A 3 phase
7000 A single phase

without knowing
what the actual service is
rack config (1x30 units, sectioned 5x6 units, etc)
actual per computer unit load

hard to make a determination
I would develop the service and switchgear up front for the 1000 units
3 phase down to load branch ckts

 

[Saturn_Europa]

Are there going to be UPSs powering the servers? I've never seen a rack mounted server with out an UPS.

 

[augie47]

1st step: Don't base the plan on "maybe". Check the existing service and confirm its 240/120 (with high leg) and make sure it can handle the additional load (Plan I, II & III).
If it is 240/120 make sure your units are straight 240 and don't require a neutral.
If that all checks out I would go with Plan B and take advantage of the 3 phase system.
SaturnEuropa makes a good point and I would check for any plans for a UPS or generator system so you can take that into account from the beginning,.

 

[zbang]

Just because the power supply is rated at 1200 watts doesn't mean it'll draw that much, actual draw could be significantly less. Also, they're probably multi-rated supplies (100-240V). No modern computer power supply requires a grounded conductor, just two power leads.

UPS? That or a power conditioner.
For really important applications, you probably want dual feeds to each computer. The draw is the same, just doubles the amount of circuits.

300 units is at least 7 full racks, then you still net things like network and monitoring equipment; most people don't put more than about 32-33 1-(rack unit) computers per rack.

Cooling? Heat (watts) going into the room also has to be removed.

This size of installation usually involves some data center design folks. Are there any?

 

[Ingenieur]

1000 units or 1.2 mw is going to require a lot of cooling
100's of tons

 

[Jraef]

Use the full 3 phase and balance the loads between all 3 phases (17, 17 &16), even though it means separating circuits on the two racks. Or just add a third rack and put 20 on each so that future expansion becomes easier. Then just tell him he needs to add in multiples of 60 to maintain balance, so 60 now, 120 later, then 240 or 360 etc.

 

[Jraef]

1000 units or 1.2 mw is going to require a lot of cooling
100's of tons

Which of course adds significantly to the service load...

But once he gets his flow of cryptocurrency going, he will be able to afford it. Either that, or the floor will drop out of that fake market and you may never hear from him again. Get most of your money up front...

 

[jim dungar]

1000 machines at 6.25/240/3 is 3600 A

The OP said 60 computers.

Why did you decide to throw off the thread by using a different number?

 

[Saturn_Europa]

The OP said 60 computers.

Why did you decide to throw off the thread by using a different number?

OP mentioned adding 1000 machines by the end of 2018.

I'm with Jreaf, the whole crypto currency thing is crazy. Although, if you got into in 2014 or 2015 you would be rich right now.

 

[Ingenieur]

The OP said 60 computers.

Why did you decide to throw off the thread by using a different number?

read the whole post before finding a reason to jump on me,,,typical

 

[electrofelon]

At what point is the service going to be upgraded? Dont forget to keep the utility apprised of the additional load on the existing service. They may not be supplying it to its full potential.

 

[quantboy5]

Are there going to be UPSs powering the servers? I've never seen a rack mounted server with out an UPS.

No UPSs... These are application specific machines designed to crunch matrix calculations -- they can and will fail - and this will not cause a problem with the main objectives. Therefore no UPSs will be included.

 

[quantboy5]

Actually NOT crypto currency -- but AI - CUDA core processing of machine learningData

Actually NOT crypto currency -- but AI - CUDA core processing of machine learningData

Great replies! and THANK you... however it's not cryptocurrency --- it's actually CUDA core matix math calculations for AI and computer learning.

If I were to start with an existing 400A breaker, and step it down... would this be acceptable?

400A 3 phase fed thru 500MCM Cu to....
3 phase main lug breaker panel....
60A 2 phase breakers.... each fed to ....

custom PCB 60A DIN rail mounted breakers for phase A-B, or B-C, or C-A..... (240V)....

fed on the PCB to 8A fast acticing 250v fuses...
fed from the pcb's to the power supply units which step down 240 to 12VDC to the computers?

Wire management would be thru wire gutter, and open THHN conductors on the side of the racks...

IF I did something like the above, would that work? Is there a better way of doing it?

 

[quantboy5]

Cooling...

Cooling...

Cooling will be air-cooled because of water cooled devices... External chilling requirements will be minimal -- there will be no spinning platter hard drive units that require cooling to decrease their MTBF's -- BUT great thought under "normal" conditions.

 

[jumper]

Shouldn't a din rail mount be a MCB(miniature circuit breaker) and not PCB (printed circuit board) mount breaker?

Just a question for clarification, not trying to nit pick.

 

[zbang]

Is there a better way of doing it?

Yes, buy off-the-shelf computer power distribution units, at least for the racks (APC, Eaton, Liebert, etc). Use regular panels to feed them. You'll spend more time that it's worth to design and build your own.

External chilling requirements will be minimal -- there will be no spinning platter hard drive units that require cooling to decrease their MTBF's

Doesn't matter about drives (even SSDs don't like too much heat), the CPUs/GPUs/memory/etc need to be cooled. Unless there is enough A/C to remove the energy being added to the room, it's going to get awfully hot in there.

 

[quantboy5]

Shouldn't a din rail mount be a MCB(miniature circuit breaker) and not PCB (printed circuit board) mount breaker?

Just a question for clarification, not trying to nit pick.

The initial thought was to have a din rail on the front of each shelf of each rack.
The first current limiter would be an MCB (correct!)...
and then a custom printed circuit board, with a DIN rail back side mount...
The PCB would take 60A, and split it down to 10 channels (circuits) of 6to7 amps each, using a 5mm x 20mm fast acting glass fuse - but as someone mentioned already, custom PCB's will be time intensive to design/build/engineer properly... so I may reconsider.

 

[zbang]

Didn't occur to me this morning- if the computers are using the usual NRTL-listed power supplies, you don't need to fuse them down to 6-7 amps; all you need are 20 amp branch circuits which supply multiple computers, one 20A 240V circuit can probably handle 5-6 of them (below).

Also, some commercial PDUs take a 30 amp 3-ph branch circuit and have smaller breakers for each receptacle.

(Doing a back-of-the-envelope calculation- an intel core i7 CPU @ 150w, 16g memory ~20w, a few fans ~100w, nvidia gtx-970 ~300w = 650-700 watts running full-on.)

 

[Ingenieur]

Cooling will be air-cooled because of water cooled devices... External chilling requirements will be minimal -- there will be no spinning platter hard drive units that require cooling to decrease their MTBF's -- BUT great thought under "normal" conditions.

what?
if you have an enclosed space with btu's being released into it they must be removed
does not matter what the source
if what you are saying you will ventilate with unconditioned air you will destroy the equip due to humidity
on a warm day blowing in 90 deg air and the room at 100 little temp drop will occur
tis will be the first 1.2 mw data center I've seen without mech cooling
and I've seen a bunch
temps could exceed 120 deg