Liebert UPS, over voltage.

[SmithBuilt]

My guys were doing some work at one of my customers. They have a Liebert UPS feeding their data center. It registered an over voltage and went on bypass (utility feeding the data center). They argue that we caused the shut down by shorting something out. My guys say they did nothing to cause the problem. They were not working on any circuits fed by the UPS nor anything live.

The customer rushed to the UPS and tried to get it back online. While they were doing this the UPS disconnected the power to the data center. They argue that they did not cause the shutdown. It cost them a lot of money and many unhappy customers.

My first question is what would cause an over voltage? Could a large load that is shut off cause it? A short? There was also talk of a couple of brownouts the same day.

Second, is it possible that the customer did make a mistake and shut off the power to the data center or the UPS malfunction? I'm researching the Liebert site now.

 

[Weaver Road]

I don't have a lot of experience with these, but a couple of ideas come to mind.

First, I assume by the post that it's a computer managed UPS. Those that I have worked with have detailed event histories either available by scrolling through various button commands and screens on the unit or by downloading. The last one I inspected had internet access for remote observation!

Second, it seems that the customer might consider using the blame game to attempt to get reimbursement for their losses. That is another opportunity to ask them to prove their theory.

 

[quogueelectric]

In my experience it is usualy not equipment failure it is usually human error who is not familiar with the equipment. I have experienced simmilar scenarios and it is usually untrained operators trying to figure out how the ups works that cause the problem. Liebert is a high quality product and will perform exactly how it is supposed to depending on user input.

 

[zbang]

I'm not sure I understand what actually happened, when you say it registered an overvoltage, was that on the line or load side?

I also expect that the customer probably fat-fingered a control, which killed the loads. Some UPS controls are a bt "non-intuitive" (like the lamp test button that also closed -all- the alarm contacts, that was fun).

 

[SmithBuilt]

Weaver, I do not believe this UPS has a computer interface. There is a small display on the front of it that has limited information, not as detailed as I would like.

Z, I assumed it was on the load side. But I never ask and need to clarify this with the customer. I actually assumed that the Liebert would be able to handle a line side spike with out going into bypass.





I now believe that the customer made a mistake as far as the shut down.

I would like to know what caused the over voltage and how to prevent it from happening again. It seems the UPS would be designed to handle a reasonable spike.

 

[catchtwentytwo]

Hard to believe an over voltage occurred without a noticeable cause or any event log. Was the building on generator? Was there a lightning storm in the area? Did anyone check with other buildings in the area or the utility? Was the overvoltage on the UPS input or output?

As quoqueelectric rightfully noted, these situations are usually operator errors or some combination of events. Someone sees a minor alarm and presses the unit EPO to silence it, etc.

Unfortunately, these type of "events" too often play out like the scene in Caddyshack where the Bishop is struck by lightning. Carl Spackler (Bill Murray) looks around to see if anyone witnessed it and then slinks away.

Sadly, it will probably play out like this:

1. The manufacturer's tech will blame everything but his product. One even once attributed a failure to "dirty commercial power"!
2. Everyone will deny touching anything at the time the event occurred. "Don't blame me...I didn't do anything".
3. The utility will deny an event happened on their end.
4. Customer will join the "League of Paranoid IT Professionals". Installs cameras to monitor UPS. Will insist that work can only be done nights and weekends...in case it happens again!

 

[masterinbama]

the UPS would have more than likely come back on line if the IT guy had left it alone. Was it a static UPS to bypass transfer or did it go down and then back up? Most lieberts I have worked with have the parameters set to close from the factory and the field techs readjust them to limit unwanted problems.

 

[ron]

An overvoltage should not have sent it to bypass, it would have gone to battery/inverter if it was an actual "out of tolerance" input.
Sounds suspicious.

 

[SmithBuilt]

Hard to believe an over voltage occurred without a noticeable cause or any event log. Was the building on generator? Was there a lightning storm in the area? Did anyone check with other buildings in the area or the utility? Was the overvoltage on the UPS input or output?

There was an event log on the UPS. I'm certain it was on the load side but will verify on Monday. The building has a generator that did not come online. No storms. We ask around and no other customers in nearby buildings noticed anything.

Masterinbama
The UPS went strait to bypass.

Ron
Would that be true if the spike was on the load side of the UPS?

 

[catchtwentytwo]

There was an event log on the UPS. I'm certain it was on the load side but will verify on Monday. The building has a generator that did not come online. No storms. We ask around and no other customers in nearby buildings noticed anything.

Masterinbama
The UPS went strait to bypass.

Ron
Would that be true if the spike was on the load side of the UPS?

Did Liebert send a technician?

 

[jdsmith]

We need to install two 50 kVA UPS's and I was talking with my boss about different brands on Thursday. He said they purchased two Liebert's maybe 5 years ago and within 6 months they both failed. The problem was a bad board design in the unit and the techs that came out were well aware of the issue and even stocked the revised board in their trucks. The techs said that management was well aware of the issue and rather than write a letter to customers with that model UPS installed and make them aware of the problem they decided to wait until the units failed then replace the board. After this experience with Liebert's attitude toward their customers I don't think we're going to purchase any more UPS's from them despite their reputation.

So on the off chance that you are dealing with one of these known faulty units, make sure you have an honest talk with the Liebert tech and see if this is the issue. I'm assuming they aren't too forthcoming with this info because some customers will want to collect for lost business.

 

[ron]

I am not intimate with the particular output monitoring of your specific model (especially since I don't know what model you have), but usually a trip to bypass is related to an output overload (preset time delays depending on the %), battery rundown (DC undervoltage) or some type of internal UPS component failure.

 

[masterinbama]

If my memory is correct I had an output over voltage on a 750 kva unit one time and it traced back to the dc input or buss voltage somehow. The liebert tech tried to blame incoming power even though we had 2 identical 750 kva units on site and only one failed.

 

[catchtwentytwo]

.........The liebert tech tried to blame incoming power even though we had 2 identical 750 kva units on site and only one failed.

Typical manufacturer deflection, blame the incoming power. If that doesn't work, blame the customer. Ask the tech if there should be a pre-UPS so his unit has "clean power". I've even heard of some customers sourcing their bypass from another brand of UPS to "protect" against this type of problem.:roll:

:-? On the other hand, end users\customers sometimes don't keep up with hardware/firmware revisions or FCOs. Either they defer maintenance or use a third-party who doesn't have access to the factory. Do you know the unit was under maintenance contract with Liebert? I'd also be concerned about the batteries if they are 5 years old and VLRA type.

 

[SmithBuilt]

John, Yes the Liebert tech did come out the same day. He said no problems on his end, I hate I was not there to question him.

They do have a maintenance contract with Liebert.


The Liebert tech is coming back out on Monday for $1000 minimum. I will get the model of the UPS and let you know. The GC is paying right now just to cover all the bases.

It's played out just as you said. But the tables are starting to turn. The customer has all but admitted they may caused the shutdown. I have proven that there's no way my employees could have shut the entire data center down. They would have had to cut off 40 breakers to do so. As the only disco is in a locked room with the UPS and we did not have access to it. It just seems odd the customer was in the room trying to get the UPS back online when the data center shut off.


As far as what put the UPS into bypass has not been determined. I doubt it ever will be. If a short happens on the load side will that cause a spike large enough? I know the customer mention having to replace some equipment, he was blaming the shutdown but what if the equipment malfunctioned and caused the UPS to bypass.

 

[wireguru]

how old is the liebert? i saw a smaller (30kva iirc), older, liebert ups go on bypass when a dumba** electrican short circuited a branch circuit he was working on to trip the breaker so he wouldnt have to trace it out. Anyways, the liebert saw the output overloaded so it went into bypass quicker than the sqd QOB breaker tripped (probably 300' of #12 round trip)

 

[CHWflorida]

We installed a new Liebert a couple of weeks ago. The epo connects to a flimsy connector on the access door. I did not have the legs exactly level which allowed the door to slowly open while the Liebert tech had his back turned. The connection pulled right out and shut the room down. Not a happy customer.

 

[SmithBuilt]

Update

Update

Had a meeting with the Liebert Tech today.
Turns out it was an over current fault on phase A load side, not over voltage. I should not have taken the owners word for this, I should have made them show me the log.

The fault lasted for several minutes, long enough for the UPS to try to go back online, it still sensed the over current fault. After about 5 minutes of the fault on static bypass the UPS disconnected the load. The static bypass according to the tech is designed for up to 125% at so many minutes and at 150% less time.

We now believe someone(drywallers that don't understand English most likely) either used a vac or drill to mix mud on a UPS outlet. I understand from the Tech that a large motor can can put the UPS into overcurrent even though the total current on the UPS is less than its rated output.

The customer thought they had redundant systems, but in reality they only have one UPS and if it fails....

Thanks for the input.

 

[catchtwentytwo]

Tim,

Glad you had a successful conclusion to the issue and thank you for sharing the outcome. Don't ever assume a technician is 100% correct, they are as failable as we electricians and often overlook the obvious. I've learned in 25 years of data center installations and operations that most often the low tech causes are ignored (like the sheetrocker situation).

Stand your ground if you have a gut feeling and express it in a non-angry way, you'll be surprised how often you are close to the truth!:grin:

 

[dbuckley]

UPSs have a very limited ability to supply more current than they are rated for, so if there is an overcurrent event the UPS willl go stright into bypass, and let the utility supply the overcurrent, on the basis the "Infinite Grid" can supply a lot of current, and can open breakers. So being on bypass for a few seconds is not unusual for a UPS.