Help!! Power Quality Issues Multiple Office UPS's Going Off Randomly!

[gar]

0141023-0729 EDT

5 S Electric:

I don't believe you are using a logical troubleshooting procedure. Guessing, replacing, and try are not an effecdtive troubleshoot approach if you can get quantitative information at various points in the system. Your system does not prevent you from making measurements at various intermediate points.

To troubleshoot a system you need to know how the system works. Then you need to make measurements at different parts of the system, and correlate the data with what would make sense for the probed poiints.

You need voltage information and so far I don't see any voltage data. Fundamentally a simple UPS makes use of input voltage to determine its action.

Ask yourself how a UPS works, there are various kinds. Search the Internet for a discussion.
In Google enter the search string:
how does a ups work
Some results are:
http://computer.howstuffworks.com/question28.htm
http://en.wikipedia.org/wiki/Uninterruptible_power_supply
and others.

In posts #2 and #3 I provided you some starting points for troubleshooting.

Why do UPSs randomly change state? If they are in good working condition the reason is the UPS has detected an input voltage that is outside its threshold limits. Not all UPSs of the same model have exactly (precisely) the same voltage thresholds, nor is a threshold likely to be precisely stable over a long time.

It is unlikely you have any UPSs that can run from battery for 40 minutes after a dip in voltage unless the input voltage rises to a level that can keep the battery charged, but not sufficiently high to reconnect the load to the input power source.

.

 

[5 S electric]

0141023-0729 EDT

5 S Electric:

I don't believe you are using a logical troubleshooting procedure. Guessing, replacing, and try are not an effecdtive troubleshoot approach if you can get quantitative information at various points in the system. Your system does not prevent you from making measurements at various intermediate points.

To troubleshoot a system you need to know how the system works. Then you need to make measurements at different parts of the system, and correlate the data with what would make sense for the probed poiints.

You need voltage information and so far I don't see any voltage data. Fundamentally a simple UPS makes use of input voltage to determine its action.

Ask yourself how a UPS works, there are various kinds. Search the Internet for a discussion.
In Google enter the search string:
how does a ups work
Some results are:
http://computer.howstuffworks.com/question28.htm
http://en.wikipedia.org/wiki/Uninterruptible_power_supply
and others.

In posts #2 and #3 I provided you some starting points for troubleshooting.

Why do UPSs randomly change state? If they are in good working condition the reason is the UPS has detected an input voltage that is outside its threshold limits. Not all UPSs of the same model have exactly (precisely) the same voltage thresholds, nor is a threshold likely to be precisely stable over a long time.

It is unlikely you have any UPSs that can run from battery for 40 minutes after a dip in voltage unless the input voltage rises to a level that can keep the battery charged, but not sufficiently high to reconnect the load to the input power source.

.

You are correct. The transformer change did not help. 2 hours after we changed it some UPSs started beeping. The current on the EGC that comes from the panel did go down though. Yesterday it was at 2.5 and now 0.19.

 

[5 S electric]

I can not get the data to transfer to this site. If anybody is interested in the data collected I can email it. You will have to download Amprobe software from there web site.
http://forms.amprobe.com/shortgate....ent.amprobe.com/softwareDownloads/DSSetup.exe

I Really am at a loss here.

 

[don_resqcapt19]

You are correct. The transformer change did not help. 2 hours after we changed it some UPSs started beeping. The current on the EGC that comes from the panel did go down though. Yesterday it was at 2.5 and now 0.19.

I strongly doubt that changing the transformer had anything to do with the change in the current on the EGC.

 

[mike_kilroy]

since we are guessing... try this:

1) how many computers are on floor 1?
2) how many computers are on floor 2?
3) how many computers are on floor 3?

I will explain why I ask this after you reply.

 

[gar]

141023-2005 EDT

5 S electric:

Use a Fluke 27, 87, or even a Simpson 260 to measure voltage. I don't care whether it RMS or average reading. For the purpose here the Simpson is not the best, but if it is a taut-band model in good condition, then it should be adequate.

When no UPSs are running on backup measure the secondary voltages on one transformer. For this one transformer next make UPS input voltage measurements near several UPSs that have previously gone into backup mode.

Repeat these voltage measurements at a time when one or more of the UPSs measured above have indicated going into the backup state.

If we assume the voltage problem is not at the input to the above selected transformer, then we are likely to see a higher or lower voltage at the input to at least one of the UPSs resulting from some problem between the transformer primary and the UPS.

To see the effect of wiring impedance see photos P26 and P29 near the end of my website page http://beta-a2.com/EE-photos.html . Power and voltage are averaged over a 1 second period, and the plots have a 1 second resolution.

The power and voltage measurements in P26 are at my main panel. There are very small voltage variations resulting from my variation in load power. There is a greater overall variation that is the result of the power company primary voltage variation.

The measurements in P29 are for a single load, a 20 cubic foot upright freezer. Both the voltage and power variations during motor start are greater than plotted because these are shorter than 1 second. The displayed values are diminished in magnitude by the 1 second averaging. If my nominal wall voltage was around 105 V instead of about 125 V, then the motor starts would probably trigger a UPS into backup.

 

[5 S electric]

since we are guessing... try this:

1) how many computers are on floor 1?
2) how many computers are on floor 2?
3) how many computers are on floor 3?

I will explain why I ask this after you reply.

1- 1 big computer lab and some offices
2- Manley offices
3- 2 Med sized computer labs and offices

sorry i would take a while to count all of them.

 

[tom baker]

I would recommend a data logger to record voltage. Fluke makes a very simple one that plugs into a receptacle.
Hobo data loggers have been around a long time and have many options
http://www.onsetcomp.com/products/hobo-data-loggers?gclid=CJvdqMv318ECFUxsfgodDZ4A1g

The full blown data logger from fluke will tell you more than you need to know, IE volts amps harmonics power factor etc.
Start with voltage first

Some UPSs have Ethernet cards or USB connections and can be used to log voltage, but they often do not log enough IE every 15 seconds or minutes.

 

[5 S electric]

I would recommend a data logger to record voltage. Fluke makes a very simple one that plugs into a receptacle.
Hobo data loggers have been around a long time and have many options
http://www.onsetcomp.com/products/hobo-data-loggers?gclid=CJvdqMv318ECFUxsfgodDZ4A1g

The full blown data logger from fluke will tell you more than you need to know, IE volts amps harmonics power factor etc.
Start with voltage first

Some UPSs have Ethernet cards or USB connections and can be used to log voltage, but they often do not log enough IE every 15 seconds or minutes.

We are monitoring a ups via ethernet. I do have an am probe dmiii but it isn't catching anything out of the norm. I am not to happy with this meter. For 3k this meter should catch everything. It is set as tight as I can get it at 5 sec intervals.

 

[iceworm]

5 S -
I'm going to be repetative - so okay to ignore.

... Short of bad connections and flat burning up, transformers don't tend to cause power problems. ...

... I'd suggest getting a quote for a report on the data collected. Just data collection alone won't help. ...

.... For 3k this meter should catch everything. It is set as tight as I can get it at 5 sec intervals.

... The full blown data logger from fluke will tell you more than you need to know, IE volts amps harmonics power factor etc. ...

As noted, changing transformers with no evidence there is anything wrong is wasted money and wasted effort - inhouse labor is not free.

You do need a full blown data logger AND someone to interpret the results. The one I am currently using is a Dranetz 5P. It's ~7 years old, but works well. I generally tend to set up a waveform capture every few seconds as well as a disturbance triggered 100 - 1000cycle waveform capture pre and post event. I could be a few days going over the data, and writing the report.

In your case, you may also want to install a trigger from one/multiple UPS(s) to also trigger a pre and post capture. And you may wish to set up two or even three to catch the location of the disturbance.

Another I am looking at is an elspec G4500. If I get another this will likely be my choice over a Fluke 1750.

As gar and others, (as well as I) discussed, guessing isn't going to get it. Bite the bullet:

Spend the money (on something besides transformers)
Hire a specialist
Put together a systematic troubleshooting plan to track the culprit.​

And yes, I'm sure there are plenty out there that can interpret your data. However, you are looking for transients - possibly sub-millisec. A capture every 5 seconds may never see anything. And I don't know what your meter looks at. It may not even be looking for transients - just recording steady state, in which case it won't ever see anything.

$3K is no garrentee of the instrument doing what you need. The Dranetz 5P was $12K, the Elspec G4500 is ~$10K, the Fluke 1750 (last time I looked) was ~$20K (all with the needed options).

Ot, keep guessing - you may indeed get it. Truely, that does sometimes happen.

ice

 

[gar]

141101-1011 EDT

5 S electric:

So far you have provided no quantitative data for us to see.

In posts #21 and #26 I have ask you to make some simple voltage measurements that do not require any recording or expensive equipment.

Pick one of the transformers.

When operation is normal, no UPSs beeping, measure all of the various combinations of secondary voltages at the transformer. Next go to one of the UPSs that typically beeps when there is some perturbation, and measure the wall outlet voltage where the UPS is connected.

Repreat these measurements at a time when the UPS is tending to beep.

Provide these values to us.

.

 

[5 S electric]

141101-1011 EDT

5 S electric:

So far you have provided no quantitative data for us to see.

In posts #21 and #26 I have ask you to make some simple voltage measurements that do not require any recording or expensive equipment.

Pick one of the transformers.

When operation is normal, no UPSs beeping, measure all of the various combinations of secondary voltages at the transformer. Next go to one of the UPSs that typically beeps when there is some perturbation, and measure the wall outlet voltage where the UPS is connected.

Repreat these measurements at a time when the UPS is tending to beep.

Provide these values to us.

.

That is the problem I can not catch when the ups, s are going off. I would have to camp out next to it. Notice that it only happens during working hours.

 

[gar]

141106-2020 EST

5 S electric:

I don't believe you understand what I am trying to get at.

When there is no great activity on the secondary side of your transformer, then what are the nominal voltages at the transformer, and in the area of the input of a UPS where there have been "beeps"?

Then at some time when the transformer is at heavy load what are the voltage measurements at the same locations?

Suppose that at heavy load the nominal voltage is around 106 V. That the UPS triggers into backup at 100 V and returns to non-backup at 105 V. Next a 7 V droop occurs for 32 millseconds, and returns to 106 V. The UPS will likely transfer to backup for a moment, beep, and return to non-backup. If it only returned to 104.9 V, then it would stay in backup until it got above 105 V, or ran out of battery energy.

If the nominal voltage at heavy load was 120 V, then a 7 V droop would not trigger backup.

Without some numbers we have no way of judging what is your relative source impedance.

If you can not be there, then leave your meter and instruct someone in the area to get the information.

Some important questions are:
(1) Do you have a source impedance problem?
(2) Is the nominal voltage low?
(3) Is the problem on the transformer primary side, or somewhere on the secondary side?
(4) I believe you have indicated that sometimes the UPSs go into backup for a long time. Is this correct? I doubt, that unless the UPSs are special, that they will last more than a few minutes with a reasonable load.

If your nominal voltages under load are in the 120 V range, and unloaded don't rise much, then you will need recording equipment with short time resolution to investigate the nature of the down or up spike, and whether or not this is also present directly at the secondary of the transformer.

.

 

[5 S electric]

141106-2020 EST

5 S electric:

I don't believe you understand what I am trying to get at.

When there is no great activity on the secondary side of your transformer, then what are the nominal voltages at the transformer, and in the area of the input of a UPS where there have been "beeps"?

Then at some time when the transformer is at heavy load what are the voltage measurements at the same locations?

Suppose that at heavy load the nominal voltage is around 106 V. That the UPS triggers into backup at 100 V and returns to non-backup at 105 V. Next a 7 V droop occurs for 32 millseconds, and returns to 106 V. The UPS will likely transfer to backup for a moment, beep, and return to non-backup. If it only returned to 104.9 V, then it would stay in backup until it got above 105 V, or ran out of battery energy.

If the nominal voltage at heavy load was 120 V, then a 7 V droop would not trigger backup.

Without some numbers we have no way of judging what is your relative source impedance.

If you can not be there, then leave your meter and instruct someone in the area to get the information.

Some important questions are:
(1) Do you have a source impedance problem?
(2) Is the nominal voltage low?
(3) Is the problem on the transformer primary side, or somewhere on the secondary side?
(4) I believe you have indicated that sometimes the UPSs go into backup for a long time. Is this correct? I doubt, that unless the UPSs are special, that they will last more than a few minutes with a reasonable load.

If your nominal voltages under load are in the 120 V range, and unloaded don't rise much, then you will need recording equipment with short time resolution to investigate the nature of the down or up spike, and whether or not this is also present directly at the secondary of the transformer.

.

The voltage readings range from 121v to 122 through all 3 phases on the secondary. The transformer never really gets a good load on it. It is a 75kva and sees 30 amps max per leg. I was there yesterday when the problem was occurring. The volt meter only went down to 119.8. It is happening so fast that the meter won't pick it up. So i tried another meter that has the min max setting on it and it does not catch it either lowest it showed was 119.4. Here is a log that the UPS is sending me.
11/07/2014 10:50:12 120.9 121.6 120.9 2.16 25.3 60.00 100.0 27.40 87.8 25.3 82 10.0
11/07/2014 10:40:12 120.9 121.6 121.6 2.16 25.3 60.25 100.0 27.40 87.0 25.3 82 10.0
11/07/2014 10:30:12 115.9 122.4 120.9 2.16 25.3 60.00 99.0 26.86 87.0 25.3 82 10.0
11/07/2014 10:20:12 120.2 121.6 121.6 2.16 25.3 60.00 100.0 27.40 87.0 24.0 82 10.0
11/07/2014 10:10:12 120.2 121.6 120.9 2.16 26.0 60.00 99.0 27.13 86.1 25.3 82 10.0
11/07/2014 10:00:11 115.9 122.4 120.9 2.16 25.3 60.00 99.0 26.86 86.1 25.3 82 10.0
11/07/2014 09:50:11 120.9 121.6 121.6 2.16 25.3 60.00 100.0 27.40 86.1 25.3 82 10.0
11/07/2014 09:40:11 120.9 123.1 121.6 2.16 25.3 60.00 100.0 27.40 86.1 25.3 82 10.0
11/07/2014 09:30:11 120.9 121.6 121.6 2.16 25.3 60.00 100.0 27.40 86.1 24.0 82 10.0
11/07/2014 09:20:11 120.9 121.6 121.6 2.16 25.3 60.00 100.0 27.40 86.1 25.3 82 10.0
11/07/2014 09:10:11 120.9 121.6 121.6 2.16 24.0 60.00 100.0 27.40 86.1 25.3 82 10.0
11/07/2014 09:00:11 120.9 121.6 120.9 2.05 24.0 60.00 100.0 27.40 86.1 25.3 82 10.0
11/07/2014 08:50:11 118.8 122.4 120.9 2.05 25.3 60.00 100.0 27.54 85.4 25.3 82 10.0
It shows it down to 115.9 volts

 

[gar]

141107-2053 EST

5 S electric:

One data sheet for a Fluke 87 indicates a min-max response time of 1 millisecond.

You need a meter with a shorter min-max response time.

Your recorded data from a UPS implies the UPS is a three phase load, and probably three phase output. I don't know what many of your columns are. I can identify date, time, three voltages, and frequency.

I have to assume at this time that the voltage is some average over 10 minutes (not very useful), or some sort of sample every 10 minutes (still not useful). I would have expected the UPS to give you some sort of transient information.

It is interesting to note that two of the voltages have a somewhat inverse relationship. One change is larger than the other. Use 120.9 for the first voltage reference, and 121.6 the reference for voltage two.
At 0850 one is -2.1 V, and two is +0.8 V.
At 0900 one is -0.0 V, and two is +0.0 V, this continues thru 0930.
At 0940 one is -0.0 V, and two is +1.5 V, independent bump up in voltage two..
At 1000 one is -5.0 V, and two is +0.8 V, repeats again at 1030.

If you plot the curves it is easier to see the correlation.

Your nominal voltage is probably at an OK level. You need a faster response instrument to catch the large transients that are probably present. Then determine if these originate on the primary or secondary side of the transformer. Look for a correlation between the two phases.

You can afford better instrumentation based on what you spent on transformers.

.

 

[Scali]

Had this Happen to me.

Had this Happen to me.

What we found was a loose connection with the neutral in a multi-branch circuit.
If I were you I would check the homerun boxes.


Hope this Helps.

 

[ActionDave]

What we found was a loose connection with the neutral in a multi-branch circuit.
If I were you I would check the homerun boxes.


Hope this Helps.

From what I can tell he is not opening any boxes or even putting a meter on anything. He's just got some data on a screen and some transformers sitting around that he hopes will make the problem go away.

 

[5 S electric]

From what I can tell he is not opening any boxes or even putting a meter on anything. He's just got some data on a screen and some transformers sitting around that he hopes will make the problem go away.

We have taken a lot of readings in numerous places. We have opened a lot of j-boxes, panels, and transformers. When it glitches it is so fast that the meter doesn't catch it. What ever is making it glitch affects all of the circuits on the 3rd floor. There is 2 transformers that feed the 3rd floor. so it makes me think that the 480 feed is bad. But the other 2 floors have no problem. We have gone through the MDP, sub-panels, and transformers with an infrared camera. The last time it glitched was on the 8th of Nov. It seams to be happening less frequently the past 3 weeks. Do not think that i am just sitting around looking at logs. I have tried everything I can think of!!!

 

[5 S electric]

141107-2053 EST

5 S electric:

One data sheet for a Fluke 87 indicates a min-max response time of 1 millisecond.

You need a meter with a shorter min-max response time.

Your recorded data from a UPS implies the UPS is a three phase load, and probably three phase output. I don't know what many of your columns are. I can identify date, time, three voltages, and frequency.

I have to assume at this time that the voltage is some average over 10 minutes (not very useful), or some sort of sample every 10 minutes (still not useful). I would have expected the UPS to give you some sort of transient information.

It is interesting to note that two of the voltages have a somewhat inverse relationship. One change is larger than the other. Use 120.9 for the first voltage reference, and 121.6 the reference for voltage two.
At 0850 one is -2.1 V, and two is +0.8 V.
At 0900 one is -0.0 V, and two is +0.0 V, this continues thru 0930.
At 0940 one is -0.0 V, and two is +1.5 V, independent bump up in voltage two..
At 1000 one is -5.0 V, and two is +0.8 V, repeats again at 1030.

If you plot the curves it is easier to see the correlation.

Your nominal voltage is probably at an OK level. You need a faster response instrument to catch the large transients that are probably present. Then determine if these originate on the primary or secondary side of the transformer. Look for a correlation between the two phases.

You can afford better instrumentation based on what you spent on transformers.

.

We have a $3,000 Amprobe DMIII meter that doesn't even catch these glitches. Here is another sample of the log that the UPS takes.
Date Time Vmin Vmax Vout Iout %Wout Freq %Cap Vbat TupsF %VAout T1ambF %Hum1 T2ambF
14:00:55 120.9 121.6 121.6 2.16 25.3 60.00 100.0 27.40 80.6 25.3 75 10.0
11/14/2014 13:50:55 120.9 121.6 120.9 2.16 25.3 60.00 100.0 27.54 81.3 25.3 75 10.0
11/14/2014 13:40:55 120.9 121.6 121.6 2.16 25.3 60.00 100.0 27.54 81.3 25.3 77 10.0
11/14/2014 13:30:55 120.9 122.4 121.6 2.22 25.3 60.00 100.0 27.54 82.2 26.0 77 10.0
11/14/2014 13:20:55 121.6 121.6 121.6 2.22 26.0 60.00 100.0 27.54 82.2 25.3 77 10.0
11/14/2014 13:10:55 120.9 122.4 121.6 2.16 25.3 60.00 100.0 27.54 82.9 25.3 77 10.0
11/14/2014 13:00:55 120.9 122.4 120.9 2.16 25.3 60.25 100.0 27.54 83.8 25.3 78 10.0
11/14/2014 12:50:55 121.6 121.6 121.6 2.16 25.3 60.00 100.0 27.54 83.8 25.3 78 10.0
11/14/2014 12:40:55 121.6 122.4 121.6 2.05 25.3 60.25 100.0 27.54 84.5 24.0 78 10.0
11/14/2014 12:30:55 121.6 123.1 122.4 2.16 25.3 60.00 100.0 27.54 85.4 25.3 80 10.0
11/14/2014 12:20:55 121.6 122.4 121.6 2.22 26.0 60.25 100.0 27.54 85.4 25.3 82 10.0
11/14/2014 12:10:55 121.6 123.1 121.6 2.05 25.3 60.00 100.0 27.40 86.1 25.3 82 10.0
11/14/2014 12:00:55 121.6 121.6 121.6 2.22 25.3 60.00 100.0 27.40 86.1 25.3 84 10.0
11/14/2014 11:50:54 121.6 121.6 121.6 2.16 25.3 60.00 100.0 27.40 86.1 25.3 84 10.0
11/14/2014 11:40:54 121.6 121.6 121.6 2.22 26.0 60.00 100.0 27.40 86.1 25.3 84 10.0
11/14/2014 11:30:54 121.6 121.6 121.6 2.16 25.3 60.00 100.0 27.40 86.1 25.3 84 10.0
11/14/2014 11:20:54 120.9 122.4 121.6 2.16 25.3 60.00 100.0 27.40 86.1 25.3 84 10.0
11/14/2014 11:10:54 120.9 121.6 121.6 2.16 25.3 60.00 100.0 27.40 86.1 25.3 84 10.0
11/14/2014 11:00:54 120.9 121.6 120.9 2.16 25.3 60.00 100.0 27.40 86.1 25.3 84 10.0
11/14/2014 10:50:54 120.9 121.6 121.6 2.16 24.0 60.00 100.0 27.40 86.1 25.3 84 10.0
11/14/2014 10:40:54 121.6 121.6 121.6 2.16 25.3 60.25 100.0 27.40 86.1 25.3 84 10.0
11/14/2014 10:30:54 120.9 121.6 121.6 2.16 25.3 60.00 100.0 27.40 86.1 26.0 84 10.0
11/14/2014 10:20:54 121.6 123.1 121.6 2.16 25.3 60.00 100.0 27.40 86.1 25.3 84 10.0
11/14/2014 10:10:54 120.9 121.6 121.6 2.16 25.3 60.00 100.0 27.54 85.4 25.3 84 10.0
11/14/2014 10:00:54 121.6 121.6 121.6 2.16 24.0 60.00 100.0 27.40 85.4 25.3 84 10.0
11/14/2014 09:50:54 120.9 121.6 121.6 2.05 25.3 60.25 100.0 27.40 86.1 25.3 84 10.0
11/14/2014 09:40:54 120.9 121.6 121.6 2.16 24.0 60.25 100.0 27.40 85.4 25.3 84 10
The UPS never tripped in this log. The log only goes back a day and a half.

 

[mike_kilroy]

...It seems to be happening less frequently the past 3 weeks. Do not think that i am just sitting around looking at logs. I have tried everything I can think of!!!

EVERYTHING?

I recently received THIS reply from a colleague about an explainable technical issue:

I showed Hurley the email from you with Tony?s email address. He thinks it?s a government conspiracy. But Hurley thinks everything, including the weather, is a government conspiracy. That is why we think there is a dedicated satellite over our office, just to keep an eye on Hurley.

Have you checked the SKY above the 3rd floor?

(sent in fun of course, to help stop you from pulling out ALL of your hair!)