Eaton 9170+ UPS modules dropping off for no reason

[11bgrunt]

This company has many radio tower sites, and most are using the 9170+ UPS. Single phase 240/120 for emergency and for filtering the POCO power.
I was asked to check power quality from the utility to the UPS at this one site. Utility power only, no genset. Control house is climate controlled.
They were using only one module on test day. At this site, they have changed batteries and changed modules. Sometime the module runs for an hour, sometimes a day. Move the modules to another tower site and no problem with the module.
With only one module running and the risk of an outage to the tower, testing on the load side was limited. PMI voltage recorders were set in the meter socket for a week, and no PQ issue was seen. Thermal imaging on the UPS tower has been done
Next week they plan to move this site to bypass so more UPS load testing can be done.
This is a rare problem for this company, but they have experienced it before, and the cure was total removal and replacement with a DC plant.
There wasn’t information on alarms. The communications group receives a notification that the site is down and when they show up, the module is offline.
We can resistively load the UPS and have meters to see the sine wave.
What test can be done to isolate and correct this problem?

Thanks,

 

[ModbusMan]

Are the modules displaying any sort of fault code, or are they completely dead by the time someone gets there? Things I would be interested in checking out...
- You say the control house is climate controlled, but does the A/C (or heater, for that matter) restart immediately after power failure, is there a delay, or is manual intervention required?
- Batteries are the absolute weakest link in the chain, and a single bad jar can bring down the entire stack. VRLA are also very heat sensitive, IIRC the rule of thumb was something like "every 15degF above 77 cuts the life expectancy in half." Given that the batteries in this thing are likely 12v 7ah which sometimes don't even see their third birthday, and that there's a series-parallel arrangement going on... well, that's a LOT of potential failure points. I'll let you do the math

TBH, I'm a little surprised they're not going with DC plants and TPPL batteries from the start. Our reliability engineer got his hands on a set of those jars when a neighboring telecom was doing a scheduled replacement and put them through all sorts of "there's no way they'll survive" experiments, including draining a couple to something like 1.5vpc and letting them sit for six months. When he came back to them, they all took charge and have been running his home lab since.

 

[11bgrunt]

We were able to visit the tower site and move the UPS to bypass. All load was moved to utility. This allowed us to test the load side of the UPS with the one module running that has never dropped off. A Fluke 345 PQ meter allowed us to view the sine wave, and we were setup to control the test load.
A transient was seen in the output. Very small and not consistent enough to know when it would appear. That module had always been in service when all the others failed. That module was removed, and others were racked in. Even with the UPS loaded well above normal, the modules did not drop out. Four days later, the UPS is running with three modules and no concerns.
The funny part to the story is that the utility circuit started operating and eventually locked open from a fault on their system. This forced us to move all load back to the UPS to get the network back up. It was a fun day.