WHAT WENT WRONG?

[dontheo]

First, a big thank you to everyone that helpe me through my 3 phase pump problem. I made the installer do his job as he should have and we are in specification.

So now this. The plant service tch called and said the generator was running. No faults on any of the wiring, and everything looked good. e

This pictre is from the front cover of transfer switch for a 125KW Kohler Generator and is the logic board. It actually is on the door itself. It is covered by a black plastic cover I removed to take these pictures. The bottom right of the board shows the two wires completely burned off and the flash and charring moving from the bottom left to the top right. The reason I mention this is two companies decribed this to me before I told them what it looked like. With no problems in the past, a stringent maintenance program I maintain that irritates the hell out of everyone, I am at a loss for the reason why. It looks like something shorted to ground. This is Texas, and even in an outdoor enclosure there were wasps inside the cabinet. Or as we call them, Warsps. Not sure if they had anything to do with it or not. I could not find any fired or parts of one like it had taken on a claynore. Just thought I would post this.

The Don

 

[GoldDigger]

I am at a loss for the reason why. It looks like something shorted to ground. This is Texas, and even in an outdoor enclosure there were wasps inside the cabinet. Or as we call them, Warsps. Not sure if they had anything to do with it or not. I could not find any fired or parts of one like it had taken on a claynore. Just thought I would post this.

Makes you stop and think for awhile. Are the two disconnected wires actually burned through or did they just get hot enough to pull out of the empty solder eyelets we see on the board?
You can often figure out the origin of the flash by looking at shadowing on the sides of components, but it appears that whatever happened reflected off the plastic cover and pretty much coated everything.
The generator manufacturer may be interested in taking a look at this, whether it becomes a warranty issue or not.
If there are no damaged or missing components on the board, it really is hard to guess what started it.

Are the two wires power or sense/control wires from elsewhere?

 

[kwired]

The dates on the photos say this has not happened yet

Maybe if we figure it out we can prevent it from happening - next December.

 

[ceb58]

Have you removed the board to look at the back? Reason I ask is your mention of wasp in the cabinet. I had dirt dupers build a nest on the back of a circuit board on an AVR on a generator. It was fine until we had a lot of rain and high humidity and the nest became moist causing the board to short across the terminals.

 

[GoldDigger]

The dates on the photos say this has not happened yet

Maybe if we figure it out we can prevent it from happening - next December.

Think outside the US box. That is not December sixth. It is the much more logical 12 June. smallest unit first.

 

[eHunter]

Have you removed the board to look at the back? Reason I ask is your mention of wasp in the cabinet. I had dirt dupers build a nest on the back of a circuit board on an AVR on a generator. It was fine until we had a lot of rain and high humidity and the nest became moist causing the board to short across the terminals.

Do you mean it become an "automatic voltage unregulator"?

 

[kwired]

Think outside the US box. That is not December sixth. It is the much more logical 12 June. smallest unit first.

I thought of that after posting, but would still like to be the guy that invents the camera that takes pictures of the future, would be a pretty high demand PM tool I would think

 

[GoldDigger]

I thought of that after posting, but would still like to be the guy that invents the camera that takes pictures of the future, would be a pretty high demand PM tool I would think

The possibilities for blackmail as well as the business and bar bet profits could be amazing too.

 

[hurk27]

First, a big thank you to everyone that helpe me through my 3 phase pump problem. I made the installer do his job as he should have and we are in specification.

So now this. The plant service tch called and said the generator was running. No faults on any of the wiring, and everything looked good. View attachment 8681View attachment 8682e

This pictre is from the front cover of transfer switch for a 125KW Kohler Generator and is the logic board. It actually is on the door itself. It is covered by a black plastic cover I removed to take these pictures. The bottom right of the board shows the two wires completely burned off and the flash and charring moving from the bottom left to the top right. The reason I mention this is two companies decribed this to me before I told them what it looked like. With no problems in the past, a stringent maintenance program I maintain that irritates the hell out of everyone, I am at a loss for the reason why. It looks like something shorted to ground. This is Texas, and even in an outdoor enclosure there were wasps inside the cabinet. Or as we call them, Warsps. Not sure if they had anything to do with it or not. I could not find any fired or parts of one like it had taken on a claynore. Just thought I would post this.

The Don

I know you guys down there just went through some storms, and the damage is very similar to a lot of arcing, so my best guess would be that it took a direct lightning hit to the control conductors, damage like that looks like major over voltage, are there other signs in the building of damaged electronics? which may or may not exist if the building is clad in metal sheathing? even if it is a all wood structure lightning may not cause damager inside, this is because lightning is a high frequency event and can take what would seem to many, very strange paths.

Are the control conductors run in the same conduit as the line conductors? if so maybe there was a failure from the line to control conductors? but what I'm seeing in the photo looks more like a much higher voltage arcing, so I'm kind of leaning toward lightning and yes lightning can damage one item and never touch anything else.

If you can point to a storm at the time of the damage then the owners should be able to be reimbursed by their insurance company as long as when the insurance company checks the lightning net and it shows strikes in the area they can't dispute it.

 

[hurk27]

To add to the above, after clicking of the photo the second time to get a larger version of the photo, we can see that the areas where the ends of the two wires point toward we can clearly see a pattern of explosive coronal discharge by the lack of soot in those areas, for the black wire this is the area to the left of the original connection point on the back metal cabinet and for the white wire the clear pattern on the metal can of the capacitor to the right of the white wire, this is most likely where the discharge arced to, the ends of each conductor looks like they were vaporised off as there would have been a little bit more conductor showing if they had been simply just came unsolder, and they seem to have a rounded end to the conductor indicating the same, noticing the small clean area around each board connection point tells me that there was an explosive flash arc at each of these two points as well as the connection point of the disk MOV under the wires.

Where do these two wire come from?, they look like the VSS supply to the board? even the choke just a little down from the MOV looks like it had a arc to its copper as it shows up brightly as if some of the copper was flashed off.

So again this kind of damage clearly looks like a high voltage hit as from a lightning event.

The problem is that the alternator could also have been hit and it's winding insulation pin holed if not shorted, the transfer switch coils could have also been damaged, so there is no telling how much could have been damaged.

If lightning is a possibility then I would look for signs of a point of contact to back up this theory, but sometimes its not always possible to find it as it could hit something that is not easily damaged like a large cast iron storm sewer lid, or some heavy metal structure member near by, other signs I would look for are any other signs of arcing in any of the other electrical cabinets or transfer switch's associated with this generator, even transformers if such apply.

Lightning is one of the worst things to deal with in our field as no matter how much protection you install or design for, it can always still do damage, we can only lower the risk but we can never eliminate it totally, never guarantee protection from lightning as you will loose in the end.

 

[Fulthrotl]

To add to the above, after clicking of the photo the second time to get a larger version of the photo, we can see that the areas where the ends of the two wires point toward we can clearly see a pattern of explosive coronal discharge by the lack of soot in those areas,.......

my first thought was a cap blew, but i can't tell from the photos if one of them had a hole in it.

the choke does seem to be ground zero here. i'd look at the backside of the board
and see if there is a point of origination there. the plastic cover does seem to have
had the bbq smoker effect and hosed everything.

with two people describing this failure to the OP without seeing it, is this a chronic
failure on this piece of equipment?

 

[ATSman]

First, a big thank you to everyone that helpe me through my 3 phase pump problem. I made the installer do his job as he should have and we are in specification.

So now this. The plant service tch called and said the generator was running. No faults on any of the wiring, and everything looked good. View attachment 8681View attachment 8682e

This pictre is from the front cover of transfer switch for a 125KW Kohler Generator and is the logic board. It actually is on the door itself. It is covered by a black plastic cover I removed to take these pictures. The bottom right of the board shows the two wires completely burned off and the flash and charring moving from the bottom left to the top right. The reason I mention this is two companies decribed this to me before I told them what it looked like. With no problems in the past, a stringent maintenance program I maintain that irritates the hell out of everyone, I am at a loss for the reason why. It looks like something shorted to ground. This is Texas, and even in an outdoor enclosure there were wasps inside the cabinet. Or as we call them, Warsps. Not sure if they had anything to do with it or not. I could not find any fired or parts of one like it had taken on a claynore. Just thought I would post this.

The Don

From the pics it looks like the controller motherboard is from a Kohler ATS. The harness plugs are the same type used by ASCO on their ATSs.
Attached you will find pics and report of a similar blowup on an ASCO controller on an ATS inside a phone co central office. Very clean environment. The voltage at the plug pins that was blown apart was 480V. In some cases we have found the cause to be arcing across the pins in the harness plug which are only 3/32 of an inch apart. Other times the arcing occurred between the tracings on the motherboard. This failure occurred during a monthly routine transfer of power test. No storms or lightning were present during the failure.
Most of the failures we find happen during transfer of the ATS which is the time when the equipment wiring and components are at their highest stress points.
Normally we end up making the repairs, but in this case we recommended that the customer call in the manufacturer to clean up the mess

Also attached is a discussion I had with 2 PE's on whether installing SPDs at the ATS terminals would have limited the damage caused by this failure.
Tony

 

[dontheo]

I ws trying to find some information on a common problem becasue the service companies dezcribed it to me befor eI copuld explain it to them. All I cam up with was articles on ASCO.

The weather was perfect here 90 degrees with 90 % humidity. Hasn't raind in two weeks and their was no lightning.

I really appreciate everyones input. I'll try to give feedback on the forum when I can.

Thanks

Ted

 

[massfd]

Think about the voltages on that conector before the generator syncs and transfers, 480 X 2 when the 2 sources are 180 out of phase. This will happen going utility to gen and again going gen to utility on a test.

 

[eHunter]

If that is an ASCO or Kohler branded ASCO built switch controller PCB it may contain the secrects of its demise in the log history NVR memory.
It may take the manufacturer to extract the data, unless ... the memory IC is damaged beyond being readable, then all is lost.
If it is readable, it would show the parameters of its last activities including date, time and ATS switching status.

 

[dontheo]

It is a Kohler transfer switch which I believe is ATS.

So what is the answer of what to do when the system has to work? I am familiar with NFPA 110 and I am brutal when it comes to facility maintenance. Most Boards don't get involved at this level. We tried to by the best we could and thoght we had.

It's possible it is not the connector but the more research I do and with you help on the forum, it looks like it to me.

Like Dad always said, don't be afraid to buy the best.

 

[eHunter]

It is a Kohler transfer switch which I believe is ATS.

So what is the answer of what to do when the system has to work? I am familiar with NFPA 110 and I am brutal when it comes to facility maintenance. Most Boards don't get involved at this level. We tried to by the best we could and thoght we had.

It's possible it is not the connector but the more research I do and with you help on the forum, it looks like it to me.

Like Dad always said, don't be afraid to buy the best.

It is my experience that CAT, Cummins/Onan, Kohler, MTU, and Wartsilla are all at the top of the power gen industry.
I am assuming that this unit is 480V, is that correct?
Is the unit setup to auto exercise per a schedule?
Is it set to transfer to a live load or connect to a load bank during exercising?
Were there any odd weather conditions 24-72 hours before the flashover?
High winds, driving rain or really cool nights followed by warm/hot humid mornings?
The reason I ask is I have seen morning condensation collect on the interior of genset equipment, especially on outdoor mounted equipment, particularly ATS, conductors, interior panels, switch gear and gen heads.
I have seen similar flashover events that were the delayed results of latent storm damage before where the flashover event occurred about a month later.
My recommendation is to enlist Kohler's help. They probably have seen this type of failure before and may be able to provide some insight on why, and how to prevent a recurrence.
It may something as simple as a spray-on conformal coating for the control PCBs.
Also, I would recommend that the whole system be thoroughly inspected for other malfunctions, signs of damage, carbon tracking, arc spatter, flashover, etc before putting back in service, a sort of mini recommissioning.

I don't belive that kohler is inferior, and by all means do not want to infer that selecting Kohler was a mistake.

 

[dontheo]

eHunter

Thanks. This is Texas. There was no weather for a week prior but usually, as they say, if you don't like the weather here just wait 24 hours. That's one of the reasons it is critical that it works every time. The weakest link is usually the circuit breakers that are rarely checked.

We are classified as a critical infrastructure under one of the Presidential orders. I don't know of a group who doesn't have something to say about what we do.

I agree on the condensate. That would make sense for this time of year here. I would expect the board to be conformal coated. I will contact the manufacturer and work with them. I was just frustrated.

But I also feel that we have a responsibility to our clients. Most Districts just say oh well, replace it, and move on as money is not an issue.

 

[kwired]

eHunter

Thanks. This is Texas. There was no weather for a week prior but usually, as they say, if you don't like the weather here just wait 24 hours. That's one of the reasons it is critical that it works every time. The weakest link is usually the circuit breakers that are rarely checked.

We are classified as a critical infrastructure under one of the Presidential orders. I don't know of a group who doesn't have something to say about what we do.

I agree on the condensate. That would make sense for this time of year here. I would expect the board to be conformal coated. I will contact the manufacturer and work with them. I was just frustrated.

But I also feel that we have a responsibility to our clients. Most Districts just say oh well, replace it, and move on as money is not an issue.

No weather for a week? Sounds like you must be on the moon Sunny and mild is still weather:happyyes:

 

[cowboyjwc]

No weather for a week? Sounds like you must be on the moon Sunny and mild is still weather:happyyes:

That's true, and here 90% humidity is, rain.