Unexplainable constant electronics damage

[GITRDUN]

I had started a thread on this discussion a few months ago but thought i would start a new one as the other had gotten rather large in posts to sort through and i think i now have a more narrowed down idea as to where the problem might be. Here is a link to the other post if you want to do a lot of reading. http://forums.mikeholt.com/showthread.php?t=163677 .

First a little information about our situation. I manage a small machine shop with around 20 + CNC machines. We are having on average about 3 to 4 machines blow a cuircut board every month. Obviously this is a big problem for us in many different ways.
The same problem plagues all CNC machinery whether it is new or old.

Power on and off sequence for machines is as follows.
1. Switch on all cuircut breakers inside of machine cabinet.
2. Switch on the machines main power disconnect (cuircut breaker) on the back side of machine.
3. Switch on the machines electronic control which powers on the computer electronics boards and monitor.
4. Operate machine.
5. Switch off electronic control.
6. Switch off main power disconnect.
7. Switch off all cuircut breakers.

CNC machines are blowing electronic cuircut boards. Specificly the majority of components damaged on cuircut boards is capacitors, voltage regulators, diodes, transformers. The problems appear to be occurring during the night when the machines power is disconnected from the power grid.

Facts which point to this are as follows -

- A machine will be running properly at the end of shift when it is turned off and the main disconnect on the machine is switched off at the end of the day. The next day when the operator switches on the main disconnect and turns machine control on the machine will turn on in an alarm state and will have a cuircut board damaged. This is the way that 99% of our break downs occur.

- Many times a machine will have numerous cuircut breakers inside of the machine tripped off when the operator powers the machine on in the morning. Typically there is no damage to a cuircut board and powering the machine off then switching on the breakers is all that is needed. Other times a cuircut board will have been damaged.
______________________________________________________

- On 8/7/14 before any machinery had been turned on in the plant i personally inspected every CNC machine in the plant and found 4 machines which had at least one cuircut breaker inside the machines switched to the off/tripped position (these breakers are either on or off the tripped position is off technically).

Out of the 4 machines with breakers switched off 3 of them had been running the day before when they were powered off. All 4 machines powered up and ran fine after reseting the breakers.

- After the incident on 8/7 i instructed all employees to switch off all breakers inside of machines after they had switched the machines main power off at the end of the day. Since that time (two months now) we have not found a single machine breaker tripped off after powering off a machine. Neither have we found a single breaker tripped off after the operator switches breakers on and main power has been switched on in the morning. However we are still experiencing cuircut boards being damaged.

- IN MY OPINION THIS IS A SIGN THAT THE BREAKERS ARE NOT BEING TRIPPED OFF WHILE THE MAIN POWER IS SWITCHED ON OR OFF BUT RATHER THE BREAKERS WERE BEING TRIPPED AT NIGHT WITH NO CONNECTION TO THE MAIN POWER GRID. I understand this is almost impossible but this is what is happening and the facts dont lie.
_____________________________________________

Have installed surge protectors on most machines. Saw no difference in problems.
Have installed electric filters on most machines. Saw no difference.
Have switched from 240V high leg delta to 208V WYE power. Saw no difference.
Have installed ground rods for machines and later disconnected ground rods. Saw no difference.

Another bit of information that may or may not be connected to the problem is that we are located on property where an old foundry was and the ground here is terribly poluted with tons of metal in the ground. And also from what i have been told there are numerous acid pits here that were simply covered over with soil when the foundry closed a long time ago. One of our CNC repair companys recently told me that we are sitting on top of a battery. That may be a little dramatic and may or may not be the case but i suppose it could possibly have something to do with our trouble.

I am currently looking into moving our business but that is a huge cost and i would certainly like to stay where i am if its at all possible. This problem is so weird i do not know who to call on to diagnose it. Most electric people simply do not believe what i am telling them.

Anybody know of a lifeline i can call.

 

[cuba_pete]

After the incident on 8/7 i instructed all employees to switch off all breakers inside of machines after they had switched the machines main power off at the end of the day. Since that time (two months now) we have not found a single machine breaker tripped off after powering off a machine. Neither have we found a single breaker tripped off after the operator switches breakers on and main power has been switched on in the morning. However we are still experiencing cuircut boards being damaged.

This leaves me scratching my head a little. How could you find the breakers tripped if they were physically switched off the night before (tripped and off being the same thing)? Something in your timeline is out of order.

Before going much further, a complete one-line or even full-phase diagram would be immensely helpful.

What circuit boards are being affected? A machine schematic would be helpful.

What type of surge protectors are in use now, and where are they installed in the one-line?

Describe these "electric filters", and on whose recommendation were those installed?

On whose recommendation did you change the service from 240 to 208? What are the input requirements of the mills? Equipment manufacturer plate data is helpful here.

How were the "auxiliary" ground rods utilized (diagram, again)? On whose recommendation (with reasoning) was grounding methodology changed?

How long has this equipment been installed? Has it always done this since commissioning?

 

[cuba_pete]

Review...

Review...

I have reviewed the other thread. I would still need to see some as-builts for the facility to assess the actual one-line installation to determine requirements.

As far as the CNC machines and any other equipment which is also on the same branches, if you could provide a link to the installation manual or scan and post it that would be immensely helpful. The installation manual should contain a schematic of the machine provided by the manufacturer in order to be helpful.

Reading through threads that provide information piece by piece and trying to reverse-engineer a complete installation schematic is tiresome. I only do that as a last resort even for my job.

 

[Timbert]

Anybody know of a lifeline i can call.

Who you gonna call?

Ghostbusters!

 

[Jraef]

In the other thread, iceworm asked you if there were shunt trips on the breakers, I didn't see an answer. But out of curiosity, did any of these machines come to you as auction buys from semiconductor tooling companies that closed down in the 2008 crash? The reason I ask is that at that time, there were rules specifict to that industry dictating that all machinery needed to have an EPO (Emergency Power Off) system, which bypassed all other controls and orderly shutdown systems in the machine, no matter what. This was frequently over interpreted to include CNC machines in the shop, it was only supposed to apply to the semiconductor processing machines themselves. I would often find that these over-interpreters would have a lot of trouble figuring out how to implement that in a complex CNC machine, so rather that try to figure it all out, they would install Shunt Trips or Under Voltage Trips on every single breaker, then wire those to a single circuit that had a big red mushroom button on it, labeled EPO (or ETO, Emergency Turn Off). Absolutely assinine of you ask me, but I saw it many times. When those companies failed or the production moved to Asia, those machine tools were a dime a dozen for a while and a lot of dealers around the country gobbled them up, cleaned them and resold them. Sometimes they had no clue that the EPO system was installed, it was not on the original drawings, or just removed the big red button without thoroughly understanding why it was there or how it was implemented (again, no drawings). So if for instance they had used UV trips, every time you kill the main, the breakers with UV trips will trip, or if they are STs, under certain circumstances a surge will bleed into the old EPO circuit (often 24VDC) and energize the STs. Over the years some of the breakers may have been changed out, which might explain the appearingly random nature of it.

The point behind this wild ass theory is this. NOTHING can make a breaker "trip" off that is not related to current flowing THROUGH IT, or an external mechanical trip coil, such as a ST or UVT. Nothing. So before you go off believing in ghosts or giant underground chemical battery theories, eliminate the only REAL possibilities here.

 

[Barbqranch]

Do you know from personal observation that when the main breaker is shut off there is no power anywhere in the machine? Is the ground connection good?

 

[cuba_pete]

Good points

Good points

The point behind this wild ass theory is this. NOTHING can make a breaker "trip" off that is not related to current flowing THROUGH IT, or an external mechanical trip coil, such as a ST or UVT. Nothing. So before you go off believing in ghosts or giant underground chemical battery theories, eliminate the only REAL possibilities here.

A good point from a solid member of CR4.

GA

 

[GoldDigger]

If you have local ground electrodes that are not, at a minimum, tied solidly to your EGC that could cause problems.
IMO you can verify the integrity of the EGC on selected machines as well as its isolation from the neutral at the machine end, then deliberately disconnect any local ground rods at those machines and see how that group fares over a couple of months.
What you absolutely must avoid is connecting a local rod to a different place on or in the machine from the place where the EGC of the supply circuit lands.
Are there any sensors or control wires that run from one machine to another?

 

[USMC1302]

I'm not sure it would make any difference, but the order in which you switch breakers on/off doesn't make sense to me. If there are individual breakers inside the machine, I would think you would switch them off before switching the "main" on the back of the machine, and I would turn on the "main" before the individual breakers in the machine to start-up?

 

[Aleman]

I read part of the original thread but not all. I did see where someone suggested having someone take a very good look at your power, with analyzers, data loggers etc.
That's a good suggestion.

I personally don't think turning machines off and on constantly is a good thing. Certainly shouldn't need to turn off all breakers in a panel as part of the power off
procedure. And it doesn't sound as if it has helped anyway. Speaking from experience machines like to stay on and in fact you have demonstrated the truth of that.
Power consumption shouldn't be too bad with motors not spinning. If something goes wrong and shorts out during the night, that's what breakers are for.

As for the breakers that trip without power applied...like Jraef points out, only an electrical current can do that. So if current is tripping breakers off then a datalogger should
catch it. Maybe install on one machine and keep it on there until you have your problem. Then see what the datalogger says. Attach it to the switched side of the
machine disconnect.

Put a camera somewhere that no one knows about and keep an eye. Because what you are telling us seems pretty much impossible. And you never know. It's a interesting problem
you have and you're lucky to have a in house guy to fix all of this stuff. Hard to guess more than this without looking at drawings etc. What exactly is blowing up? Low voltage control
cards, drives etc? Is it always different or the same cards that blow? Same components?

 

[GITRDUN]

I have contacted an electrical engineer from this forum to retain his companys services for flying down here and trying to diagnose our problem. Nice enough fellow who seems to have the credentials, equipment and knowledge.

I will certainly post the results if the culprit is found.

 

[Cheyenps]

I don't have an answer but I do have a story.

We had a customer who made compact discs and they had a problem similar to yours but it only happened every 90 days or so. Everything would be dandy, then the lights would flicker quickly and about a dozen machines of various sorts would quit running due to burned up circuit boards.

We checked everything and it all looked fine so I suggested that the customer contact the POWCO and an independent engineering firm to put logging instruments in place in case it happened again. They did that - and it was an impressive array. When it happened the next time everyone gathered around to see what happened but no one was really sure how to explain it. Logging instruments didn't say much really, and no one had any idea what to do.

The next time it happened I happened to be there and the result was similar to the time before - no one could figure anything out. Perhaps it would have gone on forever if their custodian hadn't gone outside at just the right moment. He came running in to the shop and had us all go out and look at something. Sure enough, there was smoke coming from a pine tree in the front yard.

Here's what happened. The building had a 16K Amp 277/480 service fed by it's own little exterior substation surrounded by a fence. That substation was fed overhead from the street at something like 12KV. The pine tree would grow tall enough over the course of some three months to short those overhead wires and that is what caused the problem. The tree was chopped down (along with every other tree on the property - the plant manager was taking no chances) and they ran for some 10 more years without incident. The custodian got a nice little bonus out of it, too.

I can't explain the mechanism by which the shorted lines (and presumably 12KV conducted in to the soil via the tree) caused circuit boards to fail but I'll bet someone here can. The only thing I know for sure is that when the trees went away so did the problem.

Maybe this will help. Good luck!

 

[GoldDigger]

My first guess would be that the machines were connected to individual local ground rods for "protection" and the line fault caused enough of a gradient in the soil that the machine grounds were actually introducing voltage relative to the main ground and EGC connection point.
But we will never know....

My second guess would be that the "flickering" in the main power just put more transients into the machine power supply than it could handle, possibly overvoltage, possibly just chatter on and off.

 

[John120/240]

I have contacted an electrical engineer from this forum to retain his companys services for flying down here and trying to diagnose our problem. Nice enough fellow who seems to have the credentials, equipment and knowledge.

I will certainly post the results if the culprit is found.[/QUOTE

What was the outcome of his investigation ?

 

[mike_kilroy]

I have contacted an electrical engineer from this forum to retain his companys services for flying down here and trying to diagnose our problem. Nice enough fellow who seems to have the credentials, equipment and knowledge.

I will certainly post the results if the culprit is found.

If still unexplained after this, my buddy Tom Henning near Wellington is real good at tracking this kind of thing down - you know him?....

 

[hurk27]

In the other thread you mentioned that you have a 480/277 volt WYE service to a 480/208-120 volt WYE, you just described this transformer as a WYE but didn't indicate if the primary is also a WYE? if the primary of this transformer is also a WYE is the H0 (center WYE tap) grounded to the 480/277 supply? if it is then this could produce unwanted currents and voltages in the grounding and can also cause over heating of the EGC as well as the transformer running hotter then normal, anytime you have a WYE service feeding a common core WYE step down or up transformer you never connect the center tap of the primary on the second transformer, because any unbalanced loads on the secondary of this second transformer will produces varying voltages and currents into the grounding system that could cause problems.

To add to the above you also stated that the 480 volt service from the utility transformer secondary does not have a grounded conductor coming from it to the service at the building? you need to have someone check to see if the secondary of the utility transformers is wired in a WYE and it's center tap is grounded, or if it is a delta one of the phases has been bonded to the MGN (multiple grounding neutral), if it is primary conductors coming to the building then as long as they are up on a pole its not a problem but if they are to a pad mounted transformer then even this can be a safety hazard in the event of a fault.

Having the utility transformer referenced to ground at the pole and not having a grounded conductor ran to the service disconnect sets up a very dangerous condition that not only can cause stray voltages in the grounding systems it also cause a very dangerous shock hazard if at any time one of the phase conductors faults to ground, this is because there is no fault current path back to the utility transformer except through Earth and the Earth path is not a low enough impedance to cause an OCPD to open, this fault will cause all the grounding in the building to be at the phase to ground bond at the utility transformers voltage to Earth, this could be very dangerous.

Is the transformer listed as the first item in you one line drawing in post 11 in the other thread the utility transformer or is it just another transformer after the utility's transformer, or is it fed just from the utility's primary but the service point is ahead of it? or the service meter ahead of it?
Again if this is your transformer then is the primary H0 bonded to the buildings grounding? again this would cause stray voltages in the grounding of the building just like that on the second step-down 480/208-120 transformer.


You stated that at each CNC mill you have a ground rod that you no longer have connected to anything, if you have a voltage recorder or even a Fluke DVM with a peek capture feature place it between one of these ground rods and the grounding of the CNC mill, even if it is just to the metal cabinet and see if it captures any voltages, the only problem is a hand held Fluke will shut off in a few minutes so if you have a bench fluke with capture would be a better choice, but if anything try to get a couple days of logging out of it if not more, you might be able to rent a voltage logger to run this test, it would be nice if you can have it running when an event happens but we all know Murphy's law will not let that happen:rant:

It might be possible that you could have a intermittent voltage on the grounding system that somehow is back feeding in to the control circuits on the mills, it could be coming from another path that may be at Earth potential, could be an air line or network cable or any path that is at Earth potential, and after reading where Gar stated in post 73 that the limit switch's are tied to the CNC's PE or circuit grounding which could be one such internal path that this could happen and would explain why the resistor burned up???

Yep I know that is a long shot but without being there and making my test it's about the only thing I can come up with, even if that resistor was hit with 75 volts it could be enough to burn out that carbon resistor which is .56 watts@10k ohms it just would take a little longer which it would depend upon how badly it failed, did it just show signs of over heating or did it blow apart (meaning it was hit with a much higher voltage) all these failures are part of a big puzzle and looking at each one give us a small picture of the main problem, and you don't have a 25 piece puzzle here, you have a 500 piece puzzle that we have some of the pieces missing, this is why logging each failure is very important, it should list each component that fails each breaker that tripped, and where in the circuit this component is, then you have a analyze what sequence of events took place, and when it happened, we know if all the power is off that you would not have any current flowing to cause an event, but guess what does not get shut off? Grounding!!, back up batteries, network connections, air line connections, or other conductive pathways, so we must analyze each one to see if one or more could be causing a current to flow when the main connection to power is shut off, another thing I noticed is that this problem seems to never happen while the machine is in operation, so we must look at what changes electrically when we shut it down, this includes all the afore mentioned conductive pathways.

Another question, since you don't have a grounded service from the utility is there a ground fault detector installed? it would at least give you a warning when a phase faults to ground that doesn't open a OCPD, of course this is only needed at the first 480 disconnect or transformer which ever is the first item to the utility after you derived a neutral and grounding after a transformer it's no longer needed as now you have a fault path to open the OCPD's on a fault.

 

[p real]

I find it interesting that it always happens at night. And definitely seems like a spike. Has anyone looked into the possibility that somehow the buildings exterior lighting is feeding into the machines? An easy way to check that during work hours would be to manually trip the timeclock that controls the
buildings outside lights, or cover up the photoeye if you have one, or possibly both control mechanisms exist and will have to be overridden to turn the lights on during the day. See if the problems happen at that moment the lights come on or soon after. Just a thought... the only variable you have described that has exhibited a pattern that has been constant is the timing of it. Something that bumps in the night... i would start my troubleshooting there...

 

[p real]

I also would agree with others in that figuring out WHEN these problems started exactly is a crucial question. Has it always been like this to some degree? You stated in the other thread that you didnt remember the problems being this bad. So when did they get this bad? Often there is some precursor, seemingly innocuous even, that is the root of these problems.When things are done to a building that are not seemingly related to whatever is currently broken, a big piece of the troubleshooting pie can be overlooked. For all you know someone knicked an underground wire while driving all those groundrods! Not likely, but surely you get the drift. Try and come up with a more solid timeline of when the problem started and if anything else you can think of changed in the building, even simple things. Sometimes the most obvious is the least obvious. Hope u figure it out.

 

[GoldDigger]

Even without nicking something with a ground rod, if there are at any time for any reason large earth currents under the building, then installing local ground rods that are not solidly bonded into the GES may actually BE the cause of the problem getting worse.

 

[Cheyenps]

I have seen where various actions on the part of the power company can cause the sort of problem you describe.

Can you isolate the exact time? Have you discussed your problem with the power company?