Best Practices for Troubleshooting Intermittent PLC I/O Issues

[lostintheether]

Hi all,

I’m looking for some advice on troubleshooting intermittent I/O issues on PLC-controlled equipment. We have a production line where a few input signals randomly drop out for a split second, causing nuisance shutdowns that are hard to reproduce.

So far, we’ve checked wiring and terminal connections, replaced a few suspect sensors, and even swapped input cards, but the issue still pops up occasionally.

I’d like to know:
Any tips for logging or isolating signal drops in real time?

I’d really appreciate any insights or examples from similar experiences. These random faults can be such time-eaters!

Thanks for your help in advance.

 

[ptonsparky]

I've had an 'alarm' set to log the time and which input failed. It would indicate what step in the process failed. PITA for even the small projects I had and Mine were all small and relatively low speed.

 

[ModbusMan]

Are these inputs dry contact, voltage, current loop, something else...?

 

[jim dungar]

All of the I/O cards in a rack all all multiplexed into some interface device within the communicates with the CPU, often remotely.
While it is extremely rare, these interface modules and rack assemblies have been known to 'not multiplex correctly'. When the multiplexing fails the CPU reads data from the wrong I/O even though the I/O modules lights are operating correctly.

I found one of these issues when the process was shutting down on a stop command even though the stop button was physically jumpered out. The screen update was to slow to see a contact blip so to confirm the I/O was correct we started checking each input as if this was a startup. We replaced the rack and every thing went back to normal. I had never seen a rack fail like this before.

 

[4-20mA]

Trends... Check your trends, look for correlations, something might line up... I don't hate the alarm idea but that can all be found via trends as well.

Intermittent issues are the worse. Good luck, let us know what is found.

 

[ModbusMan]

Trends... Check your trends, look for correlations, something might line up...

And look for things that aren't immediately obvious... perfect example:
Years ago (not long after being laid off from my first automation-type job), I was interviewing with a company that worked in machine vision based quality control. Think "snap a picture of the widget as it moves down the conveyor, compare against a template, kick out any with a deviation >some percentage." A technician of theirs had been sent out to diagnose a line that was rejecting samples at a far greater rate than anybody could explain. After several days and a couple of camera replacements, it was discovered that the strobe light on a neighboring piece of gear was blinding the camera, causing their algo to reject a perfectly compliant part. Solution was a piece of cheap plastic corrugated board from the office supply store to block the light pollution.

 

[steve66]

Are you sure it's an I/O error and not a bug in the program?

Otherwise, if you have the problem narrowed down to a couple of sensors, maybe consider just replacing the wiring.

If it's a major problem, maybe even consider 2 redundant sensors checking the same thing, wired back to separate inputs, and have the program checking to see if and when they differ.

 

[petersonra]

My experience is that the vast majority of the time these kind of glitches are program errors.

I actually did run across one where I jumpered two inputs together and watched to see if they were ever not the same on the same card. The program eventually showed that one of them was on and one was off even though they were jumped together right at the card. I ended up calling tech support and the guy told me he remembered something about dried up capacitors. And he sent me a fax that had a range of serial numbers that were of this particular input card that were affected. And sure enough this particular module was in that serial number range. A new card solved the problem.

I've also run across loose wires or once in awhile push buttons with contact blocks that are failing.

Intermittent problems are insanely difficult sometimes to diagnose and often I end up swapping out boards or just tightening up connections to make sure that the obvious things are handled. Many times the problem then goes away. It's satisfying that it's fixed but completely unsatisfying that I was not able to find the specific problem because it was more cost effective and much faster to just do a whole bunch of stuff that I know from experience is likely to fix the problem but leaves me without knowing exactly what went wrong.

Last year I ran across a terminal block with a single terminal that had an intermittent failure in the terminal block itself. I'm not sure what was wrong with it but it would work for a while and then quit. I had them replace a terminal which looked to me like somebody had over torqued it and the problem went away and did not return.

 

[ModbusMan]

Last year I ran across a terminal block with a single terminal that had an intermittent failure in the terminal block itself. I'm not sure what was wrong with it but it would work for a while and then quit. I had them replace a terminal which looked to me like somebody had over torqued it and the problem went away and did not return.

Sounds like you might have found a cracked cold solder joint.