Mysterious vibration in long conduit between solar inverters and main service panel

[ggunn]

Also note that acoustic tubes can create resonances. A tight wire might be a useful sound transmitter. Mechanics use screwdrivers to couple engine or other noises to their ear.

My mechanic friend gave me a stethoscope that has a metal rod in place of the disc that's on a medical one. I used it to find a clattering harmonic balancer on my car.

 

[FionaZuppa]

My mechanic friend gave me a stethoscope that has a metal rod in place of the disc that's on a medical one. I used it to find a clattering harmonic balancer on my car.

a cheap piezo mic (batt operated with pre-amp, etc) makes for a great contact tool for "listening" in to "see" the vibrations. can be had from ebay for like $2

60Hz does "beat" as 120x per sec. it's likely resonance.

 

[Redwood Infrared]

...did a test ....QUOTE]

Think we all are breathlessly awaiting anyof that test data to be shown. Sure would be helpful to share an audio waveform, voltage and current waveforms.

Need to have something factual on which to base conjectures vs. just that it drives folks out of the office.

All we have so far factual is that something makes noise and it is related to the level of sunshine and the inverter model number.

Would be great if you could get a release to share the THD, voltage, and current measurements taken by the poco.

I'm working on that but don't hold your breath. With so many stakeholders I have to be careful not to stomp on any toes. I might be able to get away with a couple of screenshots but really want to get it cleared first.

 

[Redwood Infrared]

Great discussion. I would like to know how this all plays out in the end.
One thing I have learned thru the years (48) in T-shooting problems is that my mind tends to think of the most complicated reasons for the cause of the failure. When all is said and done it usually turns out to be something simple. I would like to know if this is a normal reaction and have others felt the same way.
But I will not rule out the times when I have pulled my hair out on some weird problems....ha!

I'm still holding out hope that in the end simple thermography will have saved the day. For 2 years nobody thought to look for a heat signature until I showed up and within 10 seconds I was able to point out what didn't look right. This is just one of those cases where due to the vibration I don't have any idea whether I'm looking at a symptom or a root cause. It's entirely conceivable that a minor connection issue led to a voltage or current unbalance that set up an initial vibration and we got where we are through a positive feedback loop. We should have a good idea if that's the case once the electrician redresses terminal ends of the conductors and installs some hardware to improve all those connections. If we remove the heat but still have vibration, or if even after replacing switchgear those connections are still getting hot, well then we'll know there's something fundamentally wrong with the circuit that's inducing these crazy fields. But I'll look like a flippin' hero if it turns out it was just bad connections getting hot.

 

[GoldDigger]

Bad connections getting hot mean higher than normal voltage drop, which in turn means unbalanced current in parallel runs. And that can cause vibration, so your odds are pretty good.

 

[Ingenieur]

based on you freq measurements of ~ 60, 120, 240 imo not mechanical, ie, motor or compressor
that would be much higher

get some eh rated gloves
when humming grab and hold at noisest point
both hands or 1 hand same conduit
preferably with eh shoes or insulated ladder
see if it dampens

 

[Redwood Infrared]

based on you freq measurements of ~ 60, 120, 240 imo not mechanical, ie, motor or compressor
that would be much higher

get some eh rated gloves
when humming grab and hold at noisest point
both hands or 1 hand same conduit
preferably with eh shoes or insulated ladder
see if it dampens

We've already tried that. So far no efforts to dampen it have been very effective.

 

[Redwood Infrared]

Bad connections getting hot mean higher than normal voltage drop, which in turn means unbalanced current in parallel runs. And that can cause vibration, so your odds are pretty good.

Normally I'd absolutely agree, just can't understand why we're not seeing significant voltage drop anywhere.

Although I have to admit we didn't have good lighting on the inverter combiner breaker box and today I got a much closer look at it and sure enough, there is visible evidence of minor arcing on the tabs where the breakers are bolted to the bus bars. I couldn't measure any voltage drop under 70% conditions with things running about 50°F rise over ambient but perhaps there's a point when the system hits peak output that it gets hot enough the resistance increases exponentially, and so far that's only been transient.

 

[Ingenieur]

try this when at full output
shutdown
disconect 1/2 the capacity, inverters, etc
disconnect one set of the conductors
turn the remaining 50% on, see if it hums

 

[GoldDigger]

Normally I'd absolutely agree, just can't understand why we're not seeing significant voltage drop anywhere.

Although I have to admit we didn't have good lighting on the inverter combiner breaker box and today I got a much closer look at it and sure enough, there is visible evidence of minor arcing on the tabs where the breakers are bolted to the bus bars. I couldn't measure any voltage drop under 70% conditions with things running about 50°F rise over ambient but perhaps there's a point when the system hits peak output that it gets hot enough the resistance increases exponentially, and so far that's only been transient.

The reason that you are not seeing significant voltage drop anywhere could well be that when you have two low resistance paths in parallel it does not take very much of an added voltage drop in one of the two paths to produce a very unequal division of current between the two paths.

Example:

If you have two 119.9 ohm loads fed in parallel from a 120V source and the current split between two parallel wires each with a resistance of .2 ohms, the total circuit resistance will be 120 ohms, the current will be 1A and that 1A will be divided equally between the two wires.
If instead one path (A) has a wire resistance of .2 ohm and the other (B) has a resistance of .3 ohm the total current will still be about 1A but the current will be divided 600ma on one wire (A) and 400ma on the other (B). Note that the voltage drop measured across each of the two paths will still be identical because they are in parallel and the currents will not be equal.
The overall heating in the B path because of the extra .1 ohm resistance will in fact be lower than in the A path because of the unequal current division. If the added .1 ohm is in the form of a bad connection, you will see a voltage drop of .04 volts across that bad connection and the local heating will be only 0.016W.

Run the calculation with more reasonable numbers to match your situation and you will see similar results. A very small added voltage drop can produce wildly different currents between the two paths.

 

[Redwood Infrared]

try this when at full output
shutdown
disconect 1/2 the capacity, inverters, etc
disconnect one set of the conductors
turn the remaining 50% on, see if it hums

Ooh hey, I like that one...will add it to my list for next week just before and right after the electrician starts renewing connections.

 

[Redwood Infrared]

The reason that you are not seeing significant voltage drop anywhere could well be that when you have two low resistance paths in parallel it does not take very much of an added voltage drop in one of the two paths to produce a very unequal division of current between the two paths.

Example:

If you have two 119.9 ohm loads fed in parallel from a 120V source and the current split between two parallel wires each with a resistance of .2 ohms, the total circuit resistance will be 120 ohms, the current will be 1A and that 1A will be divided equally between the two wires.
If instead one path (A) has a wire resistance of .2 ohm and the other (B) has a resistance of .3 ohm the total current will still be about 1A but the current will be divided 600ma on one wire (A) and 400ma on the other (B). Note that the voltage drop measured across each of the two paths will still be identical because they are in parallel and the currents will not be equal.
The overall heating in the B path because of the extra .1 ohm resistance will in fact be lower than in the A path because of the unequal current division. If the added .1 ohm is in the form of a bad connection, you will see a voltage drop of .04 volts across that bad connection and the local heating will be only 0.016W.

Run the calculation with more reasonable numbers to match your situation and you will see similar results. A very small added voltage drop can produce wildly different currents between the two paths.

I see what you're saying here but the most significant heating I'm seeing is either before the current paths are split (at the breakers in the inverter combiner box) or after they come back together (at the knife contactor hinges in the AC Solar Disconnect). There's some heating in the inverter combiner box on the main disconnect switch where the split paths originate but it appears sympathetic since that switch shares the bus with those breakers.

That said, your proposed exercise is reasonable and I will try incorporate it into our analysis.

 

[Redwood Infrared]

...did a test ....QUOTE]

Think we all are breathlessly awaiting anyof that test data to be shown. Sure would be helpful to share an audio waveform, voltage and current waveforms.

Need to have something factual on which to base conjectures vs. just that it drives folks out of the office.

All we have so far factual is that something makes noise and it is related to the level of sunshine and the inverter model number.

Would be great if you could get a release to share the THD, voltage, and current measurements taken by the poco.

I got approval from both the client and the utility to release the data in question. The PDF they supplied is way too big to attach here so I'm going to have to edit down to a few relevant images and have to run right now but will try to get that posted up here this evening. I also got the raw data but have to install and learn to navigate two different PQM software platforms so don't know when I'll get that done but I'm working on it.

Have I thanked everybody here for so much attention to my questions lately?

 

[Redwood Infrared]

Here they are, the THD, Voltage, and Current data from utility PQM

Here they are, the THD, Voltage, and Current data from utility PQM

...did a test ....QUOTE]

Think we all are breathlessly awaiting anyof that test data to be shown. Sure would be helpful to share an audio waveform, voltage and current waveforms.

Need to have something factual on which to base conjectures vs. just that it drives folks out of the office.

All we have so far factual is that something makes noise and it is related to the level of sunshine and the inverter model number.

Would be great if you could get a release to share the THD, voltage, and current measurements taken by the poco.

I've got all these in a PDF but it's way too big for the forum's upload limits so had to capture JPGs. I've got the raw date too but it's in the Fluke Power Analyze format and even bigger files so these will have to do.

I also have these for two other panels, the AC Solar Disconnect and the building's MSP. Unfortunately they used a different PQ monitor for the ACSD so it's not the apples to apples comparison I'd like to have gotten. They used the same Fluke in the MSP but of course there are other building loads so that's not a very good comparison either.

I should add that the figure showing current on the ground conductors here appears to show dead zero, which was not consistent with my experience. We measured up to a total of 10A across the combined parallel grounds when the array was peaking, and there was always at least a couple amps there if the array was producing at all. Again though, just using my Fluke T5, not RMS, and frankly I could barely squeeze the fork over wire that fat, couldn't put it on the phase conductors at all.

 

[Redwood Infrared]

I am curious about the high MSP %THD though

I am curious about the high MSP %THD though

%THD for voltage and current at both the inverters and the ACSD is within normal range, under 5%. I'm not sure why it's so high for the MSP - The building doesn't have any big motors spinning unloaded so not sure what would cause really poor power factor. Is this unusual? Is it a function of loads or the grid? The PV is big enough they're a net producer by a wide margin under these conditions so don't understand what's going on in this chart.

 

[gar]

180616-2433 EDT

Redwood Infrared:

I think your troubleshooting procedure is unnecessarily complex.

My guess is that many things you are spending time or thought on are not likely to have anything to do with the problem.

One time I had some quality control people that came to me and said they were getting too much pinion position variation from the pinion shim station. They wanted to do some statistical studies to try to isolate the cause. Was it, parts, the pinion shim machine, or something else? This is the long hard way to find the problem if there are other approaches.

I went over to the machine and ran some simple tests. Within less than 15 minutes, including walking time, I told them there was a bad thrust bearing in the pinion shim station. Their experiments would have taken days, product yield would have been bad all that time, and their approach may not have foumd the problem.

You have an acoustic noise problem. You need to know what that signal looks like, and have measured values from it. You need to know how this signal varies over a spatial area. You need to know what items are vibrating and possibly amplifying the sound level. With this information you can better focus your attention on the likely cause.

Your waveforms are of little value at this time. Your neutral current is dominantly a single phase 60 Hz sine wave of low magnitude. Probably unimportant.

You might do some bench tests with single and three phase wires loose in a conduit and see if you can create acoustic noise.


Your last post implies quite different plant loads on each phase. Each individual phase looks repetitive, but each is different from the other two phases.

.

 

[junkhound]

Assuming the time scales on the waveforms posted are accurate, the MSP currents are LEADING the inverter current, and have huge distortion.

Are there large rectifier fed capacitive loads in the building, with nearly all of them being single phase loads? That would cause the current waveforms and the high THD.


Would be interesting to see the frequency domain plot of the MSP harmonics if that data is available. Just an 'eyeball' Fourier of the inverter room voltage waveform show lots of 3rd harmonics.

 

[Redwood Infrared]

Gar,

Thanks for your comment. I agree with you this is getting more complex than I ever intended. I didn't see much in these waveforms but posted them because others here asked to see them and since I'm neither an EE or electrician I figured it was worth doing that much.

I do try to stop myself every once in a while when investigation of problems like this keeps growing in complexity. I repeat the questions, what exactly is the problem? What's the root cause? How can I avoid going down so many rabbit holes?

The problem here, at least the manifestation of the problem that bothers the client, is a noisy conduit. I've stated already we're confident now the frequency is 120 Hz and does not vary other than in amplitude, and it's tough to nail the amplitude down because it's constantly changing as solar conditions change. It appears to be loudest not just when PV production is at it's peak but dynamically as PV production is ramping as clouds clear away. It actually appears to modulate down a bit in amplitude once clear skies prevail. But it's always loudest when PV production is over 80% and unfortunately I've not been able to observe those conditions in the two weeks I've been on the job so my ability to gather observations has been limited and I'm relying on the client's commentary. I would like, as it appears you suggest, to determine how the amplitude varies along the length of the conduit but I am not currently tooled up for that and am considering how to become tooled up based on suggestions made to me in this thread so far.

Which is a long way of saying that the *root* cause so far has been a moving target.

Either way, it is very clear that the proximate cause of the noise is obviously strongly vibrating conductors.

We have established that the vibrating conductors clearly have strong magnetic fields being generated between them. This was accomplished by trying out several different phone app Gauss meters, at first one that simply totaled the fields, eventually finding one that measured them in 3 axes.

(These are tools I did not know were so readily available and would never have thought of without being prompted by observations from interested parties on this forum. As I stumble through this part of my investigation I've reported here my less than scientific findings because I'm frankly fascinated by the it all and, evidently, others are as well, as referenced by comments such as "waiting with baited breath" and even one offline conversation where I was told I have a moral obligation for follow through and report out my findings. Again, not an electrician, not an EE, and I’m learning about this stuff by leaps and bounds over the last week. Again, plenty of electricians, the EE of record, and the utility have spent 2 years looking at this problem and in the first 2 minutes I found found hot spots they never knew existed, and have made more progress over the last 2 weeks than all of those people combined. With all due credit to this forum for all the help I’ve received from y’all.)

Those fields - to me anyway - appear to be anomalous. Perhaps I've missed it but I don't believe anybody here as said to me that they are normal and acceptable. They are certainly not desirable in this case. Even if the fields are normal, my long experience in thermography leads me to believe that even if we figured out a way to dampen the sound in the conduit, if we don't make the wires stop vibrating we will very likely still end up with compromised connections at either end of these long cable runs, with unreasonable maintenance costs to prevent component failures if we don't figure out WHY those vibration inducing fields exist and mitigate that condition as well.

It's because I don't know the answer to that I came here to ask people with much more knowledge and experience regarding such things. Along the way a whole lot of questions came back at me and I've been doing my best to respond to those questions, and yes along the way this thread has grown perhaps unnecessarily complex with potentially irrelevant details. I do apologize for that and I've repeatedly expressed my thanks for your continued indulgence of my relative inexperience - I did ask in my opening post that you please bear with me as I'm neither an electrician nor an EE, just a thermographer who normally finds hot spots, points, and says "you might want to fix that." (Although I actually am an environmental engineer normally focused on energy efficiency and industrial reliability. The EE stuff is just a little outside my academic background).

So the one thing I do know is that there are components in this circuit operating out of spec thermally. I have prescribed measures to remediate those conditions and that work is scheduled to be completed this week.

But I've seen similar situations hundreds of times, usually in timber mills where they don't care about the noise. They'll tighten or renew connections, or even replace components in some cases, and the problem comes right back because there's something fundamental in the circuit causing vibration that they still haven't addressed.

This has bothered me for years and I finally have a client who really wants to get to the bottom of it and understand what's going on. Both the client and the installing electrician are very invested in solar development and they want to understand what’s happening as much as I do. Consequently I'm pretty motivated to analyze the heck out of it, accounting for every variable I can identify and scratching them off the list one by one until we can definitely state: THIS is the root cause of this problem. THIS is the solution that will fix it permanently. And THIS is what we can incorporate into our designs and/or maintenance procedures to ensure we don’t encounter it again elsewhere.

So far, there is no one variable that is obviously the answer, and there are a lot of them waiting to be scratched off the list.

Is there a subpanel somewhere in the building I don't know about where somebody bonded neutral to ground? I'm going to look in to that.

Did the medical office tenant who moved in after the array went live install a big THD-inducing UPS in the server room they guard so carefully? We've asked and are awaiting answers.

Does the utility transformer outside the building have a compromised ground or neutral bond? We've asked them to rule that out.

Did the whole thing start because the inverter combiner box breaker panel came with breakers pre-installed and the bolts bonding those breakers to the bus were never torqued in the field by the installing electrician? That question only arose late Friday afternoon because it would never have occurred to me had he not brought it up, prompted by so many prodding questions asked by me. You can bet your sweet bippy we'll be taking a much closer look at those components over the next couple days.

One way or the other, we'll get to the bottom of this. I am hopeful that very soon this ever widening scope is going to become dramatically more narrow until we are laser focused on just one or two things.

Along the way I will continue to express my gratitude and can assure you I'll report out in the briefest and most concise terms possible what I've learned upon this investigation's conclusion.

(TLDR version: Gar, you're right. Point taken. But also, a bunch of other stuff.)

Thanks again,
Redwood Infrared

180616-2433 EDT

Redwood Infrared:

I think your troubleshooting procedure is unnecessarily complex.

My guess is that many things you are spending time or thought on are not likely to have anything to do with the problem.

One time I had some quality control people that came to me and said they were getting too much pinion position variation from the pinion shim station. They wanted to do some statistical studies to try to isolate the cause. Was it, parts, the pinion shim machine, or something else? This is the long hard way to find the problem if there are other approaches.

I went over to the machine and ran some simple tests. Within less than 15 minutes, including walking time, I told them there was a bad thrust bearing in the pinion shim station. Their experiments would have taken days, product yield would have been bad all that time, and their approach may not have foumd the problem.

You have an acoustic noise problem. You need to know what that signal looks like, and have measured values from it. You need to know how this signal varies over a spatial area. You need to know what items are vibrating and possibly amplifying the sound level. With this information you can better focus your attention on the likely cause.

Your waveforms are of little value at this time. Your neutral current is dominantly a single phase 60 Hz sine wave of low magnitude. Probably unimportant.

You might do some bench tests with single and three phase wires loose in a conduit and see if you can create acoustic noise.


Your last post implies quite different plant loads on each phase. Each individual phase looks repetitive, but each is different from the other two phases.

.

 

[ggunn]

Is the source of the noise the conductors vibrating in the conduit or the conduit vibrating in its supports?

 

[Redwood Infrared]

I wasn't present for this PQM exercise last October. I believe the tests were simultaneous but don't know what measures they might have taken to sync the clocks in the two different Fluke 1750s at two different locations 250 feet apart.

But thanks for the comment on the MSP distortion; it seemed huge to me but this is outside my area of expertise so needed to ask.

As far as I know there are no 3-phase loads in the building at this time. Long ago it was originally a light manufacturing facility so as I understand it that's the only reason for the existing 3-phase service. I did an energy audit several years ago before the PV array but the entire building has undergone significant renovation since then so don't know the building inside out. That said, there are no such large loads capacitive that I know of. It's primarily office on one side, with a warehousing/shipping facility on the other. The largest loads of which I'm aware of moderate HVAC motors, all single phase and under a few HP. I believe most or all lighting has been upgraded to LED although I've not had access to the warehouse so not sure about there. All modern computers, not a CRT to be found anywhere. Temperate climate so only a few tons of AC rarely used, a few gas furnaces and a couple of high efficiency heat pumps, all of which were installed prior to the PV array and never caused any issues. It's possible the warehouse has a few small (1hp or smaller) single phase motors driving conveyors that are often moving with little or no load on them.

Is it possible the long parallel conduits with redundant runs of parallel conductors might themselves represent a large capacitive load?

I might be able to provide the frequency domain plot of the MSP harmonics but am not quite sure how. I have the raw PQM data with the Fluke software but am just learning how to use it. Do either of the attached jpgs provide what you're asking for?

Assuming the time scales on the waveforms posted are accurate, the MSP currents are LEADING the inverter current, and have huge distortion.

Are there large rectifier fed capacitive loads in the building, with nearly all of them being single phase loads? That would cause the current waveforms and the high THD.


Would be interesting to see the frequency domain plot of the MSP harmonics if that data is available. Just an 'eyeball' Fourier of the inverter room voltage waveform show lots of 3rd harmonics.