Rounding measurements

[gar]

171119-1114 EST

Suppose I have an application that requires output measurements to an increment of 0.001". This is a shim application where the shims range from 0.020" to 0.050" in 0.001" increments.

The gaging instrument that determines what shim to use resolves 0.0001" and is accurate and linear to 0.0001" .

To get the 0.001" selection, does it matter whether this is done on the raw 0.0001" measurement by rounding up and down to a 0.001" value, or simply truncating the raw measurement (that is just dropping the right digit)?

.

 

[MAC702]

I don't understand the question.

All calculations are done with each figure's maximum accuracy, and your final answer is rounded to the number of significant digits of your least accurate input value.

If that sounds obvious, it's because I'm only stating it because I don't understand the actual question.

In general, don't round off or truncate a raw measurement. Retain its full accuracy throughout the process, especially when it is only 1/10th of the resolution of the target.

If you are asking what nearest shim to use after getting your answer that falls in between shims, it, of course, depends entirely on the application.

I do similar calculations with shims for muzzle devices on rifles. In these cases, you calculate which shim would line up the vertical at the appropriate torque value. For torquing a trial run, using the same thread locker you will use for final installation, you torque to the exact value in the middle of the torque range. You then calculate the nearest shim that will clock your device perfectly. Then, you install permanently, and the torque this time will be the variable, and you stop at perfectly vertical on the device.

 

[gar]

171119-1255 EST

MAC702:

I will try to clarify.

An operator on an assembly line has to pick a shim to insert into an assembly. The shims come in increments of 0.001". So there are shims at nominal values of 0.020, 0.021, 0.022, and so on up to 0.050".

There is a shim rack with an individual tube for each value of shim, and the rack is loaded with shims of the correct value for each storage tube. Each tube has a light, and there is a digital indicator as well that increments by 0.001". The light and the digital indicator both provide the same information of what shim to select based on a measurement of the part to be shimmed. The basic measurement capability of the part has a resolution of 0.0001".

The question is --- does it matter how the 0.0001" resolution measurement is rounded to obtain the shim rack selection? The possible rounding means are "up and down", or just "lop off" the 0.0001" digit (truncate).

.

 

[GoldDigger]

171119-1255 EST

MAC702:

I will try to clarify.

An operator on an assembly line has to pick a shim to insert into an assembly. The shims come in increments of 0.001". So there are shims at nominal values of 0.020, 0.021, 0.022, and so on up to 0.050".

There is a shim rack with an individual tube for each value of shim, and the rack is loaded with shims of the correct value for each storage tube. Each tube has a light, and there is a digital indicator as well that increments by 0.001". The light and the digital indicator both provide the same information of what shim to select based on a measurement of the part to be shimmed. The basic measurement capability of the part has a resolution of 0.0001".

The question is --- does it matter how the 0.0001" resolution measurement is rounded to obtain the shim rack selection? The possible rounding means are "up and down", or just "lop off" the 0.0001" digit (truncate).

.

The simple answer is that it does matter, to the extent that being off by a full shim size matters to the process.
The specification by the design engineer of which shim size is to be used for each measurement will be based on the fact that a shim which is "correct" within one half shim size will cause the product to operate properly.
That process will either be established to choose based on rounding or to choose based on truncation. Whichever is specified during the design is the procedure that must be used.
If being correct plus or minus one full shim size is all that is needed, then it does not matter.

 

[Ingenieur]

in part it depends on the usage
if the assembly heats up/gap shrinks, round down
and the tolerance: loose or tight

say measured is 0.0226
and you round up to 0.023 and use that shim
will it fit?
lop off/round down

better to gauge go/no go
0.022 go
0.023 no go
use 0.022
0.021 go
0.022 no go
use 0.021

 

[gar]

171119-1414 EST

Ingenieur:

The particular application I just described consists of a cast iron housing, two tapered roller bearings, a shaft with a head on it that needs to be positioned within about 0.001" of a perpendicular axis at the rate of one every 15 to 20 seconds. A new bearing and shaft goes into each new part.

The particular bearing to be used is measured in the cast iron part. The shaft is separately measured somewhere else and labeled with an offset correction to be applied in the shim determination. That offset is not part of the rounding operation. It is just something that gets added into the determination of the shim.

A go-nogo method is not applicable.

.

 

[Ingenieur]

what is the offset dimension?

 

[gar]

171119-1541 EST

Ingenieur:

The offset could be up to +/-0.009, typically in the range of +/-0.003, but this has nothing to do with the basic question on rounding. It just happens to be an element in the process.

A couple other factors in the process that have nothing to do with the rounding are:

The bearing has an applied axial load of about 1500 # when being gaged, and is expected to have a load near this when assembled.

When assembled the shim will be under the resulting load on the bearing. The shim usually goes between the shaft head and the bearing cone.

If the shim goes between the bearing cup and the cast iron housing, then the bearing has to be separately gaged and combined with a measurement on the cast iron housing. In this case there is an assembly force of about 15,000 # on the shim from pressing the cup into the housing. Over time all of that pressing force will not remain on the shim. The shim has a fairly large cross-sectional area and does not deform much. The bearing has a much softer spring rate, about 0.003" for 1500 #.

.

 

[gar]

171119-1649 EST

GoldDigger:

I will respond to your response later.

.

 

[Ingenieur]

offset matters
you could eliminate it by rounding up

shouldn't offset always be positive? if it is for expansion/control friction losses
like 0.005 +/- 0.001

 

[gar]

171119-2100 EST

Ingenieur:

I believe that I have stated the offset that has been mentioned so far has nothing to do with the rounding operation.

The so far said offset only enters into the application because of manufacturing variations in the thing I have called the shaft. I should have never mentioned it.

I suspect there may be 50 or more different machines producing the 10,000 or so shafts per day. Thus, you can expect moderate variation, a few thousandths of an inch, between different parts.

The said offset has nothing to do with friction.

Somehow you have not understood what I said in my first post. That question relates to how one rounds a measurement that has more resolution than required of the output function.

.

 

[gar]

171119-2320 EST

This is largely a math and/or statistics problem rather than related to a specific application.

However, it is a real world problem, and not just hypothetical.

.

 

[Ingenieur]

if nothing matters (free play, bearing play, etc.) then round up or down, or lop it off, or always round down, or up, it does not make a difference
not sure what or why you are asking?
you made it much more complicated than need be, since nothing matters and any method selected will have no impact on the assembled system

you just need to decide how to round
whether 0.0005 goes up or down
but it does not matter since no other dimensions need maintained nor are affected

 

[MAC702]

...That question relates to how one rounds a measurement that has more resolution than required of the output function.

...

This is largely a math and/or statistics problem rather than related to a specific application.

However, it is a real world problem, and not just hypothetical.

It seems very application-specific. Some applications will only care to be as close as possible, in which case rounding is appropriate. If it needed to be closer, they would make thinner shims. Just round normally and install.

In many of those applications it probably doesn't matter which way you round, but you may as well follow convention.

Some applications might have a tolerance only on one side of perfect. It may be they prefer to truncate a value to be as close as possible but with relatively no tolerance for going over. For example, when putting a wheel on a bearing with a castle nut. Find perfect, then back off (truncate) to the next looser notch on the castle nut.

 

[gar]

171120-1030 EST

The answer to my question in post #1 is that there is no difference other than translation from one of the two specified methods to the other is by a constant shift of the source data by 1/2 of the quantizing level.

Back in the days when I first built gaging equipment I used truncation because it was electronically simple, just drop the unwanted least significant digit. I found it virtually impossible to convince process engineers, electricians, and others that the rounding methods were equivalent. However, there was one person that immediately understood the equivalence of the two methods and he was head of quality control at a Chrysler plant where I had equipment.

Following is an illustration:

Assume we have a system that quantizes to a 0.1 level for measurement, and we want to round to quantize at the 1. level.

Consider a set of measurements and identify each individual measurement with a unique letter.

Identifier Measurement

A 10.0
B 10.4
C 10.5
D 10.9
E 11.0
F 11.4
G 11.5
H 11.9
I 12.0
J 12.4

Round up and down to the value of 11. The members of this set are
C, D, E, F

Just truncate and the members are
E, F, G, H

Add 0.5 to each measurement and the truncated members are
C, D, E, F
Exactly the same result as rounding up and down.

This 0.5 shift is simply a matter of how the equipment is calibrated.

I think GoldDigger's comment implied he understood this concept.

.

 

[retirede]

MS Visual Basic did not have a true rounding function. It was SOP to add 5 to the digit to be rounded and then truncate.

 

[wwhitney]

To get the 0.001" selection, does it matter whether this is done on the raw 0.0001" measurement by rounding up and down to a 0.001" value, or simply truncating the raw measurement (that is just dropping the right digit)?

I would say the answer depends on whether the shim selected should be the exact size +/- 0.0005", or whether you'd prefer that it be the exact size +0"/-0.001" or perhaps +0.001"/-0". Different applications will have different preferences among the three options given, I didn't follow the details of the particular application in your example.

Cheers, Wayne

 

[gar]

171120-2430 EST

wwhitney:

My original post question only has to do with the question of whether rounding up and down vs truncating can produce the same result in a measurement process. The answer is yes.

Somewhere independent of the measurement process is where the decision about what the quantizing level should be for the shims. Once that decision is made by product engineering and the bean counters, then manufacturing equipment and gaging equipment has to be designed to work with the specifications.

In the product I was using as background there are two or three different shimming areas. The most used quantizing levels are 0.001" or 0.002", or something close in metric
(0.02 or 0.04 mm).

Not relevant to the rounding question is the fact that the shims in the shim racks are not of the precise value assumed. Generally in checking the shims we used an acceptance band of +/-0.0007" . Thus, the actual shimmed result is worse by the combination of rounding and the shim variation.

One manufacturer we supplied assumed their production workers were smarter. To these operators we supplied the call measurement to a resolution of 0.0001". Then that operator selected a shim from the tube that they thought likely had the closest shim. The operator verified the shim relative to the 0.0001" shim call. Usual acceptance was still about +/0.0007" . Obviously this built a more consistent end product.

.

 

[gar]

171120-1403 EST

Looking at the responses to my original post I do not believe that most have understood the question.

Possibly you need to view it as an abstract question.

This is not an application specific question. It is a math and visualization question. It is a question to make you think.

.

 

[Ingenieur]

171120-1403 EST

Looking at the responses to my original post I do not believe that most have understood the question.

Possibly you need to view it as an abstract question.

This is not an application specific question. It is a math and visualization question. It is a question to make you think.

.

it is a question with no answer :lol: