Producing Valid Results:
Risk Mitigation and Measurement Assurance
Quality measurements are not produced by accident. They are not the result of a single action, occurrence, or event. They are a collection of activities that are planned, interrelated and cohesive; they should be considered alongside the development of manufacturing processes and not an afterthought. Measurement activities are inextricably linked to product quality and manufacturing systems. These activities collectively are referred to as Measurement Assurance. Measurement assurance is good for product quality and good for business, and can even be legislated. The United States Code of Federal Regulations, Title 21- Food and Drugs states, “Each manufacturer shall ensure that all inspection, measuring, and test equipment, including mechanical, automated, or electronic inspection and test equipment, is suitable for its intended purposes and is capable of producing valid results.”(21 CFR 820.72)
A comprehensive Measurement Assurance program designed to mitigate risk has many components. It starts with the product design and identification of the required measurements and the process tolerances required to efficiently build a quality process or service. Once the process tolerances have been defined, test equipment must be selected to take these measurements. The test equipment selected must be suitable and appropriate for the measurement tasks and it must also be capable of producing valid results. Even the proper instrument can produce invalid results if not handled, maintained, used and stored properly. Even if these events happen, but are not documented by objective evidence, the intent of the measurement assurance program, mitigating risk, can be derailed.
Every product begins with an idea to fill an observed need. The idea is clarified, refined and takes shape and form. These shapes and forms become sketches, drawings, operational and performance parameters all of which get broken down into sections, subsections and assemblies. At this point, we start building from the bottom up which requires procedures and work instructions. Inevitably the need for measurement crops up in many parts of this process. Acceptability limits and tolerances are developed for these measurements and become the process tolerances.
Ideally, it is the process tolerances that should drive the test and measurement equipment selection. Once we know what measurements are needed, we have to select a measurement process that has the appropriate capability, practical and affordable from a business perspective. There can be many different measurement techniques or methods to get the needed measurement information. When selecting measurement techniques considerations should be given to wiring; cables and connectors, the speed of measurement, data acquisition rates and refresh rates, manual or automated measurements. And let’s not forget about environments effects on measurements, temperature, humidity, barometric pressure, and particulate contamination. Then there are the business aspects to consider such as initial equipment acquisition costs, ongoing cost such as preventative maintenance and calibration, operator training requirement, safety requirements, instrumentation availability, product and instrument life expectancy and last by not least, overall risk related to these decisions.
There are many different factors associated with determining measurement instrumentation suitability for the intended purpose. Some of those technological factors are the measurement parameter, the range of measurement, accuracy requirements, resolution requirements, and measurement repeatability and reproducibility, long-term and short-term stability. Once all of these questions have been answered and the instrumentation has been selected and purchased, there is still work to be done to assure that test equipment will produce valid results.
What does it take to produce valid results? Assuming you have a well-defined process including process specifications, researched and selected an appropriate measurement method, and have acquired suitable measurement instrumentation, there still is no guarantee you will get valid measurement results. You can have the best instruments in the world, but if they are not properly cared for, they may not measure correctly. You can have the best-maintained instruments, but without the proper knowledge and procedures, they may not measure correctly. You can have the best instruments, properly cared for, with a well-defined process, but operated by unqualified, or under trained people, you may not get valid measurement results. There are three fundamental areas, the equipment, the process and the people that must come together to achieve valid results.
The single most important thing that should be done is to READ THE MANUAL, the ENTIRE manual, including all those pesky little footnotes. There is a tremendous amount of information located here that identifies how to get valid measurement results. You can only get so much information from marketing literature and specification datasheets, and you would be surprised how often this information will differ between these documents! Reading instrumentation manuals is not the most exciting thing in the world, but as the old saying goes “an ounce (gram) or prevention is worth a pound (kilogram) of cure”.
All measurement equipment requires some level of preventative maintenance, period, no exceptions. I can say this because calibration really is a preventative maintenance function. Calibration is a periodic proactive maintenance function to assure the device is providing valid results. But calibration may not be the only preventative maintenance required. There can be many maintenance different levels and intervals. For discussion purpose, I will classify them as operator level maintenance and organizational level maintenance.
Operator level maintenance activities are required for daily use or a short maintenance interval. Some examples include setting or checking the zero of micrometers and calipers, oscilloscope often require a “self-calibration” routine for channel compensation, torque wrenches may need to be exercised once or twice after a setting change, air driven torque wrenches have moisture traps and filter that need to be drained daily.
Organizational level maintenance activities are actions that require additional skills, equipment or time and typically have longer maintenance intervals such as weekly, monthly, quarterly or annual maintenance. For example, a two-pressure, two-temperature humidity generator uses distilled water that needs to be checked and refilled, it has a fluid-filled temperature jacket that requires a periodic level check and filling, it has a pressure pump that has multiple filters that need replacement and the pump itself has periodic maintenance of it gaskets and seals. To add another level of complexity, each of these maintenance requirements has different intervals. Typically, the greater the instrument's accuracy, the greater the maintenance requirements will be to maintain that accuracy. And of course we must include calibration here; after all, it is a preventative maintenance function. I certainly hope nobody waits for poor measurement results to appear before sending a unit in for calibration!
Preventative maintenance isn’t limited to the instrumentation and the requirements spelled out in the manual. There are other aspects which must be considered, remember we are actually dealing with a measurement process, not just instruments. Consider how the instrument is physically connected to the manufacturing process. The cable, connectors, and fittings all will need periodic inspections. There is nothing worse than chasing around an intermittent problem that leads back to a faulty cable. A leaky pressure fitting even at low pressures can lead to greater measurement variation. As a lean manufacturing engineer, I was responsible for a production cell that routinely built 100,000 psi pressure transducers. You can be assured we had a preventative maintenance program in place to inspect and replace the pressure fittings on our test stands on a regular basis for the safety of our employees! Since I also have a background in Radio Frequency and microwave calibration, I am quite particular about the cleanliness of connectors and the impact that can have on reliable readings. This also applies to DC and low-frequency applications, the banana jacks on 8.5 digit multimeters can become quite dirty and affect readings on the most sensitive ranges unless they are cleaned periodically. And let’s not forget about software and firmware updates. Even today’s handheld multimeters have firmware that may need updates that could affect the validity of its readings.
Another aspect to consider is the environment where the measurements will be taken. This is especially true when dealing with dimensional measurements. Any parts inspector or machinist understands the effect of temperature on parts. For example, a batch of part arrives on a delivery truck that has been driving around in the winter months. The parts are at 32°F and the incoming part inspector is working in a comfortable 72°F. That means there is a 40°F temperature difference between the parts being measured and the inspectors measuring tools. This 40°F difference can result in a 0.000256” error in the part size due to the material coefficients of thermal expansion (CTE), assuming it is a 1” part made of steel, and the CTE is 0.0000064”/°F/”. Temperature effects are not limited to dimensional measurements. I once assisted a customer who was performing battery tests and measuring a few different parameters. They had selected equipment that was suitable to measure the parameters in question-based on the published meter specifications. Upon visiting their facility they showed me their test stand, which happened to be in a room maintained at 37°C (98.6°F), the operating temperature of their batteries, which was a great idea. The meter they were using had an operational temperature range of 0°C (32°F) to 55°C (131°F), again great, somebody was paying attention. Unfortunately, there was a note in the manual that the specification for one of the parameters they were using that stated the accuracy specification applied to a temperature range of 18°C (64°F) to 28°C (82°F). They were operating 9°C (16.6°F) outside of the accuracy specification! Fortunate for the customer, a footnote contained a specification adder which widened the accuracy specification to cover their operating temperature. It is amazing what you can find out when you read the entire manual. All ended well, their system was producing valid results, but they were not as accurate as they thought they were, they just had a lower Process Accuracy Ratio (PAR) than they thought they were getting.
Another process aspect worthy of consideration is the proper handling and storage of measurement equipment. When equipment is not being used it is important to store it properly. For instance, let’s look at torque wrenches. Most torque wrenches use a spring in their design. When this spring is left compressed for prolonged periods of time, it permanently changes the response of the spring, which causes a change in the value of the applied torque. Therefore, most torque wrenches have a note instructing the user to return it to the lowest setting after use. That is also why period calibration is important for torque wrenches. Torque wrenches can also be knocked out-of-tolerance if they are dropped. Basically, this applies to most mechanical measurement devices such as calipers and micrometers. Dropped instruments should be checked by the calibration lab. Handling consideration applies to many different parameters, in the temperature parameter, precision temperature probes are highly susceptible to physical shock that can cause a permanent shift in the probe's response and render the probe useless if the shock is severe enough. They are also susceptible to thermal shock if dramatic temperature changes happen like removal from a hot dry block or oven. When test equipment is not in use, it should be stored in appropriate cases and locations whenever possible.
Another often overlooked process is the movement and control of test equipment. Handheld tools and instruments have a tendency to migrate around the production floor and “borrowed” for a quick check somewhere else. This uncontrolled and undocumented movement and use of measurement equipment for undesignated purposes, by untrained personnel, increases the potential risk of misuse and abuse. I have seen the results of many abused instruments, torque wrenches used to loosen stuck bolts, and “cheater” pipes and bars to increase leverage, handheld meters used to measure a car battery currents, gage blocks used as leveling blocks and to prop up fixtures, weights used as doorstops, overzealous tapping of pressure gages to “un-stick” pointers, and temperature probes used as stir sticks. It is also amazing how often an abused piece of test equipment suddenly reappears in its original location. It is important to maintain the integrity of the measurement process to control and document the movement of test and measurement equipment. The history of movement and use of measurement equipment is critically important in the evaluation process when equipment is found out-of-tolerance at its next calibration. If you don’t know when, where and how an out-of-tolerance unit was used, you will have very little knowledge as to what may be at risk.
And finally, we get to the greatest variable of all, your greatest resource, your people. Your operators and technicians are a key element in assuring valid measurement results. They must be training on the how to properly operate the measurement equipment they are provided. It one thing to read a procedure or a manual, but real learning takes place with hands-on training and interaction. Even after training, there are some areas of measurement that may require additional focus to achieve repeatable and reproducible valid result. Some measurement methods are heavily reliant on the operator measurement technique, such as dimensional measurement, pipette use, and torque wrench use. I recently had the tires on my car rotated and I watched the technician grab a torque wrench, get the proper adapter, look up the proper torque value, set the wrench. Then he yanked on the wrench until he hears the click and promptly repeats that three times for each lug nut. He “torqued “all twenty of the lug nut in less than one minute. All his time and efforts were wasted because of poor technique, there is no way those lug nuts were at the proper torque. I have run studies on experienced calibration technicians calibrating torque wrenches without a mechanical loader and have witness results that vary 4% to 6% differences in readings between operators, and as much as 16% variations, all using a 4% torque wrench. I can only imagine what results are being delivered by inexperienced users who are in a hurry like my automotive technician. Torque and any measurement procedures that rely on operator technique should have a gage repeatability and reproducibility study completed. This is an effective way to provide feedback to the operators and help them develop effective techniques. Gage R&R studies are not a onetime event, just like calibration and training they should be repeated on a regular basis, especially when there is a change in the associated measurement process or when new operators are introduced into the process.
Measurement assurance is not a “set it and forget it” process, it requires on-going efforts and planning to maintain your ability to produce valid results and risk mitigation. The interaction your equipment, your process and you people are critical to your ability to achieve valid results. A failure in any one of these areas can nullify all your other efforts or at the minimum, increases your potential risk. I do not have all the answers and cannot possibly identify all the potential areas of risk, but, isn’t that what all this is about, increasing your knowledge, and continuous process improvement? Hopefully, this paper will spark some ideas that you can build upon and assist you on your measurement journey.
About the Author
Phil Mistretta is a Metrology Manager for Transcat Inc. in Rochester, NY. Phil has over 25 years of experience in the field of Metrology. He started his metrology training on active duty in the U.S. Army from 1987 until 1996. He also has a background in lean manufacturing engineering and EMC/EMI Compliance Testing. He is a member of IEEE and ASQ and is an ASQ-Certified Calibration Technician. He is a graduate of Central Texas College and pursuing a BS in Engineering Physics at the University of Buffalo New York on a part-time basis.