Product Testing by Jerry W. Thomas

Home | Marketing Research White Papers | Marketing Research Techniques | Product Testing

Product Testing
by
Jerry W. Thomas

Based upon 30 years of marketing research experience, spanning thousands of research projects, I am convinced that product testing is the single most valuable marketing research that most companies ever do. The great value of product testing is, perhaps, best illustrated by some of its many uses. It can be used to:

Achieve product superiority over competitive products.
Continuously improve product performance and customer satisfaction (i.e., to maintain product superiority, especially as consumer tastes evolve over time).
Monitor the potential threat levels posed by competitive products to understand competitive strengths and weaknesses.
Cost-reduce product formulations and/or processing methods, while maintaining product superiority.
Measure the effects of aging upon product quality (shelf-life studies).
Implicitly measure the effects of price, brand name, or packaging upon perceived product performance/quality.
Provide guidance to research and development in creating new products or upgrading existing products.
Monitor product quality from different factories, through different channels of distribution, and from year to year.
Predict consumer acceptance of new products.

Companies committed to rigorous product testing and continuous product improvement can, in most instances, achieve product superiority over their competitors. Product superiority, in turn, helps strengthen brand share, magnifies the positive effects of all marketing activities (advertising, promotion, selling, etc.), and often allows the superior product to command a premium price relative to competitors.

Most companies, unfortunately, do very little product testing. Few companies really understand the power of continuous product improvement and product testing. Even fewer companies know how to do product testing the right way. Fewer yet budget enough money to support a serious product-testing program. These shortcomings in the majority of companies create opportunities for the minority of companies who are dedicated to continuous product improvement. How can companies realize optimal value from product testing?

Product Testing Secrets

The secrets to truly accurate and actionable product testing are several:

A systems approach. The methods and procedures of product testing should constitute a standardized system, so that every like product is tested exactly the same way, including identical product preparation, age, packaging, and coding; identical questionnaires (of course, parts of the questionnaire must be adapted to different product categories); identical sampling plans, typically employing blocking-screening grids to ensure matched samples; identical data preparation and tabulation methods; and similar analytical methods.
Normative data. As products are tested over time, the goal is to build normative databases, so that successive product tests become more meaningful and valuable. The normative data, or norms, continually improve a company’s ability to correctly interpret product-testing scores, and the norms help reveal exactly how good, or how bad, the test product is.
Same research company. Use one research company for all of your product testing. This is the only way you can make sure all tests are conducted in exactly the same way.
Real environment test. If the product is used in offices, it should be tested in offices by people who work in offices. If the product is typically used at home, it should be tested at home. If the product is consumed in restaurants, it should be tested in restaurants, and so on. In general, this kind of “real environment” test will produce the most accurate results. For example, for food products, an in-home usage test is almost always more accurate and predictive than a central-location taste test.
Relevant universe. Sampling is a critical variable in product testing. For new products or low-share products, the sample should reflect, or represent, the brand share makeup of the market. For well-established, high-share or highly differentiated products, the sample should contain a readable subsample of that product’s users, and a readable cell of nonusers. If the product category is underdeveloped (e.g., a relatively new category), then the sample should include nonusers of the category, as well as users. Also, it’s always important to represent medium to heavy users of the product category in the final sample.

In summary, if a company’s brand share is very low, it’s important to assign more weight (or importance) to the opinions from nonusers of the brand. If brand share is very high, then what brand users think is more important.
Critical variables. Product performance and quality must be defined from the consumer’s perspective, not the manufacturer’s. What aspects of the product are truly important to consumers? What critical variables determine the consumer’s satisfaction with the product? These critical variables must be identified for each product category (typically, with focus groups or depth interviews) to design an accurate product-testing system.
Conservative actions. The formulation of an established product should never be changed without careful testing and evaluation of the new formulation. Once you are sure you have a better product, introduce it into a limited geographic area for a reasonable time period (several product repeat purchase cycles). Then, and only then, roll the new product out to all markets. The smaller the market share, the greater the risks that can be taken with a new formulation. The larger the market share, the more conservative one should be in introducing a new formulation.

The Major Techniques

The monadic, sequential monadic, paired-comparison, and protomonadic are the most widely used research designs for product testing.

Monadic testing typically is the best method. Testing a product by its own offers many advantages. Interaction between products (which occurs in paired-comparison tests) is eliminated. The monadic test simulates real life (that’s the way we usually use products, one at a time). By focusing the respondent’s attention upon one product, the monadic test provides the most accurate and actionable diagnostic information.

Additionally, the monadic design permits the use of normative data and the development of norms and action standards. Virtually all products can be tested monadically, whereas many cannot be accurately tested in paired-comparison designs. For example, a product with a very strong flavor (hot peppers, alcohol, etc.) may deaden or inhibit the taste buds so that the respondent cannot really taste the second product.
Sequential monadic designs are often used to reduce costs. In this design, each respondent evaluates two products (he or she uses one product and evaluates it, then uses the second product and evaluates it). The sequential monadic design works reasonably well in most instances, and offers some of the same advantages as pure monadic testing.

One must be aware of what we call the “suppression effect” in sequential monadic testing, however. All the test scores will be lower in a sequential monadic design, compared to a pure monadic test. Therefore, the results from sequential monadic tests cannot be compared to results from monadic tests. Also, as in paired-comparison testing, an “interaction effect” is at work in sequential monadic designs. If one of the two products is exceptionally good, then the other product’s test scores are disproportionately lower, and vice versa.
Paired-comparison designs (in which the consumer is asked to use two products and determine which product is better) appeal to our common sense. It’s a wonderful design if presenting evidence to a jury, because of its “face value” or “face validity.” The paired-comparison can be a very sensitive testing technique (i.e., it can measure very small differences) between two products. Also, the paired-comparison test is often less expensive than other methods, because sample sizes can be smaller in some instances.

Paired-comparison testing, however, is limited in value for a serious, ongoing product-testing program. The paired-comparison test does not tell us when both products are bad. The paired-comparison test does not lend itself to the use of normative data. The paired-comparison test is heavily influenced by the “interaction effect” (i.e., any variations in the control product will create corresponding variance in the test product’s scores).
The protomonadic design (and the definition of this term varies greatly from researcher to researcher) begins as a monadic test, followed by a paired-comparison. Often, sequential monadic tests are also followed by a paired-comparison test. The protomonadic design yields good diagnostic data, and the paired-comparison at the end can be thought of as a safety net—as added insurance that the results are correct. The protomonadic design is typically used in central-location taste testing, not in-home testing (because of the complexity of execution in the home).

Nonpackaged Goods Categories

While most product testing is conducted in the food and beverage industries, the concepts and methods of product testing are applicable to virtually all product categories, although the structure and mechanics of execution will vary greatly from product category to product category. For example, computer software can be tested, furniture can be tested, store environments can be tested, dog food can be tested, airline service can be tested, equipment prototypes can be tested, etc.

Competitive Advantage

The ultimate benefit of product testing is competitive advantage. Product superiority is the surest way to dominate a product category or an industry. Companies dedicated to ongoing product improvement and product testing can achieve product superiority, and achieve a competitive advantage of great strategic significance.

Companies that ignore product improvement and product testing, on the other hand, may wake up one morning to find themselves on the brink of extinction from a competitor who has built “a better mousetrap.”

About the Author

Jerry W. Thomas (jthomas@decisionanalyst.com) is President/CEO of Dallas-Fort Worth based Decision Analyst. He may be reached at 1-800-262-5974 or 1-817-640-6166.

Additional Resources from Decision Analyst

Related Services

Related Case Histories

Related White Papers