Jerry W. Thomas
Based upon 40 years of marketing research experience spanning thousands of research projects, I am convinced that product testing is the single most valuable
marketing research that most companies ever do. The great value of product testing is perhaps best illustrated by some of its many uses. It can be used to:
- Achieve product superiority over competitive products.
- Continuously improve product performance and customer satisfaction (i.e., to maintain product superiority, especially as consumer tastes evolve
- Monitor the potential threat levels posed by competitive products, to understand competitive strengths and weaknesses.
- Reduce costs of product formulations and/or processing methods, while maintaining product superiority.
- Measure the effects of aging upon product quality (shelf-life studies).
- Implicitly measure the effects of price, brand name, or packaging upon perceived product performance/quality.
- Provide guidance to research and development in creating new products or upgrading existing products.
- Monitor product quality from different factories, through different channels of distribution, and from year to year.
- Predict consumer acceptance of new products.
Companies committed to rigorous product testing and continuous product improvement can, in most instances, achieve product superiority over their competitors.
Product superiority, in turn, helps strengthen brand share, magnifies the positive effects of all marketing activities (advertising, promotion, selling, etc.),
and often allows the superior product to command a premium price relative to competitors.
Most companies, unfortunately, do very little product testing. Few companies
really understand the power of continuous product improvement and product testing.
Even fewer companies know how to do product testing the right way. Fewer yet budget
enough money to support a serious product-testing program. These shortcomings
in the majority of companies create opportunities for the minority of companies
who are dedicated to continuous product improvement. How can companies realize
optimal value from product testing?
The secrets to truly accurate and actionable product testing are several:
- A Systems Approach. The methods and procedures of product testing should constitute a standardized system, so that every like
product is tested exactly the same way, including;
- Identical product preparation, age, packaging and coding.
- Identical questionnaires (of course, parts of the questionnaire must be adapted to different product categories).
- Identical sampling plans, typically employing blocking-screening grids to ensure matched samples.
- Identical data preparation and tabulation methods.Similar analytical methods
- Normative Data. As products are tested over time, the goal is to build normative databases, so that successive product tests become
more meaningful and valuable. The normative data, or norms, continually improve a company’s ability to correctly interpret product-testing scores, and the
norms help reveal exactly how good, or how bad, the test product is.
- Same Research Company. Use one research company for all of your product testing. This is the only way you can make sure all tests
are conducted in exactly the same way.
- Real Environment Test. If the product is used in offices, it should be tested in offices by people who work in offices. If the
product is typically used at home, it should be tested at home. If the product is consumed in restaurants, it should be tested in restaurants, and so on. In
general, this kind of “real environment” test will produce the most accurate results. For example, for food products an in-home usage test is almost
always more accurate and predictive than a central-location taste test.
- Relevant Universe. Sampling is a critical variable in product testing. For new products or low-share products, the sample should
reflect, or represent, the brand-share makeup of the market. For well established, high-share (or highly differentiated) products, the sample should contain a readable
subsample of that product’s users and a readable cell of nonusers. If the product category is underdeveloped (e.g., a relatively new category), then the
sample should include nonusers of the category, as well as users. Also, it’s always important to represent medium to heavy users of the product category in
the final sample.
In summary, if a company’s brand share is very low, it’s important to assign more weight (or importance) to the opinions of nonusers of the brand.
If brand share is very high, then what brand users think is more important.
- Critical Variables. Product performance and quality must be defined from the consumer’s perspective, not the manufacturer’s.
What aspects of the product are truly important to consumers? What critical variables determine the consumer’s satisfaction with the product? These critical variables
must be identified for each product category (typically, with focus groups or depth interviews) to design an accurate product-testing system.
- Conservative Actions. The formulation of an established product should never be changed without careful testing and evaluation of the
new formulation. Once you are sure you have a better product, introduce it into a limited geographic area for a reasonable time period (several product repeat
purchase cycles). Then, and only then, roll the new product out to all markets. The smaller the market share, the greater the risks that can be taken with a new
formulation. The larger the market share, the more conservative one should be in introducing a new formulation.
The Major Techniques
The monadic, sequential monadic, paired-comparison, and protomonadic research designs are the most widely used research designs for product testing.
- Monadic Testing typically is the best method. Testing
a product on its own offers many advantages. Interaction between products (which
occurs in paired-comparison tests) is eliminated. The monadic test simulates real
life (that’s the way we usually use products—one at a time). By focusing
the respondent’s attention upon one product, the monadic test provides the
most accurate and actionable diagnostic information. Additionally, the monadic
design permits the use of normative data and the development of norms and action
standards. Virtually all products can be tested monadically, whereas many cannot
be accurately tested in paired-comparison designs. For example, a product with
a very strong flavor (hot peppers, alcohol, etc.) may deaden or inhibit the taste
buds so that the respondent cannot really taste the second product.
- Sequential Monadic Designs are often used to reduce
costs. In this design, each respondent evaluates two products (he or she uses
one product and evaluates it, then uses the second product and evaluates it).
The sequential monadic design works reasonably well in most instances, and offers
some of the same advantages as pure monadic testing. One must be aware of what
we call the “suppression effect” in sequential monadic testing, however.
All the test scores will be lower in a sequential monadic design, compared to
a pure monadic test. Therefore, the results from sequential monadic tests cannot
be compared to results from monadic tests. Also, as in paired-comparison testing,
an “interaction effect” is at work in sequential monadic designs.
If one of the two products is exceptionally good, then the other product’s
test scores are disproportionately lower, and vice versa.
- Paired-Comparison Designs (in which the consumer
is asked to use two products and determine which product is better) appeal to
our common sense. It’s a wonderful design if presenting evidence to a jury,
because of its “face value” or “face validity.” The paired
comparison can be a very sensitive testing technique (i.e., it can measure very
small differences) between two products. Also the paired-comparison test is often
less expensive than other methods, because sample sizes can be smaller in some
Paired-comparison testing, however, is limited in value for a serious, ongoing
product-testing program. The paired-comparison test does not tell us when both
products are bad. The paired-comparison test does not lend itself to the use of
normative data. The paired-comparison test is heavily influenced by the “interaction
effect” (i.e., any variations in the control product will create corresponding
variance in the test product’s scores).
- The Protomonadic Design (and the definition of
this term varies from researcher to researcher) begins as a monadic test, followed
by a paired comparison. Often sequential monadic tests are also followed by a
paired-comparison test. The protomonadic design yields good diagnostic data, and
the paired comparison at the end can be thought of as a safety net—as added
insurance that the results are correct. The protomonadic design is typically used
in centrallocation taste testing, not in-home testing (because of the complexity
of execution in the home).
Nonpackaged Goods Categories
While most product testing is conducted in the food and beverage industries, the concepts and methods of product testing are applicable to virtually all product
categories, although the structure and mechanics of execution will vary from product category to product category. For example, computer software can be tested, furniture
can be tested, store environments can be tested, toys can be tested, airline service can be tested, equipment prototypes can be tested, etc.
The ultimate benefit of product testing is competitive advantage. Product superiority is the surest way to dominate a product category or an industry. Companies
dedicated to ongoing product testing can achieve product superiority. Companies that ignore product testing, on the other hand, may wake up one morning to find
themselves on the brink of extinction from a competitor who has built “a better mousetrap.”
Copyright © 2010 by Decision Analyst, Inc.
This article may not be copied, published, or used in any way without written permission of Decision Analyst.
|Share This Article
| Download This Article