Product Testing

Product testing is, perhaps, the single-most-important type of consumer research any company ever conducts. A company with consistently superior products tends to consistently outperform its competitors in the marketplace. Achieving clear-cut product superiority in a category is the surest way to build brand share, engender customer loyalty, and boost profitability. Better products tend to command higher prices and be more responsive to advertising investments.

Our Product-Testing Philosophy

Consumer tastes and preferences evolve over time. A consumer's palate is a moving target. That's why product testing must be viewed as a strategic activity.

"Real environment" testing (i.e., testing a product the way it is typically used by consumers) is almost always the most accurate method of product evaluation. For example, it is generally best to let consumers evaluate food products in their homes rather than in a laboratory or test kitchen.

Products should be tested using a standardized analytical research system, so that each product is tested in exactly the same way. Our analytical research system is Optima®.

Optima® Product Testing

Optima® is Decision Analyst's proprietary, copyrighted product-testing system. The Optima® core questionnaire consists of the following modules:

• Screening/Qualification Questions
• Overall Rating of Product
• Likes
• Suggested Product Improvements
• Diagnostic Ratings of Product Attributes
• Ratings of Product Components
• Purchase Intent, Priced
• Expected Purchase Frequency
• Value Rating
• Demographics

If the product test is a part of a new-product volumetric forecast, then additional modules are added to the core questionnaire.

How Does Optima® Work?

Typically, a representative sample of category users (150 to 200 households) is given a test product to use in home for a few days. Then these consumers are asked a series of standard questions about the product.

Based on internal diagnostics, normative data, and analytical models, these questions tell us whether the product is optimal or not, and indicate what needs to be changed to improve the product. Our primary analytic model is Pii®.

Product Improvement Index® (Pii®)

Decision Analyst developed the Pii® mathematical model to help guide product-development efforts for new products and the reformulation of existing products. Pii®was developed primarily because of problems encountered in using various types of regression models in product-testing analyses. Regression models assume that all of the input variables are independent (i.e., not intercorrelated in any way).

The reality is, however, that virtually all of the input variables that might explain a product's performance are typically intercorrelated. The result is a regression equation that masks or omits important input variables. For example, if the color and the sweetness of a product happened to be highly correlated with each other, the regression equation would omit one of the variables. We might think we had a color problem, when in fact we had a color and a sweetness problem.

The Pii® model was designed to circumvent the "missing variables" problem associated with regression. This model is based upon a type of correlation, using "dummy" variables, to examine the relationship between the diagnostic ratings (e.g., too sweet, about right, not sweet enough; or too much salt, about right, not enough salt) and the consumer's overall rating of the product. The overall rating is typically measured with an 11-point scale.

The output of the Pii® model is a table of important explanatory variables along with the Pii® rating and the indicated action, as illustrated here.

Generally, any Pii® score greater than 4.0 indicates that some modification of the product might be necessary. The greater the Pii® score, the more important that variable is and the more that variable should be modified.

Optimization Methods

In addition to Pii® analyses, response surface and choice modeling analyses are the primary optimization techniques. Experimental designs and simulation models are employed to optimize products. By testing chosen subsets of product possibilities, response surface and choice modeling can simulate and predict consumer preferences for hundreds of product possibilities, as defined by variations in ingredients, features, elements, or packages. The resulting equations are used to build an optimization simulator so that “what if” products can be fully explored and understood. The goal of optimization can vary. It might be maximizing consumer preference, or maximizing the profit margin without losing market share, or maximizing sales potential. The optimization simulator also helps reveal “cause and effect” as inputs are changed and outcomes vary.

Optima® Uses

The value of Optima® product testing is illustrated by its many uses:

• To evaluate and improve existing products.

• To measure the threat posed by competitive products.

• To continuously improve product performance over time (i.e., to optimize products).

• To evaluate cost-reduction formulations while maintaining product superiority.

• To measure the effects of aging upon product quality (shelf-life studies).

• To provide guidance to R&D in developing or upgrading products.

• To monitor product quality from different suppliers and/or different factories.

• To implicitly evaluate marketing variables (i.e., packaging, pricing, sizing, etc.).

• To predict the success of new products.

Optima® Features

The essential features of Optima® product testing are:

• Monadic Design. Each product is tested alone. This provides the most accurate evaluation of the product and the best diagnostic feedback about how to improve the product.

• Standardized Systems. The sampling, data collection, data preparation, and data tabulation methods and procedures (i.e., the system) are standardized for each product. It is essential that every product be tested in precisely the same way.

• Standardized Questions. The standardized questionnaire is modular in structure and flexible in design. All of the questionnaire modules are copyrighted by Decision Analyst.

• Analytical Research System. Optima® is an integrated research system with mathematical models to yield greater analytical insights.

Sampling Options

The recommended sample source for most product tests is American Consumer Opinion® Online. This Decision Analyst panel of over 8 million online consumers is an economical way to screen for low-incidence product categories, and it provides high-quality data and fast turnaround at the conclusion of each product test.

Other sampling options include mall-intercept, recruit to central location, and telephone recruitment. Central location, in-store, and on-site product testing is also an option.

Non-food Product Testing

The concepts, methods, and techniques of product testing can be adapted and applied to almost any product category. Our staff has evaluated:

Product Testing Services

Decision Analyst is a recognized leader in consumer product testing and optimization. Its staff has evaluated more than 1,000 foods, beverages, and other products during the past 30 years. The firms has over 50 staff members with extensive experience in the conduct and analysis of product test and optimization studies. The company is a leader in the development of analytical techniques to enhance product testing and optimization.

If you would like more information on product testing, please contact Jerry W. Thomas by emailing jthomas@decisionanalyst.com or calling 1-800-ANALYSIS (262-5974) or 1-817-640-6166.

Additional Resources From Decision Analyst

Related Services

• Custom Product Testing Designs
• Product Quality Monitoring Systems

Brochures

• ConceptTest® Brochure (opens a pdf file)
• Innovation-Services Brochure (opens a pdf file)
• New-Product Metamorphosis Brochure (opens a pdf file)
• Product-Testing Brochure (opens a pdf file)
• Package-Design Research Brochure (opens a pdf file)
• Complete List of Decision Analyst Brochures

• Choice Modeling for New Product Sales Forecasting
• New Product Sales Forecasting
• New Products
• New Products for Tough Times
• Product Testing 
• Restaurant Industry Losing Low-Fat War