Cochran and McNemar test (CATA)

Cochran and McNemar test (CATA)

Purpose 

Cochran and McNemar tests are used to test for differences between products when the data has been collected through a ‘Check All That Apply’ (CATA) design. Using a CATA method for sensory research means that the responses collected are binary (attribute applies to product or doesn’t) and therefore these tests are the most appropriate to use.

Cochran’s Q will test for whether there are any product differences for each attribute. If this test suggests that there are differences, then McNemar pairwise tests establish which products are different from each other.

Data Format

  1. Example dataset: CATA emotions.xlsx
For EyeOpenR to read your data, the first five columns must be in the following order: assessor (consumer), product, session, replicate and order. The CATA data should be in the sixth column (column F) onwards. There should be one column for each attribute. The data should be binary, ie either 0 (not checked) or 1 (checked). 
If there is no session, replicate or order information then these columns should contain the value ‘1’ in each cell.
See the example spreadsheet for an illustration of the data format.

Background 

CATA (Check All That Apply) is a method for collecting information about sensory, functional or emotion attributes from consumers. It is a way of using consumers to assess attributes directly, as an alternative or to complement using a trained panel of assessors. Consumers are asked to assess whether an attribute applies to a product, and to repeat for all attributes. The aim is to understand which attributes are associated with which products. When each consumer assesses every product and identifies the attributes that they think fit each product the test is described as a sequential monadic test.

For sequential monadic tests with CATA data Cochran’s Q test will assess whether there is evidence that products are different for an attribute, in other words whether the proportion of consumers selecting the attribute varies between products. This is equivalent to the F-test in an ANOVA, it does not tell you which products are different, but it confirms whether differences exist. 

The McNemar test is used to test for pairwise differences between products for each attribute. The test is based on analysing the 2-way contingency table of product A vs product B and comparing marginal frequencies.

If each consumer only evaluates one product, instead of all products for the design, the test is described as a monadic test and EyeOpenR will perform a test of the difference between two proportions.

Options

  1. Treat Sessions/Replicates separately: If sessions or replicates are part of the design, the user can choose to treat sessions or replicas of a product as unique products (Product By Session / Product by Replica option). E.g. If you would like to treat three replicates of a product as three distinct individual products then check the ‘Product by Replica’ option. If the same judge ID was used for multiple sessions or replicas, the user can choose to treat sessions or replicas of a judge as unique judges (Judge by Session / Judge by Replica option). E.g. If you would like to treat three sessions of the same judge ID as three distinct individual assessors then check the 'Judge by Session' option.
  2. Display of Multiple Comparison test results: There are two options: pairwise or group. This refers to the display of the multiple comparison test in the Counts results tab (the McNemar results tab will not be changed). The Counts results tab will indicate in its name whether you have selected ‘pairwise’ or ‘group’. Also, the note at the bottom of the table will adjust to help you understand the output. If ‘group’ is selected, then the letters shown in the cells next to the counts indicate groups of products that are not significantly different (using the significance level selected in the next option). If ‘pairwise’ is selected, then the letters shown in the cells next to the counts indicate other products that are significantly different from the current product (using the significance levels selected in the ‘level of signif. (pairwise)’ option.
  3. Significance level: The significance level used for the McNemar test. It effects the results in the McNemar results tab and if ‘group’ is selected for the display of multiple comparison test results it effects the Counts results tab.
  4. Level of signif. (pairwise): If ‘pairwise’ is selected in the multiple comparison test this sets the significance level used for the pairwise comparisons in the Counts results tab.
  5. Number of decimals for values: Specify the number of decimal places shown for values in the results.
  6. Number of decimals for p-values: Specify the number of decimal places shown for p-values in the results.

Results and Interpretation

  1. Counts: This shows a table with the products in the rows against the attributes in the columns with each cell in the table showing the number of consumers checking the attribute for that product. Also shown on the table are the results of the McNemar tests. The way the results of the tests are presented depends on the option selected for ‘display of multiple comparison test results’. The Counts results tab will indicate in its name whether you have selected ‘pairwise’ or ‘group’. Also, the note at the bottom of the table will adjust to help you understand the output. If ‘group’ is selected, then the letters shown in the cells next to the counts indicate groups of products that are not significantly different (using the significance level selected in the next option). If ‘pairwise’ is selected, then the letters shown in the cells next to the counts indicate other products that are significantly different from the current product (using the significance levels selected in the ‘level of signif. (pairwise)’ option.
    Underneath the table is a bar chart of the counts, with bars coloured to indicate the results of the tests.
  2. Word Cloud: This displays the count data for each product as a word cloud. Each word is an attribute, and the size of the word relates to the number of consumers that associated that attribute with the product. The aim is to give a visual interpretation of the attributes associated with each product.
  3. Cochran: This shows the results of the Cochran’s Q tests. In the table there is one row for each attribute, shown with the value of Cochran’s Q (‘Q’), the degrees of freedom (‘df’) and the p-value. The Q value measures the difference between products in the proportion of consumers selecting the attribute, based on a chi-squared distribution. The degrees of freedom relate to the number of products in the test. The p-value is the probability of seeing a value of Cochran’s Q of the size shown with those degrees of freedom if there are no differences between products. The smaller the p-value the more evidence that at least one pair of products are different. In this way the Cochran’s Q statistic is like the F-value in an ANOVA.
  4. McNemar: This shows the results of the McNemar tests in a table for each attribute. In each table each row relates to a product. The rows are ordered by the number of consumers checking that attribute for that product, with the product with the most consumers ticking that attribute at the top of the table. Each column represents a group of products that are not statistically significantly different (using the McNemar test and the significance level set by the option ‘significance level’). If a product belongs to the column group, the count of consumers checking that attribute for that product is shown in the cell of the table. Use this table to understand which products are different for each attribute.
  5. Percentage: This shows the percentage of consumers who checked the attribute for each product.
  6. Information: This shows any notes or warnings that are relevant to the procedure.

Technical Information

R packages used:
  1. aggregate (stats) – for creating the contingency table.
  2. symmetry_test (coin) – using teststat=”quad” to run Cochran’s Q test.
  3. binom.test (stats) – for the McNemar test.
  4. ​prop.test (stats) – for the test of difference between two proportions if monadic test.

References 

  1. William G. Cochran (December 1950). "The Comparison of Percentages in Matched Samples". Biometrika. 37 (3/4): 256–266.
  2. McNemar, Quinn (June 18, 1947). "Note on the sampling error of the difference between correlated proportions or percentages". Psychometrika. 12 (2): 153–157. 

    • Related Articles

    • Check All That Apply (CATA)

      Introduction The "Check-All-That-Apply" (CATA) method is utilized in sensory evaluation to collect information regarding the sensory characteristics of a product. In this method, participants are presented with a predetermined list of sensory ...

    • Same/Different Test Analysis

      Available from version: 5.0.8.6 Purpose The Same Different Test is a discrimination test that is a variation of the paired comparison test. The assessor is presented with two samples and is asked to decide whether these samples are the same or ...

    • Penalty Analysis

      Purpose To provide a penalty analysis of a consumer data set, that is to investigate how liking or acceptability of product decreases when product attributes are not at the optimal intensity. Data Format Consumer.xlsx Note: for EyeOpenR to read your ...

    • T-test

      Purpose To make a statistical test for differences between two means. Or in the case that there are more than two samples, make separate difference tests for each possible pair of samples. Data Format The test is quite general and can be applied in ...

    • CATA Word Cloud

      Purpose A word cloud (or tag cloud) is a visual display of text data that represents each word (or phrase) in a font size that is proportional to the importance of each word. The relative importance of the words can then be easily appreciated simply ...