Ranking and Ranking (continuous data)

Ranking and Ranking (continuous data)

Purpose 

Sometimes products are assessed by asking assessors to rank them for a particular attribute, for example asking them to rank products by their sweetness. 

The Ranking module uses Friedman’s test to assess whether there are any differences between products for each attribute, and Nemenyi’s multiple comparison test to establish which products are different from each other. These are non-parametric tests used when every assessor has ranked every product.

The Ranking (continuous data) module performs the same analysis, but the inputs are continuous data (rating scores or assessments on a continuous scale) that are converted to ranks in the analysis.

Data Format

Ranking

  1. Ranking.xlsx
For EyeOpenR to read your data, the first five columns must be in the following order: assessor (consumer), product, session, replicate and order. The attribute data with the ranks should be in the sixth column (column F) onwards. There should be one column for each attribute. 

The analysis requires a complete design and if an incomplete design is detected the analysis will not run and a note will be added to the information tab in the results. The session, replicate and order information is not used for this analysis and these columns should contain the value ‘1’ in each cell (order can be recorded as ‘NA’). 

If two or more products are tied for an assessor the rank must be the average of the positions that they represent. Consider the example where each assessor has ranked the sweetness of 5 products, A, B, C, D and E. Here are a selection of ranks that could be recorded:
  1. The assessor can discriminate between all five products and ranks them in the following order: B, D, A, C, E. The ranks are recorded as A=3, B=1, C=4, D=2, E=5.
  2. The assessor cannot discriminate between D and A and ranks them in the following order: B, D & A, C, E. The ranks are recorded as A=2.5, B=1, C=4, D=2.5, E=5.
  3. The assessor cannot discriminate between D, A and C  and ranks them in the following order: B, D & A & C, E. The ranks are recorded as A=3, B=1, C=3, D=3, E=5.
The data sheet is necessary, the other sheets provide metadata and if they are not in the spreadsheet you can assign the metadata in EyeOpenR. 
See the example spreadsheet for an illustration of the data format.

Ranking (continous data)

  1. Ranking.xlsx
For EyeOpenR to read your data, the first five columns must be in the following order: assessor (consumer), product, session, replicate and order. The attribute data with the rating scores or assessments should be in the sixth column (column F) onwards. There should be one column for each attribute. 

The analysis requires a complete design and if an incomplete design is detected the analysis will not run and a note will be added to the information tab in the results. The session, replicate and order information is not used for this analysis and these columns should contain the value ‘1’ in each cell (order can be recorded as ‘NA’). 

The data sheet is necessary, the other sheets provide metadata and if they are not in the spreadsheet you can assign the metadata in EyeOpenR. 
See the example spreadsheet for an illustration of the data format.

Background 

Untrained assessors may find it easier to rank products for a given attribute rather than scoring on a rating scale, e.g. being asked to sort products based on sweetness is more straightforward than assigning a level of sweetness to each product. It is generally quicker and easier to collect data in this way. Non-parametric methods that use no distributional assumptions can be used to analyse data collected as ranks.

The Friedman test is designed to assess whether there is a consistent pattern in ranks between the products, in other words whether any of the products are consistently ranked higher or lower than the rest. It is a non-parametric test that is equivalent to two way analysis of variance. It does not tell you which products are different, but it confirms whether differences exist.

Nemenyi multiple comparison tests are used to evaluate which groups of products are different from each other. It is a non parametric comparison of all possible pairwise combinations.
These methods can also be used for continuous (interval data type) data if you have concerns about the assumptions that underpin ANOVA.

Options

  1. Level of significance: There are three options, 0.01, 0.05 and 0.1. This is the significance level used for the Friedman test and the Nemenyi multiple comparison test.
  2. Multiple comparison test: The only option here is Nemenyi. This means that the multiple comparison test will be Nemenyi.
  3. For continuous Invert ranks: Select yes if a higher value in the continuous data indicates a product that would have a lower number rank. The ranks shown in the Frequency tab of the results are unchanged by this option, but the rank sums shown in the Summary and Nemenyi tabs of the results are changed.
  4. Number of decimals for values: Specify the number of decimal places shown for values in the results.
  5. Number of decimals for p-values: Specify the number of decimal places shown for p-values in the results.

Results and Interpretation

The results described below are shown for each attribute separately. They are accessed by selecting the attribute from the tabs below the results tab.
  1. Frequency: This shows a table with the ranks in the rows against the products in the columns with each cell in the table showing the number of assessors who assigned each product each rank. The ranks are either those in the input data for the Ranking module or those calculated by EyeOpenR for the Ranking (continuous data) module. This table is useful for checking whether there is a product that divides opinion between assessors.
  2. Summary: This table shows a summary of the rank statistics for each product. The top row is the rank sum, these are used in the Friedman and Nemenyi tests. Also shown are the median rank, the mean rank and the standard deviation of the ranks. 
  3. Friedman: This shows the results of the Friedman tests. The ‘Chi-squared’ value is the test statistic based on the differences in the sum of ranks for each product; the ‘Degrees of freedom’ gives context to the size of the chi-squared value and relates to the number of products (number of products minus one); and the ‘P-value’ is the probability of a chi-squared value of the size shown with those degrees of freedom if there were no differences between the products. The smaller the p-value the more evidence that at least one product is different from the others. In this respect the chi-squared statistic from the Friedman test is like the F-value in an ANOVA. Two footnotes below the table explains the meaning of the Significance column. These say: ‘A ‘*’ in the Significance column indicates that the p-value is lower than the threshold considered (here the value selected on the Options page)’ and ‘NS in the Significance column indicates that the p-value is higher than the threshold considered (here the value selected on the Options page)’. 
  4. Nemenyi: This shows the results of the Nemenyi multiple comparison test based on the differences in rank sums. Each row of the table relates to a product. The rows are ordered by the sum of ranks for that product, with the product with the lowest sum of ranks (ranked highest) at the top of the table. Each column represents a group of products that are not statistically significantly different (using the Nemenyi multiple comparison test and the significance threshold set by the option ‘level of significance’). If a product belongs to the column group, the sum of ranks for that product is shown in the cell of the table. Use this table to understand which products are different for each attribute. Note that a multiple comparison test needs to adjust for many comparisons and therefore it is possible for the Friedman test to suggest that there are differences between products and the Nemenyi test to not show any differences, particularly when the p-value in the Friedman test is close to the threshold.
  5. General information: This shows any notes or warnings that are relevant to the procedure.

Technical Information

R packages used:
  1. Friedman.test (stats) – for the Friedman test
  2. Posthoc.friedman.nemenyi.test (PMCMR) for the Nemenyi multiple comparison tests. Note that in PMCMRplus the equivalent function is frdAllPairsNemenyiTest.

References 

  1. M. Hollander and D. A. Wolfe (1973), Nonparametric Statistical Methods. New York: John Wiley & Sons. Pages 139--146. (Friedman test).
  2. Nemenyi, P. (1963), Distribution-free Multiple Comparisons. Ph.D. thesis, Princeton University. (Nemenyi test).

    • Related Articles

    • Ranking

      Introduction The aim of ranking as a sensory methodology is to compare a set of products in terms of their perceived intensities on a set of defined sensory attributes. The process is designed to be simple: for each attribute separately in turn, the ...

    • R-Index (ranked data)

      Purpose R-index is a signal detection measure that assesses the degree to which assessors can discriminate between a pair of products for a given attribute. R-index is expressed as a proportion that represents the probability that the one product is ...

    • How Can I Analyse My Data?

      In EyeQuestion there are multiple options to analyse the project data. When you go to the export page in the project you will find a dropdown menu called "Analysis". Auto Reports Via the option for Auto reports EyeQuestion will analyse the data and ...

    • How to Merge Data

      Introduction One of the key features of EyeQuestion is the ability to merge data from multiple projects. This can be especially useful when you have conducted several studies and want to combine the data to gain more comprehensive insights. In this ...

    • How to Handle Data Sensitivity

      Introduction Data sensitivity refers to protecting the personal, medical or financial data by restricting the access of such information to only authorized users. It also includes scrambling (anonymizing) sensitive data after a panellist is removed ...