The Same Different Test is a discrimination test that is a variation of the paired comparison test. The assessor is presented with two samples and is asked to decide whether these samples are the same or different. There are four possible presentations orders, namely: AA, AB, BA, BB.
The analysis can be performed on data in the ‘long’ and ‘short’ format: for the ‘long’ format it is assumed that the same assessor has evaluated at least one same (AA or BB) and at least one different (AB or BA) pair. For the ‘short’ format, it is assumed that each assessor only has one pair to evaluate and therefore only make one response. The analysis of the ‘short’ format returns the results of a Chi-Squared test for independence and includes the Thurstonian ‘Yardstick’ model (d-prime and tau). For the ‘long’ format, in addition to the short format, the McNemar Chi-squared test is performed.
Alongside formal statistical tests and models, this analysis includes contingency tables and a summary of the presentation order for inspection. This also includes a paired contingency table for the McNemar test.
Note: when using the Simple Difference project template
from EyeQuestion, the data format will automatically be suitable for the Same
Different analysis.
The attributes must be of datatype Binary (where 1 indicates a correct answer
and 0 indicates an incorrect answer). In the dataset the products must be separated
by a “-” and only include one of three cases. For example P01-P02, P01-P01,
P02-P02, but not P02-P01.
If you want to include presentation order information, then another attribute is required. For example an attribute labelled “Q1__info” to correspond to Q1. This should be formatted as text “[A]-[B]” where A, B are either 1 or 2. For example “1-2” means product 1 was presented first then product two; “2-1” means product 2 was presented first then product one; “1-1” means product 1 was presented both times and similarly for “2-2”.
A richer format can also be used. EyeQuestion creates this richer format where the separator is either “-” if the panellist answered “same” or “~” if the panellist answered “different”. For example “2~1” means product 2 was presented first then product 1 and the panellist answered “different”; “1~1” can be read as product 1 being presented both times then the panellist answering “different”. The analysis understands this format and if there is a disagreement between the information attribute and the attribute concerning what the panellist answered then the panellist answer is taken from the attribute.
This is a contingency table with the responses split for all four combinations of first and second product presented. For each attribute this is a table with columns First Product, Second Product, Same and Different. The First/Second Product columns include the names of the products, whereas the Same column contains the total number of same responses for that presentation order and the Different column contains the total number of different responses for that presentation order.
These tables can be useful for spotting potential presentation order bias. For example, if assessors’ responded ‘Same’ more often to an AB pair than a BA pair and the design is balanced then this could indicate presentation order bias (for an unbalanced design you should instead consider the proportion of ‘Same’ responses for each pair).
The presentation order table will only be made for attributes that have a corresponding ‘info’ attribute, e.g., Q1 and Q1__info, because the info attribute includes the presentation order information.
If there are no information attributes then there won’t be any presentation order tables but there will be a relevant message in the warning table. Any info attribute that doesn’t have a corresponding attribute, for example Q2__info exists but Q2 does not, will be removed with a warning included.
This is a contingency table of the same/different responses
in columns against the correct answers in rows. If there are multiple
attributes then there will be a table for each.
This is a Thurstonian model, we’ll call it the ‘yardstick’ model but its name varies the literature, for example tau skimming or differencing. Essentially it models the assessors as using the decision strategy of taking the absolute difference between the two products observed then answering different if this is greater than a threshold, otherwise answering same. This threshold is called tau or