The pairwise A-Not A design involves two stimuli presented multiple times in a block of trials: a reference stimulus (A) and a comparison stimulus (B). The combined A-Not A design employs A and several levels of B in a block of trials. Both designs were compared using ice tea with five levels of sucrose. Six judges were assessed for sensitivity using the same number of trials in each design; including their overall sensitivity, their average sensitivity across four replicated blocks, and the variability in sensitivity across those blocks. The pairwise design gave higher mean sensitivity, but also higher variability, than the combined design. A secondary analysis considered fewer trials in the combined design, such that sensitivity estimates were based on the same number of trials as for the pairwise design. The pattern for sensitivity within each design did not change, but variability was now comparable. This suggests the combined design yields lower variation for an equivalent expenditure of resource, or that the combined design yields similar levels of variation with reduced resourcing. The trade-off is getting slightly lower estimates of sensitivity. Additionally, the combined design produced sensitivity estimates significantly above chance performance levels when test stimuli were identical. However, the magnitudes of these estimates were small.
