Example Model Comparisons
Do you judge significance by judging whether your error bars overlap? Guess what: Your type-I-error rate is
about 15% (assuming error in estimates is normally distributed and SEs are equal). Impress your colleagues and do a proper
model comparison! Palamedes makes it easy. Here are some examples of what you can do:
Are PFs same or different? (cf. one-way ANOVA):
Palamedes allows one to constrain any of the PF's parameters to be equal between any number of conditions.
This allows one to test whether parameter values differ between two or more conditions. Simple models can be specified in
PAL_PFLR_ModelComparison by setting the constraints on parameters using the words 'unconstrained' (each condition gets it
own value independent of other conditions), 'constrained' (a common, shared parameter value is estimated across all conditions),
or 'fixed' (a fixed value is used across all conditions). The example above shows a model comparison between a model
in which thresholds and slopes are constrained to be equal between two conditions and a model in which they are not (i.e.,
do the conditions differ with respect to thresholds and/or slope?). This test is performed by PAL_PFLR_Demo in the PalmedesDemos
folder. Other tests performed by the demo: Do thresholds differ between conditions? Do slopes differ between conditions? Does the
model in which each condition gets it own threshold but the conditions are constrained to have equal slope differ from the
saturated model? (i.e., does this model provide adequate fit to the data?)
Testing the significance of the interaction in a factorial design (cf. two-way ANOVA):
In the example, conditions varied as a function of two IVs in a factorial design. One IV was Reference Frame Duration
(RFD) and had three levels, the other was Token Distribution (TD) and had two levels. Thus, there were six conditions. Of
interest was whether the effect of RFD on thresholds (alpha) interacts with the effect of TD. In order to test this, two
models were defined: one in which RFD and TD are allowed to interact, another in which they are not. Both models also assumed
that slope (beta) was equal across the six conditions and lapse rate (lambda) was equal across the six conditions. The model
in which RFD and TD are allowed to interact is indicated by the black lines, the model in which they are not allowed to interact
is indicated by the gray lines. A comparison between the models by way of the likelihood ratio test indicates that there is
a marginally significant (whatever that means) difference between the models. Akaike's Information Criterion (AIC) says
that the model that includes the interaction should be preferred.
Constraints on thresholds were specified
in PAL_PFLR_ModelComparison using contrasts. 'fullerThresholds' was set to [1 1 1 1 1 1; -1 -1 -1 1 1 1; -2 1 1 -2 1 1; 0
-1 1 0 -1 1; 2 -1 -1 -2 1 1; 0 1 -1 0 -1 1] which allows all six thresholds to take on individual values independent of the
other thresholds (the same could be accomplished by setting 'fullerThresholds' to 'unconstrained'). The last two rows of the
contrast matrix allow for TD and RFD to interact. Thus, to define the lesser model, we leave out the last two rows and
set 'lesserThresholds' to [1 1 1 1 1 1; -1 -1 -1 1 1 1; -2 1 1 -2 1 1; 0 -1 1 0 -1 1].
Not familiar with contrasts?
It's not as hard as it might look! They are taught in many introductory statistics courses in the context of ANOVA. Judd,
McClelland & Ryan, Data analysis: a model comparison approach is a great source on how to use contrasts to compare models.
in this example produced by human observer. Taken from: Prins, N. (2008). Correspondence matching
in long-range apparent motion precedes featural analysis. Perception, 37, 1022-1036. (analysis in Prins (2008) differs
slightly from that shown here).
Testing the nature of the relationship between a quantitative IV and threshold (cf. Trend Analysis):
Fuller model allows a quadratic trend on thresholds, lesser model does not. Models are specified using
contrasts: lesser model: [1 1 1 1; -3 -1 1 3] (intercept, linear trend), fuller model: [1 1 1 1; -3 -1 1 3; 1 -1 -1 1] (intercept,
linear trend, quadratic trend). Model Comparison performed by PAL_PFLR_FourGroup_Demo in the PalamedesDemos folder.
Defining non-linear reparametrizations:
Palamedes allows one to define one's own reparametrizations. Just write some code that implements your
reparametrization and Palamedes will fit your model. In the example above the symbols show individual
fits for all conditions. Two models of these data were compared. In the fuller of the two the
solid symbols were constrained to follow a (three parameter) exponential decay function, while the open symbols
were unconstrained. The lesser model was identical to the fuller except that
all but the open square symbols were constrained to follow an exponential decay
function. Such a comparison answers the question: do the open triangles deviate significantly
from the learning curve?
One more non-linear
This model comparison is performed by PAL_PFLR_CustomDefine_Demo in the PalamedesDemos folder. It tests
whether the learning curves differ between Task A and Task B.