HomeOverviewDownload PalamedesSupporting documentationFrequently asked questionsSubmit comments and questionsSubmit a bug reportNews and updates (last update: June 22, 2016Why "Palamedes"?About usOptions besides PalamedesDemos figure galleryModel Comparison ExamplesConfused about Weibull?Failed Fits?prinsvss2017

Example Model Comparisons

Do you judge significance by judging whether your error bars overlap? Guess what: Your type-I-error rate is about 15% (assuming error in estimates is normally distributed and SEs are equal). Impress your colleagues and do a proper model comparison! Palamedes makes it easy. Here are some examples of what you can do:

Are PFs same or different? (cf. one-way ANOVA):

PAL_PFLR_DEMO_Figure2.jpg

Palamedes allows one to constrain any of the PF's parameters to be equal between any number of conditions. This allows one to test whether parameter values differ between two or more conditions. Simple models can be specified in PAL_PFLR_ModelComparison by setting the constraints on parameters using the words 'unconstrained' (each condition gets it own value independent of other conditions), 'constrained' (a common, shared parameter value is estimated across all conditions), or 'fixed' (a fixed value is used across all conditions). The example above shows a model comparison between a model in which thresholds and slopes are constrained to be equal between two conditions and a model in which they are not (i.e., do the conditions differ with respect to thresholds and/or slope?). This test is performed by PAL_PFLR_Demo in the PalmedesDemos folder. Other tests performed by the demo: Do thresholds differ between conditions? Do slopes differ between conditions? Does the model in which each condition gets it own threshold but the conditions are constrained to have equal slope differ from the saturated model? (i.e., does this model provide adequate fit to the data?)

Testing the significance of the interaction in a factorial design (cf. two-way ANOVA):

CS2.jpg

In the example, conditions varied as a function of two IVs in a factorial design. One IV was Reference Frame Duration (RFD) and had three levels, the other was Token Distribution (TD) and had two levels. Thus, there were six conditions. Of interest was whether the effect of RFD on thresholds (alpha) interacts with the effect of TD. In order to test this, two models were defined: one in which RFD and TD are allowed to interact, another in which they are not. Both models also assumed that slope (beta) was equal across the six conditions and lapse rate (lambda) was equal across the six conditions. The model in which RFD and TD are allowed to interact is indicated by the black lines, the model in which they are not allowed to interact is indicated by the gray lines. A comparison between the models by way of the likelihood ratio test indicates that there is a marginally significant (whatever that means) difference between the models. Akaike's Information Criterion (AIC) says that the model that includes the interaction should be preferred. 

Constraints on thresholds were specified in PAL_PFLR_ModelComparison using contrasts. 'fullerThresholds' was set to [1 1 1 1 1 1; -1 -1 -1 1 1 1; -2 1 1 -2 1 1; 0 -1 1 0 -1 1; 2 -1 -1 -2 1 1; 0 1 -1 0 -1 1] which allows all six thresholds to take on individual values independent of the other thresholds (the same could be accomplished by setting 'fullerThresholds' to 'unconstrained'). The last two rows of the contrast matrix allow for TD and RFD to interact. Thus, to define the lesser model, we leave out the last two rows and set 'lesserThresholds' to [1 1 1 1 1 1; -1 -1 -1 1 1 1; -2 1 1 -2 1 1; 0 -1 1 0 -1 1].

Not familiar with contrasts? It's not as hard as it might look! They are taught in many introductory statistics courses in the context of ANOVA. Judd, McClelland & Ryan, Data analysis: a model comparison approach is a great source on how to use contrasts to compare models.

Data in this example produced by human observer. Taken from: Prins, N. (2008). Correspondence matching in long-range apparent motion precedes featural analysis. Perception, 37, 1022-1036. (analysis in Prins (2008) differs slightly from that shown here).

Testing the nature of the relationship between a quantitative IV and threshold (cf. Trend Analysis):

PAL_PFLR_FourGroup_Demo_Figure3.jpg

Fuller model allows a quadratic trend on thresholds, lesser model does not. Models are specified using contrasts: lesser model: [1 1 1 1; -3 -1 1 3] (intercept, linear trend), fuller model: [1 1 1 1; -3 -1 1 3; 1 -1 -1 1] (intercept, linear trend, quadratic trend). Model Comparison performed by PAL_PFLR_FourGroup_Demo in the PalamedesDemos folder.

Defining non-linear reparametrizations:

MHfit.jpg
MHchi2.jpg

Palamedes allows one to define one's own reparametrizations. Just write some code that implements your reparametrization and Palamedes will fit your model. In the example above the symbols show individual fits for all conditions. Two models of these data were compared. In the fuller of the two the solid symbols were constrained to follow a (three parameter) exponential decay function, while the open symbols were unconstrained. The lesser model was identical to the fuller except that all but the open square symbols were constrained to follow an exponential decay function. Such a comparison answers the question: do the open triangles deviate significantly from the learning curve?

One more non-linear reparametrization example:

CD.jpg

This model comparison is performed by PAL_PFLR_CustomDefine_Demo in the PalamedesDemos folder. It tests whether the learning curves differ between Task A and Task B.