Multitrait-multimethod matrix
The multitrait-multimethod matrix is an approach to examining construct validity developed by Campbell and Fiske. It organizes convergent and discriminant validity evidence for comparison of how a measure relates to other measures.
Definitions and key components
Multiple traits are used in this approach to examine similar or dissimilar traits, as to establish convergent and discriminant validity between traits. Similarly, multiple methods are used in this approach to examine the differential effects caused by method specific variance.There are six major considerations when examining a construct's validity through the MTMM matrix, which are as follows:
- Evaluation of convergent validity - Tests designed to measure the same construct should correlate highly amongst themselves.
- Evaluation of discriminant validity - The construct being measured by a test should not correlate highly with different constructs.
- Trait-method unit- Each task or test used in measuring a construct is considered a trait-method unit; in that the variance contained in the measure is part trait, and part method. Generally, researchers desire low method specific variance and high trait variance.
- Multitrait-multimethod - More than one trait and more than one method must be used to establish discriminant validity and the relative contributions of the trait or method specific variance. This tenet is consistent with the ideas proposed in Platt's concept of Strong inference.
- Truly different methodology - When using multiple methods, one must consider how different the actual measures are. For instance, delivering two self-report measures are not truly different measures; whereas using an interview scale or a psychosomatic reading would be.
- Trait characteristics - Traits should be different enough to be distinct, but similar enough to be worth examining in the MTMM.
Example
| Test | Beck Depression Inv | Hepner Depression Interview | Beck Anxiety Inv | Hepner Anxiety Interview |
| BDI | ||||
| HDIv | Heteromethod-monotrait | - | ||
| BAI | Monomethod-heterotrait | Heteromethod-heterotrait | ||
| HAIv | Heteromethod-heterotrait | Monomethod-heterotrait | Heteromethod-monotrait |
In this example the first row and the first column display the trait being assessed as well as the method of assessing this trait. The term heteromethod indicates that in this cell the correlation between two separate methods is being reported. Monomethod indicates the opposite, in that the same method is being used. Heterotrait indicates that the cell is reporting two supposedly different traits. Monotrait indicates the opposite- that the same trait is being used.
In evaluating an actual matrix one wishes to examine the proportion of variance shared amongst traits and methods as to establish a sense of how much method specific variance is induced by the measurement method, as well as provide a look at how unique the trait is, as compared to another trait.
That is, for example, the trait should matter more than the specific method of measuring. For example, if a person is measured as being highly depressed by one measure, then another type of measure should also indicate that the person is highly depressed. On the other hand, people who appear highly depressed on the Beck Depression Inventory should not necessarily get high anxiety scores on Beck's Anxiety Inventory. Since the inventories were written by the same person, and are similar in style, there might be some correlation, but this similarity in method should not affect the scores much, so the correlations between these measures of different traits should be low.
Analysis
A variety of statistical approaches have been used to analyze the data from the MTMM matrix. The standard method from Campbell and Fiske can be implemented using the MTMM.EXE program available at: https://web.archive.org/web/20160304173400/http://gim.med.ucla.edu/FacultyPages/Hays/utils/ One can also use confirmatory factor analysis due to the complexities in considering all of the data in the matrix. The Sawilowsky I test, however, considers all of the data in the matrix with a distribution-free statistical test for trend.The test is conducted by reducing the heterotrait-heteromethod and heterotrait-monomethod triangles, and the validity and reliability diagonals, into a matrix of four levels. Each level consists of the minimum, median, and maximum value. The null hypothesis is these values are unordered, which is tested against the alternative hypothesis of an increasing ordered trend.
The test statistic is found by counting the number of inversions. The critical value for alpha = 0.05 is 10, and for alpha =.01 is 14.
One of the most used models to analyze MTMM data is the True Score model proposed by Saris and Andrews.
The True Score model can be expressed using the following standardized equations:
1) where:
is the standardized observed variable measured with the ith trait and jth method.
is the reliability coefficient, which is equal to:
is the standardized true score variable
is the random error, which is equal to:
Consequently:
where:
is the reliability
2) where:
is the validity coefficient, which is equal to:
is the standardized latent factor for the ith variable of interest
is the method effect, which is equal to:
is the standardized latent factor for the reaction to the jthmethod
Consequently:
where:
is the validity
3) where:
is the quality coefficient, which is equal to:
Consequently:
where:
is the quality
The assumptions are the following:
* The errors are random, thus the mean of the errors is zero:
* The random errors are uncorrelated with each other:
* The random errors are uncorrelated with the independent variables: , and
* The method factors are assumed to be uncorrelated with one another and with the trait factors:
Typically, the respondent must answer at least three different questions measured using at least three different methods.
This model has been used to estimate the quality of thousands of survey questions, in particular in the frame of the European Social Survey.