Archive

By measurement I shall mean the procedures we employ to assign numbers to the values of a variable. And, given the definition of “validity” provided previously, we must acknowledge that we have a gap between the conceptual meaning of a construct and the procedures we use to measure it. For this reason, we have the task of demonstrating that our measures have high validity.

When we use the term “construct” we refer to a dimension on which people, or sometimes other units of analysis (e.g., groups, messages, etc.), differ. On occasion, we may succumb to the error of believing that a dimension of individual difference exists because we can imagine it, failing to realize that constructs reside, not necessarily in our theoretical or conceptual thinking, but in the cognition, affect, and action of those we study. We can make two errors: (1) treating a concept as a construct when it does not have the status of a dimension of individual difference, and (2) failing to identify or acknowledge a construct that does exist. Correcting the former error produces refuse for the construct dump. Correcting the latter error results in a panoply that can inform our understanding of human communication. For the most part we have employed statistical techniques like factor analysis, both the exploratory and confirmatory models, to help us recognize and avoid these errors. Our efforts have not always resulted in success. Consider some examples.

Example 1: Machiavellianism. Christie and Geis’s 1970 volume, Studies in Machiavellianism, introduced the social scientific community to the construct “Machiavellianism,” the development of scales to measure it, and a series of studies providing evidence of the validation of those measures. By Machiavellianism scholars mean the willingness to exhibit callous disregard for others and to exploit and manipulate them in order to get one’s way. Christie and Geis (1970) thought of Machiavellianism as a construct, a single dimension of individual difference.

If Machiavellianism forms a single dimension of individual difference, and Christie and Geis’s commonly employed Mach IV scale provides an acceptable measure of it, then factoring these items would yield a unidimensional solution. As Hunter et al. (1982) showed long ago, however, it does not. Rather, their confirmatory factor analysis (and exploratory factor analysis as well) generated a radical multidimensional solution with four factors: deceit, flattery, immorality, and cynicism emerging, and seven of the 20 items exhibiting a lack of content validity as they fit into none of these four factors nor any other factor.

One might object that the Mach IV might exhibit second order unidimensionality (Hunter & Gerbing, 1982). That is, treating the four factors as four items and factoring them would produce a unidimensional solution, and the four factors, treated as items, would exhibit parallelism, i.e., they would correlate with other constructs in the same way (sign and magnitude). Meeting these conditions would warrant treating the Mach IV as a single dimension. But, the Hunter et al.’s (1982) data showed that the Mach IV failed to exhibit second order unidimensionality as well (see Table 2, p. 1300). Particularly, correlations across the four factors differed on constructs such as self-esteem, locus of control, and religiosity, among others.

These results indicate that subject responses to the Mach IV do not form a single dimension of individual difference. They also indicate the possible existence of four constructs: deceit, flattery, immorality, and cynicism, each of which could exert a powerful impact on various dimensions of human communication behavior. And, the modest reliability of each of these four factors (unreported) indicated that they would benefit from increased attention in the form of constructing additional indicators so as to increase the reliability of each of them.

I have a bibliography of more than 100 articles published prior to 1980 that measured Machiavellianism and assessed its relationship with other variables. Because the Machiavellianism Scale used in these studies does not assess a single dimension of individual difference, none of the reported results inform us about human behavior. Circumstances have not changed. Readers can satisfy themselves with a Google search, and discover that investigators continue to study Machiavellianism and continue to use the Mach IV Scale which they then examine for its association with other variables.

Consider for a moment the implications of that correlation. For example, a correlation between the Machiavellianism Scale and, say, dogmatism (Rokeach, 1960) results from the weighted (by number of indicators) mean of the correlations of dogmatism with the various dimensions contained in the Machiavellianism Scale. Dogmatism might correlate substantially with none of the dimensions, one of them, two of them, three of them, or all four of them. Hunter et al. (1982) report that summed scores on the Mach IV scale correlation .41 with dogmatism. But, that results masks important findings. Specifically, dogmatism correlates -.06 with deceit, .29 with flattery, .04 with immorality, but .84 with cynicism. The only way to reach substantively meaningful conclusions would require examining the correlation with each of the four dimensions separately.

One would likely conclude, and conclude correctly, that most people who think about Machiavellianism as a construct and who measure it with the Mach IV Scale have not read Hunter et al. (1982). A citation search indicates relatively few citations. And, I find that state of affairs perfectly understandable. To read everything written about Machiavellianism prior to doing a study focusing on it might take so long as to make the study obsolete prior to conducting it.

But, I fear the problem goes deeper than that, and that many scholars fail to see the implications of what constitutes a construct for their own research. And, we need go no further than Christie and Geis themselves to see the fear realized. They knew in 1970 that their Mach IV scale did not fit a unidimensional solution, as they report multidimensional factor analytic solutions in Appendix A of their book! Parroting a phrase I have heard or read on numerous occasions, they appear to think of Machiavellianism as a “multi-dimensional construct,” failing to realize that the phrase forms an oxymoron.

Of course, the possibility always exists that Machiavellianism does form a dimension of individual difference, but that the Mach IV or Mach V or other existing measures fail to include the indicators that would assess it. The burden of proof, however, lies with those who endorse such a hypothesis. We must retain our skepticism that Machiavellianism exists and relegate it to the construct dump, resurrecting it only if/when someone produces indictors which tap such a dimension of individual difference.

Example 2: Opinion Leadership. Katz and Lazarsfeld emphasized the concept of opinion leadership in their classic 1955 volume, Personal Influence. And, although Katz later marked the concept for retirement (Katz & Fialkoff, 2017), it remains one to which scholars refer frequently. Both Katz and Lazarsfeld (1955) and Rogers (2003) refer to opinion leaders as having three attributes that distinguish them from non-opinion leaders. Specifically, they characterize opinion leaders as well connected, persuasive, and experts in a domain (mavens). That characterization raises the issue of whether opinion leadership forms a single dimension of individual difference or three dimensions of individual difference (or perhaps neither).

Subsequent factor analytic work (Boster et al., 2011) developed indicators of connection, persuasiveness, and mavenness. They found their data consistent with the predicted three factor solution. Moreover, they found that the internal consistency test of the second order unidimensional model failed, and that the parallelism test of second order unidimensionality failed as well. They also showed that commonly employed measures of opinion leadership (e.g., Flynn et al., 1996) correlated substantially only with the maven scale, so that studies using such measures to identify opinion leaders produce results limited to the expertise of the person exhibiting influence.

This set of three studies and the research program that followed it have implications for the ontology of opinion leadership. Katz and Lazarsfeld’s (1955) and Rogers’s (2003) conceptual thinking, asserts that an opinion leader has high connectivity, high persuasiveness, and high mavenness. Suppose for a moment that we falsely think of these three variables as dichotomies. We can then imagine a 2 × 2 × 2 matrix in which opinion leaders fall into one cell of the matrix, namely the high/high/high cell. To say that opinion leaders, and only opinion leaders, had exceptional influence (the dependent variable, say) would imply a three-way non-additive effect in which those in the high/high/high cell exerted exceptional influence, and those in the other seven cells exerted less, and approximately equal, influence. Conceptually and empirically, however, we gain little by employing the opinion leader concept. Instead, we can profitably rely on the three constructs: connectivity, persuasiveness, and mavenness, to contribute to our theoretical thinking. Affirming Katz and Fialkoff ’s (2017) position we can retire opinion leadership to an assisted living facility in the construct dump.

Example 3: Communication Anxiety. In the first two examples I suggested eliminating concepts that fail to meet the criteria that define a construct. In this example I examine a concept that likely meets the criteria that define a construct, but that needs consolidation.

Communication scholars have long focused on communication anxiety, in its various forms and labels, and continue to do so. In 1959, Clevenger responded to the need to summarize and integrate this burgeoning literature, terming it “stage fright.” Among his conclusions he noted that three ways of measuring the concept had emerged: self-report, observers’ reports, and physiological measures. He also observed modest correlations among these three methods. Currently, self-report measures dominate, but these self-report measures appear in many forms and under many labels. Consider some of them: communication apprehension (McCroskey, 2005), unwillingness to communicate (Burgoon, 1976), social anxiety (Leary, 1983), shyness (Zimbardo, 1977), extraversion (Eysenck, 1973), and need for affiliation (McClelland, 1985).

The manner in which scholars conceptualize these ideas, and the manner in which they measure them, suggests three differences among them. Fear, or anxiety, associated with unpleasant consequences that result from interacting with others would form one point of frequent emphasis, a skills deficit another, and preferring (substantially) one’s own company to that of others the third. Although sorting out causal order would prove a daunting task, we can expect these three attributes to correlate very highly because we can anticipate some causal order among them. For example, feeling anxious about talking with others could lead to a skills deficit and, ultimately, to a preference for one’s own company. If so, we would identify few people who did not have similar scores (either high/high, medium/medium, or low/low and all points between) on all three attributes. Thus, these three attributes would not make a difference when assessing the factor structure, nor would they correlate differently with other constructs. Put differently, they would not make a conceptual or empirical difference, and could be ignored safely. And, although it would require paring some items in existing measures because they lacked validity (e.g., see Levine & McCroskey, 1990), we would expect these many measures to fit a second order unidimensional factor model. We could then affix one label to them and relegate the alternative terms to the construct dump. Such a hypothesis awaits empirical test.

Example 4. Loneliness and Social Support. The first two examples suggest that we might advance theory and research by expanding constructs; abandoning Machiavellianism for four constructs, and replacing opinion leadership with three constructs. The third example took the opposite tack, suggesting the consolidation of the many terms and measures used to refer to communication anxiety. The fourth example continues in this direction, although in a different manner. It suggests the possibility that we might have redundant constructs so construed because they lie at opposite ends of a continuum.

Both loneliness and social support refer to the quality of our attachments with others. The former refers to their absence, the latter their presence. Because loneliness and social support measures predict various facets of physical and emotional well-being, they play an important role for scholars focusing on interpersonal relationships and on health communication.

In a 1986 manuscript Newcomb and Bentler examined the psychometric properties of two social support measures, the Social Resources Scale (see Newcomb & Bentler, 1986, p. 524 for items) and the Socially Supportive Relationships Scale (see Newcomb & Bentler, 1986, p. 524 for items), and two loneliness measures, The Differential Loneliness Scale (Schmidt & Sermat, 1983) and the UCLA Loneliness Scale (Russell et al., 1980). A second order factor analysis indicated that these four measures formed a single unidimensional second order factor. Newcomb and Bentler write,

. . . a single higher-order factor accounts for the correlations among the loneliness and social support constructs. Thus, the current data do not support a conceptual model that gives priority to support as generating decreased loneliness. The current data imply strong equivalence between the constructs. (1986, p. 532)

Judging from subsequent research (e.g., Zhang & Dong, 2022) scholars have either overlooked Newcomb and Bentler’s (1986) findings or, perhaps, found that they failed to hold for other measures of loneliness, social support, or both. Nevertheless, the point remains. We need exhibit caution in order to avoid mistaking an extreme on a continuum for a construct.

More examples would only add redundancy. Nevertheless, I should add two points. First, one can find misconstrued constructs in other content areas in communication science as well as those mentioned in the examples (e.g., organizational communication, Manata & Grubb, 2022). And, second, I have found measures employed commonly in research focusing on personality and social attitude research (such as authoritarianism: Adorno et al., 1950; Altemeyer, 1988 and dogmatism: Rokeach, 1960), particularly likely to have radically multidimensional factor structures, and thus fail to meet the criteria of a construct.

The examples provided illustrate two errors that we can make. We can identify, as a construct, a concept that does not meet the criteria required for a dimension of individual difference, and we can fail to identify a concept that meets those criteria. And, as the examples also illustrate, when we make the first error, we usually make the second.

On a positive note though, the first, second, and fourth examples also illustrate that we can avoid these errors by conducting careful psychometric work. Doing so would involve conducting and publishing measurement studies. And, in studies that do not focus on measurement alone (perhaps those focusing on cause and effect), it would entail engaging in rigorous tests of our measures before examining the effect of our focal construct on other variables, or the effect of other variables on our focal construct.

In the main, we would use confirmatory factor analysis as the primary statistical procedure to test for the dimensionality of our measures, focusing on the internal consistency and parallelism (see Hunter & Gerbing, 1982) of our indicators as a means of identifying those indicators with low validity. And, particularly, we would profit from using the internal consistency and parallelism theorems to generate predicted correlations among all indicators, subtracting these predicted correlations from those obtained, and then examining the magnitude of the resulting residuals, so that we could discard those indicators with large residuals (i.e., those likely to have low validity). Such a procedure, while time consuming, provides a much more fine-grained analysis than only examining commonly employed fit indices.

Popular confirmatory factor analysis algorithms assume linear item characteristic curves (ICCs), relationships between indicators and true score, and linear ICCs dominate communication research. Nevertheless, other types of ICCs do occur, in particular the ogive appearing occasionally. In such cases formulae generating predicted correlations for internal consistency and parallelism theorems differ, but do exist (Keating & Boster, 2019; Tracey, 2000).

The examples discussed involved self-report measures. Notably, the same points hold with respect to measures obtained from observers, such as coding group interaction, measures of artifacts, physiological measures, and the like. And, they also hold with respect to experimental inductions. Our inductions, like our measures, may lack validity, i.e., they may fail to induce what they intend to and nothing else. Or, put another way, they may induce confounds. Perhaps they deserve their own induction dump. And, as with our measures, careful pretesting and conducting pilot studies would contribute substantially to diminishing this problem.

Having the ability to operationalize, or operationize, would make our task easier. The problems with establishing that our measures and inductions have high validity or not would vanish. We would have no misidentified constructs. But, we do, and we also do know how to attack the problem. We must demand of ourselves and other scholars that we demonstrate that our purported constructs meet the criteria of a dimension of individual differences, and assign those that do not to the construct (or induction) dump.