From Theory Extremism and Methodological Arrogance to Method-Theory Synergy

Greenwald’s Arguments

If Platt’s strong inference research strategy leads to efficiency and faster progress in the social sciences (in cognitive and social psychology for Greenwald), then—over time—we should see theory controversies being resolved and accomplishments in theory development being acknowledged and rewarded by the scientific community in form of major scientific awards. However, as Greenwald (2012) demonstrates in his article, this is not the case. In his article, Greenwald lists 13 major theory controversies in cognitive and social psychology, which were investigated following a strong inference approach and, so far, remain unresolved (some controversies also applied in communication contexts).

A comprehensive list of prevailing theory controversies in core areas of communication science is beyond the scope of this brief essay highlighting one particular “hidden gem” paper, but it is not difficult to find examples for theory controversies in communication that prevail despite evidence from strong inference research strategies. The survival of cue theories in deception research (e.g., Levine, 2018) may serve as one example; the lack of any comprehensive strong inference resolution (or even an attempt for a comprehensive test) of social cognitive theory (SCT) in mass communication (Bandura, 2009), one of the most cited theories in communication scholarship, may serve as another example. But wait, social cognitive theory provides an “agentic conceptual framework” and framework theorists usually claim that theoretical frameworks should not be confused with theories that qualify for strong inference research strategies—if you are confused you are not the only one, I am confused as well and to this date have problems with selling the application of conceptual frameworks as theory-driven, strong inference research, but I digress (but will refer back to this point below with a confession).

Regarding Greenwald’s (2012) second point on scientific recognition for theory developments, he carefully analyzed and content coded information on Nobel Prizes over the 21 years before the publication of his paper. He found that 82% of Nobel Prizes in this time frame were awarded for method contributions and only 18% for theory contributions. One could argue that Nobel Prizes are primarily given for STEM scholars who often differ from social scientists in research strategies. However, strong inference research strategies are traditionally easier to pursue in STEM areas than in the social sciences (due to the higher degree of propositions’ specification via mathematical reasoning and thus well-specified mutual exclusion of alternative hypotheses), and by just analyzing Nobel Prizes awarded to psychologists, Greenwald found that the finding remains largely the same. It seems that the resolution of theory controversies and the recognition of theoretical developments in science provide rather weak evidence for Platt’s promise of advancing scientific inquiry via a strong inference research strategy.

Theory Extremism

Despite the aforementioned weak evidence for theory-driven, strong inference based research strategies, I contend that most active communication scholars have experienced a strong push for theoretical contributions and were confronted with reviewers who in general found the questions posed in a journal submission interesting, the methodology applied valid, the results important, and the discussion illuminating with important implications, but rejected the paper on grounds of wondering “where is the theory or the theoretical contribution?” This is what happened in 2009, when I was an assistant professor and tried to publish a paper with my graduate student Patrick Shaw in the International Journal of Gaming and Computer-Mediated Simulations (Weber & Shaw, 2009) and here is my confession that I announced earlier: we produced a study in which we meticulously identified dozens of individual quality perceptions of video games via in-depth interviews and surveys, and then used these quality perceptions to predict video game use in identified player types with, at the time, unseen predictive validity (readers, be aware of my bias in summarizing the study!). The submission was well-received and provided a deliberate deviation of traditional usesand- gratification studies at the time, but two of the three reviewers recommended the rejection of the submission because we failed to provide a theory and hence did not make a theoretical contribution. Oddly enough, but generously, both reviewers recommended that we should “find” a theory for the study. Desperate for publications as an untenured assistant professor, and due to feeling responsible for all the time my wonderful and very hard-working graduate student Patrick (today Director of Production at a major video game company and owner of his own video game company) has invested in this project, I was weak and followed the reviewers’ uncommon advice. I was searching for a theory that has been used in communication research rather frequently and is broad enough to basically explain any phenomenon. I quickly found a good candidate— social cognitive theory of mass communication (Bandura, 2009) mentioned above. All that was changed in the fully written paper was the frontend and a small part of the discussion where I made sure the word theory is mentioned frequently, but without any unfounded claim of pursuing a strong inference strategy. Even our original propositions and predictions remained the same. Upon re-submission, the paper was accepted within one week and without any further demands for revisions. I still like our paper and find it innovative for the time after 15 years have passed, but I am not proud of what I did and find it an ill-guided strategy to get a paper published (and a bad example for graduate students).

While I obviously do not recommend this opportunistic strategy and have not engaged in it since (perhaps a little here and there), I know from honest conversations that many of my fellow scholars are “guilty” of applying similar strategies once in a while. Sometimes, this behavior can even lead to more counter-productive strategies known as Hypothesis Testing after the Results are Known (HARKING; Kerr, 1998), in which original alternative hypotheses are changed and made fit with the findings, and some theory, and which has clearly detrimental consequences for the advancement of science and scholarship (see also Levine, Weber, Hullett, et al., 2008; Levine, Weber, Park, et al., 2008).

I have also experienced that in discussions about the credentials of a candidate for a faculty position, theory-contributions are used to argue for a candidate’s scholarly quality, while methodcontributions are used to question a candidate’s scholarly quality and instead label a candidate as a “method nerd.” I have witnessed that this assessment is often based on the flawed belief that methodological expertise and analytical skills can be easily learned by everyone if one just invests the time and efforts, and thus cannot count as an indicator of scholarship, while theoretical expertise is a product of many years of profound training that cannot be easily obtained. In my experience, it is often just the opposite, because it seems easier to pretend theoretical expertise than methodological/ analytical expertise. I invite fellow senior scholars and major theorists in an area to a friendly competition in which I present and defend (or criticize) an unfamiliar contemporary theory of my colleague’s area for an informed audience after one week preparation time, and in which my colleague presents and defends (or criticizes) an unfamiliar contemporary methodological or analytical approach in my areas of computational communication research and media neuroscience. Although I cannot be sure, I believe I will win this friendly competition most of the time. Many graduate students have told me over the years that their theory classes are by far easier than their methods and statistics classes at the same advanced level, which I believe also makes the point.

In my roles as journal editor and editorial board member I experienced that the quality of research and the profoundness of a scholarly distribution in the communication discipline correlates only weakly with the number of times references to theory are made (which often correlates with the number of different theories mentioned in a paper) and may even be inversely related to quality. Yet, published communication scholarship is saturated with references to theory (Levine & Markowitz, 2023). To be clear and stating the obvious, the number of times the words “theory, theoretical, or theoretical contribution” appear in a paper neither makes a paper more theoretical and important nor answers Platt’s call for a strong inference strategy.

I often see that after research talks in department colloquia or at conferences, presenters are confronted with justifying and defending their research with its relevance for communication theory, despite the relevance and important implications of the findings. All too often the seemingly unquestionable demand for upholding theory above everything else, even above the increase of knowledge, the core mission of scientific inquiry, is made by rather inexperienced graduate students who just have completed their entry-level theory class, or by rather seasoned scholars who obtained their scientific socialization at a time when the field of communication was still in its infancy and scholars were craving for recognition as serious social scientists by scholars of other, more established disciplines.

I label and summarize all these select experiences and observations under the term theory extremism and I assert that theory extremism is equivalent with “strong inference research strategies going astray” and of little help in producing meaningful knowledge. In my view, theory extremism is harmful and not only hinders progress in communication science, but also impedes the further establishment of communication science as a unique and serious discipline. Theory serves specific purposes in scientific inquiries, among them the purpose of definition, explanation, and the derivation of testable (i.e. falsifiable) alternative hypotheses or predictions (for more details see Levine & Markowitz, 2023). Considering advancements in statistical testing towards effect- and equivalence tests (Levine, Weber, Park, et al., 2008; Weber & Popova, 2012), the theoretical specification (i.e., specification independent from the observed data in a study) of potential effect sizes becomes increasingly important for a meaningful contribution. It may be a good exercise to ask a social science-based communication scholar, who put forth a new communication theory, what their theory unequivocally defines, what it specifically explains and predicts (and not predict), and with what accuracy they would expect this prediction to occur (effect sizes). If this scholar provides clear answers to these questions than there is indeed “nothing so practical as a good theory” (Lewin, 1951). If, however, the scholar evades these questions by retreating to notions of general definitional diversity, provides a very broad scope of possible explanations (a “theory of everything,” a meta theory, or a theoretical framework), and is unable to specify even rough estimates of minimum substantial effects or maximum no-effects (Weber & Popova, 2012), then beware and critically evaluate the actual contribution of the proposed theory. This scrutiny will likely result in recognizing that socalled theoretical frameworks as mentioned above are of little use for advancing knowledge via specific research studies. Theoretical frameworks usually do not provide context specific alternative hypotheses that are falsifiable and hypotheses that are not falsifiable do not say or mean anything. Theoretical frameworks are primarily useful for organizing a research area (who has found what in which context and why), writing book chapters, garnering high numbers of citations, and perhaps demonstrating scholarly leadership, which is good for reputation management (see above).

Methodological-Analytical Arrogance

At this point, the reader of this paper may conclude that Greenwald’s hidden gem paper is simply an homage to folks who do not like theory, are not good at developing alternative hypotheses based on logical theoretical propositions, are methods or statistics nerd, etc. These would be wrong conclusions. Again, it is beyond the scope of this brief “hidden gem” essay to provide a complete review of critical papers on methoddriven science, but the reader will quickly find plenty of papers on this issue, especially as “big data,” “data science,” and “computational methods” become increasingly popular in communication research (e.g., Van Atteveldt et al., 2019). However, I would not want to leave the reader with the impression that I have a blind eye when evaluating contributions that are primarily method-driven. I have received most of my initial methodological and analytical training at the Department of Psychology at the University of Technology in Berlin, Germany, in the 1990s, under the guidance of Professor Juergen Bortz (deceased in 2007), who is mostly unknown outside of Europe, but is known in Germany as the “Pope of Methods and Statistics.” I spent 12 years at the department as student, scientific research assistant, and assistant professor (Wissenschaftlicher Assistent) before I continued my career in the United States. I am mentioning this because during my time with Juergen Bortz, I met the best methods-theory synergists and analysts in my career (e.g., Gerd Gigerenzer, who was a close friend of Juergen Bortz), and I have learned so much about the value and limitations of method-driven science, which became essential for my entire career. One of the essential things I have learned was to beware of (and be not impressed by) the methodologically-analytically arrogant. Why?

In a statistics textbook for communication researchers that I wrote together with graduate student Ryan Fuller (Weber & Fuller, 2013), I invited Timothy Levine to write a critical epilogue for this textbook regarding matters of statistical analysis, to address the statistically arrogant, and to address the relationship between statistical analysis, substantive focus, and theory. Levine responded with “Four Statistical Rants” with which—while identifying as a methodtheory synergist with a passion for statistics—I wholeheartedly agree. I recommend reading this epilogue; it will be a good investment of time. His last rant, the conflation of complexity and sophistication is especially important in my view, especially in times of high-paced advancements in big data analytics, computational methods, machine learning approaches, etc. As Levine nicely puts it, complexity is not a virtue in statistical analysis. A well-trained statistician (or data scientist as we now call statisticians) will always recommend what Albert Einstein has presumably said in one of his lectures in 1933: “everything should be made as simple as possible, but not simpler.”(Calaprice, 2010) A few examples of this fundamental principle may be useful: If relationships are primarily linear and specified by a good rationale, there is no need to impress audiences with your skills in machine learning approaches. A simple general or generalized linear model will do and reveal much more about your data than any machine learning technique will do. Often, this is even true when relationships are mildly non-linear. If validated measures of observations are available, and observations are mostly independent or only dependent in time, there is no need for a Latent Class Linear Mixed Model or any other more complex analysis with a shiny new name – often a simple, traditional repeat measures ANOVA or even a simple, very robust t-test will do the trick without any major issues. If a model is well-specified with a few solidly justified relationships, there is no need to define complex path models where arrows go everywhere and to conduct Structural Equation Modelling (SEM). A few straightforward partial correlation analyses will be just fine and test the relationships robustly with plenty of insights, especially if sample sizes are small. In fact, the overand misuse of SEM in communication studies has clear negative consequences on communication scholarship (Seaman & Weber, 2015).

From my experience working as a data scientist in the corporate world I know that senior, highly skilled, and experienced statisticians will not recommend the most general and complex analysis. On the contrary! Only the mediocre statisticians who want to appear smart and are motivated to hide their shortcomings recommend the most complex analytics and, as Germans say, “shoot at little birds with canons.” My students, who know my background in and passion for statistics, and who occasionally want to impress me with their just acquired new analytical skills in machine learning, structural equation modeling, natural language processing tools, etc. (which are all commendable skills) are always surprised when I asked them why they not just apply a t-test and account for some error inflation in proper and straightforward ways (Weber, 2007). However, in order to not be misunderstood, the second part of Einstein’s quote in which he suggests “… as simple as possible, but not simpler” is also important. If indeed one can show that data violate certain assumptions, and that there are specific data properties that must be accounted for (local or global dependencies, non-linearity, latency, nestedness, etc.), then by all means, mastering and applying properly complex statistical analytics is not a sign of methodological-analytical arrogance, it is avoiding methodological-analytical ignorance, which I strongly support.

Methodological-analytical arrogance does not only occur in analytical matters, but also in the process of collecting data. Good examples stem from my own primary research area, media neuroscience. With all due modesty, I was lucky enough to contribute the first media neuroscientific (brain imaging) study that was designed and executed by a communication scholar and published in a communication venue (Weber et al., 2006). When I introduced neuroscientific methodology to communication science in the early 2000s I experienced considerable push-back culminating in one senior and prominent communication scholar demonstrably leaving one of my ICA presentations expressing loudly his discomfort with this “flawed type of research.” While I was surprised by this strong reaction and slightly worried about my prospects to receive tenure as a junior assistant professor in communication, I later understood and even welcomed this very critical response. It taught me that I have to avoid any impression that I have applied sophisticated methods because they are sophisticated. The experience taught me that I will have to reflect deeply on the opportunities and pitfalls of neuroscientific inquiries into communication phenomena and have to justify and explain my methodological rationale to my fellow communication scholars; like I have to justify and explain my arguments at the theoretical level. I also have to educate my fellow communication scholars that actually only a limited set of questions are adequate questions to address with brain imaging methods and that concepts worthy of brain imaging investigations need to be well-specified and predictable on behavioral level. “Fishing expeditions” are especially detrimental endeavors in brain imaging studies and most certainly only produce “the sky is blue” knowledge. More than 20 years have passed since my early presentations at ICA and luckily, times have changed. Media and communication neuroscience is now a widely accepted and recognized research field with many important contributions and discoveries. While it is beyond the scope of this brief essay to introduce into the opportunities and pitfalls of neuroscientific methodologies in communication, many of my fellow communication scholars as well as members of my Media Neuroscience Lab at UC Santa Barbara and I have written pioneering papers about the relevance and challenges of neuroscientific inquiries in communication. As a start, I recommend reading two special issues, one published in the Journal of Media Psychology (Weber, 2015a) and one published in Communication Methods and Measures (Weber, 2015b). Other examples for emerging frontiers of interpersonal communication, mass communication, mass-personal (integrated) perspectives, evolutionary communication, and neuroscience, with critical discussions of opportunities and pitfalls, can be found in an entire ICA Handbook devoted to this topic (Floyd & Weber, 2020).

To sum up, experienced scholars who provide the most useful contributions to the advancement of communication science are methodologically humble, critically evaluate and explain the opportunities and pitfalls of a particular choice of method and analysis, are aware of and able to master new methodological-analytical approaches (and at times confidently ignore the latest new cool procedure, python/R library, or SPSS module), do not conflate analytical complexity with sophistication, and provide profound justifications of why the testing of specified alternative hypotheses or the exploration of a phenomenon requires advanced data collection techniques and analytical procedures.

Method-Theory Synergy

Greenwald’s hidden gem paper goes far beyond theory extremism and methodological-analytical arrogance. Greenwald argues for advanced method-theory synergy as a successful strategy for the advancement of science and backs up his recommendation with observations. For instance, in his analysis of the justifications for Nobel Prizes over 21 years he found that existing theories often played an important role in developing awarded methods, and at the same time, awarded methods produced unanticipated observations that led to previously inconceivable theory. I strongly recommend that the reader of this brief essay processes the many examples given in Greenwald’s hidden gem paper together with Greenwald’s and others earlier and related papers on the issue (e.g., Greenberg et al., 1988; Greenwald et al., 1986).

Specif ic examples in my own area of communication/media neuroscience for the advancement of knowledge in communication via the method-theory synergistic approach can be found in Weber et al. (2018). Among the examples I would like to highlight here is the recently developed brain-as-predictor approach in persuasion studies (Falk et al., 2012; Weber et al., 2015). According to various histories of communication, persuasion research can be considered as one of the intellectual foundations of the social science-based branch of communication research. Much has been accomplished in this prolific research area since the 1930s, but leading persuasion scholars have stated that the field has become stuck in only modest theoretical advancement with multiple competing theories and hypothesis, which are repeatedly tested with small to moderate predictive accuracies hovering around a median effect size of r ≈ 0.13 (for an overview, see O’Keefe, 2016; for effect sizes, see Weber & Popova, 2012). With the introduction of brain imaging methodologies and corresponding analy tics that allow for the meaningful investigation of persuasive messages in realworld public service announcements within a brain-imaging scanner, it became possible to integrate brain responses to persuasive messages with self-report data. This in turn has led to predictive accuracies in persuasion studies that have exceeded traditional accuracies sixfold and higher, even in high-risk or high-involvement large groups (Weber et al., 2015). Based on these vastly improved predictive models it was possible to better identify the cognitive mechanisms underlying successful persuasion in high-risk and other groups, which in turn advanced theory and subsequently has led to further refined methodology and so on. In my view, brain-aspredictor approaches in persuasion studies are a prime example of method-theory synergy and its potential for the advancement of communication science and knowledge production.

Drawing on this one example, and referring back to my point at the beginning of this essay, I am asking the reader who in this method-theory synergy sequence are the “real” or “superior” scholars who have started this success story and have moved the field forward? The theorists? The methodologists? Neither or both would likely be the most appropriate answer. No type of scholar can claim superiority because the synergistic relationship between methodological and theoretical developments has led to progress. We should stop categorizing scholars in theorists and methodologists/analysts and rather evaluate individuals’ scholarship based on the level with which they have increased relevant knowledge and contributed to method-theory synergy in a particular field.

Concluding Recommendations

In conclusion, I invite my fellow communication scholars to reject theory extremism and methodological-analytical arrogance whenever and wherever they occur and to advocate for a method-theory synergy driven approach to scientific inquiry. Research that pretends to provide a theoretical contribution by offering mere theoretical frameworks for alternative hypotheses or research that suggests simple reviews of previous research findings (often from a researcher’s own recent collection of studies) as another “new” theory is as detrimental to scientific progress as research that sells undue analytical complexity as scientific advancement or tries to impress scholars with the technological sophistication of research designs. If this reads too negative, then allow me to conclude with saying that the future does look bright because there is clear indication that method-synergy theory is increasingly a respected perspective and approach. In my role as an associate editor of the journal Computational Communication Research and as founder and active member of the Communication Science and Biology interest group and the Computational Methods division within the International Communication Association (ICA) I have seen a large number of young/junior communication scholars who would be likely bored when reading this essay, because there may be little in it they do not already know and have integrated in their scholarship and work. It makes me happy and gives me a lot of hope for our relatively young discipline to see how promising junior scholars naturally reject theory extremism and methodological-analytical arrogance and how they provide shining examples of method-theory synergy in their work. Way to go!