Design Matters in Web Credibility Assessment: Interactive Design as a Social Validation Tool for Online Health Information Seekers

Factors Affecting Credibility Assessment of Online Health Information

Among many theoretical frameworks for web credibility, Fogg’s (2003) framework that classifies various web elements into three categories—operator, content, and design—has been widely adopted by both theoretical (Choi & Stvilia, 2015; Sun et al., 2019) and empirical (Chang et al., 2021; Choi, 2020; Wu et al., 2020) investigations to identify the sources of crediblity cues and heuristics in the web context. The first category, operator, involves the perceived ability and willingness of the source to provide high-quality health information online. The source can be either the personal or organizational agent who runs the health-related website or the author who is responsible for the presented content (e.g., an author of an answer on a social question-and-answer site). For example, domain type may be an important factor affecting perceived web credibility. Specifically, websites run by government agencies (.gov) or educational institutions (.edu) tend to be perceived as more credible than those owned by pharmaceutical companies (.com) due to the belief that for-profit companies prioritize financial gains over public interest (Choi, 2013; Liu & Shi, 2021). On social media, recognizable experts or celebrities enjoy higher credibility than an anonymous source via perceived authority (Cooley & Parks-Yancy, 2019; Djafarova & Trofimenko, 2019).

The second category, content, considers the perceived quality of the presented health information, assessed based on cues and heuristics, allowing users to infer such attributes as accuracy, semantic and structural completeness, or recency. For instance, the presence of grammatical or typological errors can serve as a cue decreasing the credibility of the content or website (Choi, 2020; Jiaying et al., 2021). The presence of proper citations and the use of clear and succinct language can hint at the credibility of the presented health information (Cunningham & Johnson, 2016; Scantlebury et al., 2017). In addition, one common finding in the literature on online health information credibility is that people often rely on what others think and do when assessing the credibility of health information on the web (Borah & Xiao, 2018; Lederman et al., 2014; Rueger et al., 2021). The influences of others on the credibility assessment of online health information are supported by well-established heuristics in the credibility literature, such as the bandwagon heuristic (i.e., people perceive information as credible if others do; Sundar, 2008), the endorsement heuristic (i.e., people perceive the information as credible when it is recommended by a source they trust; Hilligoss & Rieh, 2008), and the consistency heuristic (i.e., people perceive the information as credible when the same information is found from different sources; Metzger et al., 2010).

Design constitutes the last category of web credibility elements. Design is not limited to the aesthetic quality (e.g., professional look) of the website. Functional elements account for much of this category, encompassing the ease of understanding the hierarchy of the content (i.e., information design), maintenance of the basic interface to remain operant (i.e., technical design), and accessibility of additional information through the interface (i.e., interaction design; for a comprehensive review, see Fogg, 2003). Empirical findings largely indicate that health-related websites with a more comprehensible information architecture and interactive or easier-to-use interface have higher web credibility (Johnson et al., 2015; Machackova & Smahel, 2018; Rowley et al., 2015; Song et al., 2021).

In addition to such intrinsic attributes of web resources, user characteristics (e.g., demographic background, domain knowledge, attitudes toward information seeking and processing, and technology proficiency) and contextual factors (e.g., urgency, goals, or motivations in the moment) are known to affect credibility assessments of web-based health information (Chen et al., 2018; Choi, 2020; Eastin, 2001; Kim & Syn, 2016; Wang et al., 2021; Yu & Han, 2018). Therefore, it is important to consider these variable factors when examining people’s credibility assessments of web-based health information (Sbaffi & Rowley, 2017).

Outcomes of Credibility Assessment of Online Health Information

Research has indicated that the outcome of web credibility assessments is multidimensional, encompassing cognitive, attitudinal, and behavioral domains. For example, Stvilia et al. (2007, 2009) reported that credibility was among the most highly ranked attributes, along with accuracy and reliability, contributing to the perceived quality of health information on the web and the degree of usefulness for dealing with a given health issue. Rowley et al. (2015) found that credibility was the most influential factor affecting users’ trust in online health information among eight factors examined in the study, such as brand, content, ease of use, recommendation, style, usefulness, and verification.

Empirical studies reported that the perceived credibility of a health-related website affects users’ online health information behavior (for a review and meta-analysis, see Wang et al., 2021). Specifically, credibility is known to influence users’ selection of a website over others in search engine results (Haas & Unkel, 2017), intention to revisit a health-related website (Hong, 2006), and adaptation of health information for self or others (Choi, 2020; Freeman & Spyridakis, 2009; Hu & Sundar, 2010). Marketing research shows that credibility is associated with consumer attitudes toward e-commerce websites or products being sold online and purchase intention (Cooley & Parks-Yancy, 2019; Kim et al., 2015).

Study Objectives

Past research has explored various web elements related to the characteristics of the operator (or author), content, and design of the site as potential cues and heuristics influencing users’ credibility assessments of online health information (Sbaffi & Rowley, 2017; Sun et al., 2019). However, what remains relatively less examined is which elements (a) represent the construct (i.e., web credibility) better than others in the context of online health information and (b) better establish the users’ perceptions, attitudes, and behaviors toward the website and the health information it presents. In other words, we aimed to provide a synthetized view of what makes users perceive a website as a credible source of health information and how web credibility assessment, in turn, influences their health information behavior on the web. The significance of the present study involves two aspects. First, the study examined relative impacts of different web elements (i.e., operator, content, and design) on online health information seekers’ web credibility assessments and associated outcomes (e.g., perceptual, attitudinal, and behavioral outcomes) based on well-established theoretical frameworks in the credibility literature (Fogg, 2003; Hovland et al., 1953). Second, the study focused on health as the target domain, in which web credibility matters, in the context of addressing various health-related concerns in users’ everyday lives, as opposed to in controlled experimental settings. This research goal led to two research questions (RQs):

• RQ1: What types of elements affect the perceived credibility of a health-related website?
• RQ2: What are the relative impacts of the types of web elements on the user’s cognitive, attitudinal, and behavioral reactions to health-related websites?

Participants

College students (N = 141) attending a university in the United States completed a cross-sectional online survey in exchange for extra credit. Participants were recruited from information studies and communication classes. All participants were 18 years old or older, spoke English, and had sought for health information online at least once during the preceding 6 months. They read and signed the informed consent before completing the survey. The survey instrument received institutional review board approval.

Participants were 26 years old on average (SD = 7.5 years). About 54% were women and 73% were Caucasian. In terms of education, 61.7% were undergraduates, 35.5% had a 2-year associate or 4-year bachelor’s degree, and 2.8% had a master’s degree. Mean daily internet use was 7.8 hours (SD = 3.7). All but one participant reported having more than 5 years of experience using the internet.

Survey Questionnaire Development

We developed an online questionnaire focusing on health as an important subject domain for web credibility assessment in four stages. In the first stage, we searched three online databases—ACM Digital Library, IEEE Xplore, and Web of Science—to establish a comprehensive list of elements known to affect users’ web credibility assessments in the literature. We identified previous studies that (a) were published in peer-reviewed journals or conference proceedings, (b) were written in English, and (c) used data collection methods that directly involved human subjects, such as surveys, interviews, and experiments. This third criterion was used to select user studies investigating the target audience’s cognitive and behavioral characteristics in the process of assessing credibility of web-based resources, as opposed to secondary analysis of existing data (e.g., tweets, questions and answers posted on social platforms), which often aim to develop a predictive model of popular content (e.g., retweets, best answers) using machine learning techniques. After removing articles that did not meet the inclusion criteria, 105 relevant articles remained, from which we extracted 407 elements. It should be noted that we adopted the search flow suggested by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines (Liberati et al., 2009), which identifies identification, screening, eligibility, and inclusion phases, to facilitate the database search process. However, we did not intend to conduct a systematic review or meta-analysis using the full PRISMA process.

In the second stage, we examined individual elements to group them into more general, higher-level categories. For example, similar items from different sources—“The site content offers precise and detailed information with source references and dates” (Robins & Holmes, 2008) and “The site has articles containing citations and references” (Fogg et al., 2003)—were combined: “The site post has sources, references, and dates.” As a result, 65 unique markers remained.

In the third stage, each credibility marker was phrased as a statement following the format: “When you look for health information regarding your health concerns or questions, to what extent do you consider each of the following statements important in your credibility assessment of health-related websites?” The final wordings of the items included in the questionnaire are shown in the Appendix. In the online questionnaire, the order of the credibility questions was randomized to remove the potential order effect.

In the fourth stage, we pretested the questionnaire with eight graduate research assistants. They reviewed the survey to cross-check the readability of the items and overall survey length. They confirmed that all questions were easy to understand and the overall length was adequate (Choi et al., 2020). Table 1 shows the design of the questionnaire.

Table 1.

Design of the Survey Questionnaire

We note that the survey was designed to explore what affects users’ credibility assessments of health-related websites that they use in their everyday lives as opposed to having them evaluate a particular website preselected or manipulated by the researcher in a controlled environment. To establish a commonality regarding what participants would think of as a health concern or health-related website, participants received an identical definition denoting what health information meant in the current study: “Health information refers to a wide range of topics including infectious diseases, injuries, mental and physical health, substance use, nutrition, or anything else related to health.” Then, they listed up to three health websites they would visit to address a health concern that fit the definition before taking the survey. The same approach was used in Fogg et al.’s (2001) seminal survey study on web credibility, in which participants answered questions by drawing on their cumulative experience using the web rather than evaluating certain types of websites given by the researcher.

Measurements

Data Analysis Plan

Sources of online health information mentioned by respondents were analyzed using frequency and percentages. An exploratory factor analysis (EFA) was conducted using 65 items regarding credibility cues to identify the latent structure of perceptions of web credibility. Factors were extracted using maximum likelihood estimation and varimax rotated for interpretation. A hierarchical linear regression followed to observe the impacts of the types of web credibility markers identified by the EFA from context by adding potential covariates into the equation. Specifically, participant demographics (i.e., age, sex, education, annual income) constituted the first block. The second block included measures of internet use (i.e., everyday internet use at home, at workplace, in public spaces, and via mobile devices). The third featured measures of involvement with health information seeking (i.e., level of interest in health topics, frequency of seeking health information online). The final two models included types of web credibility markers identified by the EFA. This same hierarchical model repeatedly predicted each of the five outcome measures in case the pattern of effects changed. All statistical analyses were conducted using R version 4.0.3.

Sources of Online Health Information

Among 141 participants, 136 participants (96.5%) provided at least one health source that they had used to obtain health information online. In total, 67 sources were mentioned 296 times (about two sites per participant). WebMD was most popular (26.7%), followed by the Mayo Clinic (13.9%) and the Centers for Disease Control and Prevention (7.1%). Some participants mentioned websites not specific to health issues such as Google (10.5%), YouTube (2%), and Wikipedia (2%). Table 2 presents the 10 most frequently mentioned sources of online health information.

Table 2.

Ten Most Frequently Mentioned Sources of Health Information Online

Exploratory Factor Analysis

EFA was used to examine RQ1, exploring the latent structure of 65 elements presumed to represent web credibility. Factors were extracted using maximum likelihood estimation and then varimax rotated for interpretation. Seventeen factors had an eigenvalue exceeding 1.00, and the scree plot showed no clear point where the diminishing return began. Alternatively, Horn’s parallel analysis (Horn, 1965; Parsons, 2007) was adopted to make the decision. This approach creates a random dataset equivalent with the one studied in terms of the number of variables and observations, establishes a correlation matrix employing a Monte Carlo simulation, and puts the data into the same EFA. To the extent that the data were created randomly, the eigenvalues stemming from such data should provide little information to determine the coherence of the factor structure. Therefore, a factor solution with an eigenvalue below that of the same solution from the parallel analysis-created random data should be rejected for adding no meaningful information. Fit indexes also considered the 4-factor solution as a sufficient summary of the data, explaining 40.9% of the variance in the dataset, χ²(1,826, N = 141) = 2,454.03, p < .001; RMSEA = .049, 90% CI [.045, .055]. Results from a simulation study (de Winter et al., 2009) indicate that the current sample size was sufficient to produce reliable outcomes under various conditions of EFA.

Table 3 specifies the items representing their respective factors, item statistics, and standardized Cronbach’s alpha for each factor. Cross-loaded items were identified and removed from further analysis when the absolute distance between the first and second largest loadings fell below .20 (Stamper & Masterson, 2002). Remaining items that loaded onto a common construct by more than .50 were considered unidimensional and averaged to create composite indexes. Internal consistencies were acceptable, ranging between .78 and .88.

Table 3.

Results of Exploratory Factor Analysis

Current results partially match Fogg’s (2003) theoretical framework, and the four extracted factors can be labeled content, interaction design, information design, and source. Content items appear to measure the extent to which the presented health information is recent, unbiased, and evidence based. Interaction design items represent the site’s affordance of user interaction with the constructed environment (e.g., customizability) and other users. Information design items seem to involve the efficiency of information organization and the navigation system of the site. Source-related items assess perceived reputation and unbiasedness of the source, particularly as oriented toward the public interest.

Hierarchical Linear Regression

To examine RQ2, analysis determined the relative impacts of the four components of web credibility on the outcomes of interest in three domains—attitude, belief, and behavior. Specifically, a hierarchical linear model was established with five blocks. Participant demographics (i.e., age, sex, education, annual income) constituted the first block. The second block included measures of internet use. The third featured measures of involvement with health information seeking. Finally, the fourth and fifth blocks included the content-related (i.e., content and source) and design-related (i.e., information design and interaction design) web credibility items, respectively. This same hierarchical model was used to predict each outcome measures (Table 4). Analysis showed that the multicollinearity remains relatively low, with generalized variance inflation factors ranging between 1.16 and 2.01 for the full model involving all predictors. This demonstrates a relatively minor distortion in the vector space of the predictors and therefore, the estimated parameters can be considered reliable (see Fox & Monette, 1992). The normality of the residuals and the level of homoscedasticity were also checked based on a visual inspection. The residuals approximating a normal curve in a QQ plot and no apparent pattern of residuals across the fitted values indicate no substantial violation of the major assumptions of regression analysis. The visuals used for the diagnostics can be obtained from the corresponding author.

Table 4.

Results of Hierarchical Regression Analysis

Table 4.

Results of Hierarchical Regression Analysis (Continued)

Willingness to Return to Site

Content-related attributes solely constituted a significant and powerful predictor of willingness to return, β = .31, p < .001; ΔR² = .12; F(2, 100) = 10.65, p < .001. Adding the source- and design-related cues did little to improve the model. Education had a significant effect on willingness to return to the site across the five models.

Willingness to Recommend

Individual level of interest in health topics predicted willingness to recommend most powerfully, β = .42, p < .001; ΔR² = .14; F(2, 102) = 16.02, p < .001. Its impact, however, tended to decline—β = .26, p = .003 and β = .23, p = .009, respectively—when the model expanded to incorporate content quality, β = .35, p < .001, and interactivity by design, β = .24, p = .002. Results indicate both features (i.e., content quality, interactivity by design) helped explain unique variance in the outcome measure, ΔR² = .10, F(2, 100) = 9.93, p < .001 and ΔR² = .04, F(2, 98) = 5.09, p = .008, respectively. Current results demonstrate that people who visit websites to attain health information tend to consider not only content-related attributes but also communication attributes—that is, interactivity with the content, source, and other users—when deciding whether to recommend the website to others who may need similar health information.

It also should be noted that neither source credibility (β = -.04, p = .632 in Model 4 and β = -.06, p = .429 in Model 5) nor design elements related to the organization of information or navigability of the site (β = -.05, p = .546) contributed to predicting the dependent measure. The trivial effect of source credibility, alongside the significant effects of individuals’ interest in health topics (β = .23, p = .009), content credibility (β = .30, p < .001), and interactivity (β = .24, p = .002), seems to represent the particularity of the current research context. Before deciding whether to recommend a health website to others, involved users who cannot independently comprehend the information seem to mobilize social resources instead of blindly counting on the source expertise as advertised. Also, efficient information design seems to be a commonality, yet does not necessarily improve user perceptions.

Willingness to Follow Advice

A similar pattern of results followed. Individual level of interest in health topics continued to dominate, albeit at a smaller magnitude, β = .27, p = .003; ΔR² = .09; F(2, 102) = 6.07, p = .003. The frequency of seeking health information online also inversely predicted the willingness to follow advice, β = -.18, p = .030, implying that people who frequently visit health related websites tend to become skeptical about the validity of the content provided.

As in the previous analysis, perceived credibility of content (β = .20, p = .039) improved the model, ΔR² = .08, F(2, 100) = 5.47, p = .006. Its magnitude, however, deteriorated (β = .16, p = .109) with the addition of interactivity by design, β = .26, p = .003, which further improved the explanatory power of the model, ΔR² = .07, F(2, 98) = 4.73, p = .011. Interactivity, potentially as a means of social validation, reemerged as an important criterion for evaluating health-related websites.

Trust the Health Information

Similarly, level of interest in health topics (β = .28, p < .001) and frequency of visiting websites for health information (β = -.18, p = .018) jointly contributed to model improvement, ΔR² = .10, F(2, 102) = 7.21, p = .001. Content (β = .26, p = .006 in Model 4 and β = .22, p = .018 in Model 5) and interaction design (β = .17, p = .044) further enhanced the predictive power of the model, ΔR² = .06, F(2, 100) = 4.16, p = .018 and ΔR² = .03, F(2, 98) = 2.14, p = .123, respectively. The impact of the interactivity consideration, however, remained relatively low compared to willingness to recommend or follow advice.

Find the Site Favorable

Level of interest in health topics weakly but significantly predicted attitudes toward the website, β = .15, p = .041. The third block, however, failed to explain additional variance in the dependent measure, ΔR² = .04, F(2, 102) = 2.74, p = .069. Among the four components of web credibility, content alone made a substantive contribution to improving the model, β = .30; ΔR² = .11; F(2, 100) = 8.56, p < .001. Its magnitude dropped only slightly, β = .27, p < .001, when adding design elements to the model. The design block (β = .05, p = .472 for usability; β = .07, p = .328 for interactivity) failed to deliver additional power to the model, ΔR² = .01, F(2, 98) = 1.06, p = .350. The current pattern of data roughly replicates the results found with willingness to return or trust.

Principal Results

The participants in the present study were asked to list a few websites they used to find health-related information in their everyday lives before rating each cue or heuristic in terms of its importance in judging the credibility of the health information found on the sites. Therefore, the participants’ ratings could have been provided in the context of how they would normally evaluate the importance of credibility cues and heuristics on favored websites and validate their judgments using interactive and social cues. Based on the three modes of college students’ credibility judgments—predictive, evaluative, and validation judgments (Rieh & Hilligoss, 2008)—our findings can be understood in the context of the last two modes, excluding the first mode, in which users predict a site’s credibility as a criterion to determine whether they should pursue further interaction.

Current findings show that web credibility in the context of health information seeking is a construct featuring four components playing different roles depending on which outcome of web credibility is at stake. Overall, the current factor solution (Table 3) captured all three major types of web credibility cues and heuristics—operator (or author), content, and design (Fogg, 2003). Interaction design elements, which represent the ability to interact with the site or share information with others, had particularly strong effects on behavioral outcomes, such as willingness to follow advice and recommend the website to others (Table 4). We interpret this finding as indicating that individuals who cannot yet need to process online advice regarding their health problems are unlikely to count on only the perceived quality of the content or perceived expertise of the source. Especially when having to act on the information, such users might want to mobilize a secondary source (e.g., communicating with others) to validate the information, and such motivations could be achieved via interaction design. In particular, such processes appear to require at least three forms of validation when considering the items composing the interaction design factor identified in the present study (Table 3): (a) interaction with the content provider (e.g., requesting additional information or answers to an inquiry); (b) interaction with other users, or so-called “social validation” (e.g., sharing information, personal experiences, and opinions); and (c) interaction with the content (e.g., manipulating the web environment to make the information more accessible).

Such interactive design features help promote the open and collaborative nature of the current web environment, enabling users to interact with other users (e.g., via liking, commenting) or the site operator or moderator (e.g., via flagging, using the “contact us” feature) to reach a consensus or collective knowledge on a given topic (Chun & Lee, 2022). In the context of web credibility assessment, the fact that a site provides such interactive features may influence the user’s perception of the site as having the intent or ability, or both, to facilitate an open and collaborative process for providing high-quality information. Therefore, the interactive design features may serve as means for users to exchange ideas and opinions regarding a topic.

Social validation has been identified as an important factor influencing web credibility assessments in the literature (Borah & Xiao, 2018; Jucks & Thon, 2017; Lederman et al., 2014; Rueger et al., 2021). In particular, social validation happens commonly among young adults in cyberspace. Cues implying collective endorsement and popularity of the given information (e.g., likes, ratings, or comments from fellow users) prompt the bandwagon heuristic (Hilligoss & Rieh, 2008; Sundar, 2008), which helps reduce uneasiness about acting on the information. Studies have indicated that social validation can be as powerful as source expertise among young adults or adolescents when assessing health information online (Jucks & Thon, 2017; Rueger et al., 2021).

Of note, recent studies comparing the relative impacts of credibility cues and heuristics have shown that design constitutes the least powerful predictor of web credibility (e.g., Chang et al., 2021; Wu et al., 2020). Such results, however, may not be conclusive for at least two reasons. The first involves external validity—specifically, the observations were made in a controlled experiment (Chang et al., 2021); in topic domains where participants’ involvement remains relatively low, such as weather, shopping, and education (Al-Omar, 2016; Djafarova & Trofimenko, 2019; Wu et al., 2020); or among participant groups with relatively lower information technology proficiency (Choi, 2020). Thus, it is questionable whether the same pattern would be expected among experienced online users seeking information to address actual health concerns. The second involves conceptual unclarity—design, especially concerning interactivity, remains either limited to features enabling communication with the source (Al-Omar, 2016; Chang et al., 2021) or conflated with related yet disparate constructs, such as navigability and source credentials (Djafarova & Trofimenko, 2019). The construct may have different effects if further distilled or situated in a more relevant context.

Content-related cues or heuristics (e.g., unbiasedness, recency) had a substantial impact across all outcomes (Table 4). This finding is in line with well-established research evidence in the literature—see reviews on content-related criteria for assessing the credibility or quality of online health information (Sbaffi & Rowley, 2017; Sun et al., 2019).

Credibility cues or heuristics involving source characteristics or the efficiency of information design (e.g., organization of information, navigation features) had little impact across all outcomes. Presumably, these features are expected by most visitors to health-related websites today. That is, most websites on health issues now present quality content by reputed sources—at least in the eyes of lay users—in a user-friendly manner. To the extent that such features are perceived as a required minimum, their presence is unlikely to boost the credibility of the website. However, their absence may result in a substantial decline in credibility. It is well established in health domains that websites failing to satisfy user expectations tend to lose credibility (Metzger et al., 2010).

Limitations and Future Studies

This study was limited by its content domain and population and therefore, current findings may explain the attitudes and behaviors of college students seeking health information online but may not be applicable to other populations visiting websites for different purposes. Results from future replication studies would demonstrate the stability of the current findings.

The current study solely attempted to identify a comprehensive list of elements constituting perceived web credibility. Its factor structure was not fully explored and remains unknown. Future studies should conduct a confirmatory factor analysis to determine the structure among the factors identified by this investigation. Consistent results would improve the validity of the current conclusions.