Quantitative Content Analysis: Its Use in Technical Communication

Researcher’s note (June 2026). Laura Palmer and I wrote this in 2010 to make a methodological case the field kept skipping: that content analysis is most powerful when it is genuinely quantitative—predefined categories, trained raters, interrater reliability, and inferential statistics—rather than a relabeled thematic reading. We illustrated it with two case studies, one conceptual and one relational, to show how the same method scales from a manual tally to commercial software. I keep returning to this paper in my current work on AI in writing and assessment, because the questions it raises about coding schemes, mutual exclusivity, and reliability between raters are exactly the questions we now face when we ask machines to read and score text at scale.

Abstract

Quantitative content analysis can enrich research in technical communication by identifying the frequency of thematic or rhetorical patterns and then exploring their relationship through inferential statistics. Over the last decade, the field has published few content analyses, and several of these applications have been qualitative, diluting the method’s inherent rigor. This paper describes the versatility of quantitative content analysis and offers a broader application for its use in the field. This discussion frames two original case studies that illustrate the design variability that content analysis offers researchers.

Index Terms—Content analysis, quantitative research, research methods, technical communication.

Introduction

In “Empiricism Is Not a Four-letter Word,” Charney described an imbalance of research methods used in composition studies and related fields such as technical communication [1]. She advocated the promotion of complex and quantitative empirical methods that improved future inquiry and tested existing hypotheses. A more recent analysis of academics and industry professionals in technical communication offered similar conclusions concerning the state of research in the field [2]. This study documented how leading technical communicators assessed the state of research in their field. Charney, among others, suggested that the field’s researchers can neglect the front-end preparation of their projects and analyze data with insufficient rigor. Others have noted that the field may be conducting too many textual analyses and critiques when alternative approaches could produce more meaningful results. This paper demonstrates how quantitative content analysis can address these concerns by offering technical communicators an empirical framework for investigating a collection of texts.

Content analysis assesses words, phrases, or in-text relationships. We define content analysis here as “a research technique for making replicable and valid inferences from texts (and other meaningful matter) in the contexts of their use” [3, p. 18]. Texts can be broadly classified to include printed matter, images, maps, art, sounds, signs, or symbols. Similarly, we propose content analysis be applied quantitatively, which includes identifying meaning through valid measurement rules and making relational inferences with statistical methods [4]. Quantitative content analysis can prove to be a more powerful method than surveys and interviews because of its unobtrusive nature and its lack of reliance on subjective perceptions [3]. Researchers can also use the method to investigate a communicative situation that no longer exists or cannot be accessed. Finally, quantitative approaches enable a broader investigation of texts over an extended period of time; these goals are often logistically impossible for researchers investigating the same phenomena qualitatively [4].

While contemporary content analysis remains empirically grounded, it is often applied qualitatively and relies on the researcher to determine the thematic inferences [5]. This inductive approach violates the traditional, scientific guidelines of the method. Over the last decade, technical communication journals published an average of one content analysis per year. These studies demonstrated the versatility of the method, but the qualitative applications of several of these studies arguably diminish the overall benefits of the method [6]–[9]. Qualitative analyses do not employ the use of raters or mutually exclusive categories, reducing the validity and the reliability inherent in the method’s design. Categorizing a single unit of data under multiple categories can lead to false inferences and can impede accurate statistical analyses.

In this paper, we focus discussion on quantitative content analysis as a method well suited for technical communication. We first review the fundamentals of the methodological design, distinguishing between its qualitative and quantitative approaches. This discussion motivates new ways the field can use the method and frames two original case studies that demonstrate the historical and longitudinal aspects of content analysis. We conclude by discussing the manual and computer-based applications of the method.

Fundamentals of Content Analysis

The following text serves as an overview for conducting a content analysis. Researchers interested in content analysis should consult additional sources and acknowledge the varying approaches to using the method. A major misconception of content analysis is that the term applies to any examination of texts [10]. In fact, when reviewing literature for this paper, we found that technical communication journals classified several articles as a “content analysis” when researchers instead applied an alternative type of analysis (e.g., rhetorical, narrative, discourse). Whether it is applied quantitatively or qualitatively, content analysis is empirical and, therefore, includes a series of fixed characteristics that enhance its validity and reliability.

The general framework of any content analysis begins with conceptualizing the idea for investigation. Researchers begin by identifying a corpus of texts that will explore a research question or hypothesis. A research question should acknowledge how the texts elicit social action and consider who the texts were written for, why they were written, and how they have been used. For example, analyses in technical communication have investigated the values and commitment issues of family-owned-business employees [11] as well as how trust is developed in online courses [12]. Researchers have also conducted their analyses with a variety of texts, including questionnaires [11], websites [13]–[16], transcripts [12], and journal articles [7]–[9].

Identifying a Sample

Content analysts must also consider scope, sampling, reliability, and the appropriate statistical analyses for their project. The rise of electronic media now offers an abundance of texts to evaluate, and the longitudinal nature of content analysis allows researchers to explore these texts over an extended time period. To manage the size of the text corpus, researchers must devise a careful, replicable selection process. For example, one study explored how the leading journals in technical communication addressed the topics of women and feminism in the field [8]. The researcher identified articles beginning in 1989, which marked the publication of an award-winning gender study on collaborative workplace writing. Further, the researcher only examined full-length articles in prominent technical communication journals. This scope produced a data corpus of 1,072 articles that was then narrowed down to 40 articles based on the application of relevant keywords (e.g., female, gender, sexist). The rigor applied to this sample selection offers a specific and replicable framework for researchers interested in reproducing or extending the study.

Collecting and Categorizing Data

Once researchers identify their sample, they must determine how data will be categorized. This step marks the first substantial difference in the qualitative and quantitative applications of content analysis. In its qualitative form, content analysts evaluate the text collection for emergent and recurring themes. This application does not begin with fixed, mutually exclusive categories; instead, researchers refine their categories as themes emerge, a process that can continue through data analysis [8]. Final data analysis offers a rich discussion of textual patterns, comparable to ethnography. In exploring the role of collaboration in the workplace and classroom, one content analysis in technical communication identified seven themes from its qualitative approach [7]. This researcher acknowledged that the emergent categories depend on that individual’s experiences and interpretations. Since mutual exclusivity is not a criterion for qualitative content analysis, this researcher classified the articles on collaboration under multiple themes.

In quantitative research, content analysts evaluate texts for predefined terms or phrases and use inferential statistics to make conclusions about their presence. A recent study in business communication used quantitative content analysis to determine the balance of positive to negative language in a collection of earnings press releases as well as the effect of release length on the investors’ reactions to the content [17]. This type of quantitative content analysis could be classified as conceptual because it identified the presence and frequencies of specific words or concepts. This application reflects the traditional use of the method, but contemporary content analyses can extend this approach by examining the relationship among the words or concepts [5]. Another business communication study used a relational approach to examine the content and function of the email signatures used in organizations [18]. While the data analysis quantified the email signatures by organizational type and the different pieces of information the signatures included, inferential statistics investigated the possible relationships between these variables. For example, a chi-square test found that the fields of medicine and education were more likely to include their education and title, respectively, suggesting these fields used email signatures to convey authority.

Developing a Coding Scheme and Training Raters

Creating a coding scheme for content categories is a paramount step in quantitative content analysis. The coding scheme includes a code book that defines and illustrates all of the variables being evaluated. Likewise, raters record their scores independently on a coding sheet, which includes all of the categories under investigation. Raters commonly use one coding sheet for each text they evaluate [10].

Using the code book to train raters further establishes the validity of the analysis; the collaborative process allows researchers to determine the clarity or interpretability of the categories while also assessing the between-rater agreement. Low interrater reliability is often attributed to evaluators responding to elements outside the code book’s language [19]. Neuendorf recommends that researchers remove themselves from coding their own data and instead train at least two raters to independently assess the texts [10]. Interrater reliability should be assessed during the training phase and during the coding of the actual study. A representative sample, a minimum of 10%, is sufficient for assessing this reliability [4], [10]. The minimum for interrater agreement varies between 70% [20] and 80% [21]; however, little consensus exists for calculating the agreement. The technical communication content analyses that did assess reliability used Cohen’s kappa, percent agreement, as well as Perrault and Leigh’s index of reliability [13], [14], [16], [22].

Analyzing Data

How researchers approach their data analysis and discussion depends on the methodological design. The qualitative approach allows researchers to make deeper inferences on the variables’ explicit and implicit meanings, but conclusions concerning the relationship of variables may be difficult to achieve since a single variable can be classified into multiple categories. This inductive approach, however, does not allow for interrater reliability and produces observations rather than statistically measured data. In fact, two noted flaws that destroy the utility of a content analysis are faulty definitions and the lack of mutual exclusivity among categories [23]. On the other hand, the quantitative approach only evaluates the explicit features of a text [5], but the statistical analyses can extend beyond counting the presence frequencies of words and offer insight into how variables relate. The earlier-cited business communication study, for example, used statistics to show relationships between various organizational fields and the specific content in their email signatures. Inferential statistics suggest these relationships but determine the likelihood that the relationships are due to chance. By design, qualitative applications cannot provide this rigor.

Broader Applications of Content Analysis

The handful of content analyses published in technical communication journals in the last decade, combined with some of these journals’ misclassification of “content analysis,” suggests a need for further exploring how the method can be used. Similarly, the predominance of qualitative content analyses in the field suggests the need for expanding the method’s utility.

For technical communication, quantitative content analysis provides a rigorous method for examining a broad range of topics that impacts the field as an academic discipline and industry as well as influencing organizational communication in general. Quantitative content analysis, for example, can be used to:

• evaluate transcripts from usability studies to identify common mental models held by users;
• reveal differences in communication strategies for intercultural audiences;
• assess bias in publications produced by government at the federal, state, and local levels;
• codify the images used in marketing communications;
• locate intent in the annual reports produced by corporations;
• identify and deconstruct the rhetorical strategies from successfully funded proposals;
• assess how the rhetorical qualities and characteristics of environmental-impact statements have evolved since the signing of the Environmental Policy Act of 1969.

All of these studies can employ the rigor inherent in quantitative content analysis but also advance the method beyond its traditional use of counting frequencies of words into a more ethnomethodological investigation. The following case studies illustrate these approaches and demonstrate the variability that content analysts have in methodological design.

Using Conceptual Content Analysis: Trends in Technical Communication Journals

This first case study is a conceptual content analysis that measures knowledge claims in technical communication journals and tabulates the explicit language of visual rhetoric and design in four technical communication journals. Evaluating the frequency of design-based language was seen as a way to measure visual communication in the field. A steady increase in the complexity of the language, particularly in reference to theories of the visual in publications, could be an indicator of the profession’s enhanced interest in visual communication.

Background and Design

Quantitative content analysis provides a method to isolate trends and make the directionality of these trends explicit. Smith and Thompson use content analysis to isolate frequently occurring themes with respect to feminism and the workplace styles of men and women [9]. Their research points to the power of disciplinary “mythos” [9, p. 468]; that is, scholarly articles lend themselves to more than the transmission of pure fact. They can convey, by conjecture, assumption, or unsubstantiated “intuition” [9, p. 467], concepts that misrepresent the factual elements of the literature. Allen states, “We are disciplined by our disciplines”; that is, disciplines provide the parameters for shaping “shared values and beliefs of a community” [24]. Therefore, quantitative content analysis provides a systematic means to examine texts, see directional indicators of a trend, and support or refute disciplinary knowledge making.

This case study assessed four journals for the years 1998–2002. Journal of Business and Technical Communication (JBTC), Journal of Technical Writing and Communication (JTWC), Technical Communication (TC), and Technical Communication Quarterly (TCQ) were reviewed for the overall frequency of visual rhetoric articles. In addition, articles then identified as being relevant to visual rhetoric and design were scrutinized using content analysis to quantify the language used to inform and convey information about the visual. This date range was selected because it represented the five most recent years of publication at the time.

The research questions governing this case study were twofold. First, this study sought to answer the question about the prevalence of visual design/rhetoric articles in major technical communication journals. It was hypothesized that while visual rhetoric was receiving considerable attention as a scholarly pursuit, that attention was not the result of the focus it received in academic or professional publications. Second, this study sought to examine the type of explicit language found in the journal articles published in JBTC, JTWC, TC, and TCQ. Specifically, it was postulated that few articles referred to the theories underpinning visual rhetoric and design. While there may be many individual words referencing design or visual rhetoric, references to theories would be scant.

These four journals were selected because of their focus on publishing more articles on theories—such as visual rhetoric—than on practice. Also, these journals are very specific to the discipline of technical communication and knowledge making within that community. As markers of the discipline’s knowledge creation and perpetuation, JBTC, JTWC, TC, and TCQ are peer-reviewed or vetted by a board comprising professionals and academics. They are also published by academic or professional organizations.

Code Book

The content analysis study required two methods of discriminating the relevant content dimensions of the journal articles. To distinguish articles directly relating to visual rhetoric from the other significant content areas of the journals, a preliminary coding sheet was developed. This coding sheet was a condensation of the submission areas listed in the masthead of JTWC. Based on this journal’s guidelines, the following eight content areas formed a basic categorization method: teaching, technology, research, documentation, communication, theory, written rhetoric, and visual rhetoric. Coding required a review of each article’s title, abstract, and, if necessary, introductory paragraphs. From this information, articles were coded under the appropriate rubric, and the length of the article, expressed as the number of pages, was noted.

Articles identified as relevant to visual rhetoric were identified for a second level of coding. Each article was reviewed to determine if it had been coded correctly into the visual rhetoric category based on the title, abstract, and introductory paragraphs. Of the 25 articles originally identified for further content analysis, 7 were, upon review, not considered relevant to visual rhetoric and/or design.

The remaining 18 articles were reviewed and coded per the words/phrases on the secondary coding sheet. The terms for this sheet were derived from the index entries found in Schriver’s Dynamics in Document Design [25] and supplemented with index entries from Hopkins’s After Modern Art [26]. Terms from the index were chosen from words/phrases that reflected the nature of design, whether as theoretical dimensions or as descriptors of visual rhetoric practice. In total, 61 terms/phrases, either in the singular or plural, were used to determine the level of theory and the occurrence of design language in the technical communication journal articles. The occurrence of each specific theory or word/phrase was tallied for the 18 journal articles under consideration. Excluded from these tallies were words/phrases in headings, marginalia, callouts, tables, or figures.

Interrater reliability, as measured by Holsti’s formula, assessed the reliability of the principle rater. This statistic measures the agreement between two raters, which is a recommended approach for content analysis [10]. After 7 articles were returned to other categories and the remaining 18 articles coded, one week lapsed before a coding reliability check was conducted. This one-week break would be sufficient to reduce memory effects and produce accurate recoding. In total, four randomly selected articles were recoded, and the reliability coefficient was calculated at 0.91.

Results, Discussion, and Conclusion

One hypothesis in this study related to how visual rhetoric appeared to dominate the field of technical communication. As visual rhetoric receives a great deal of attention in the academy, it could be construed that the scholarly publications dedicate many pages to the topic. The preliminary coding analysis, however, indicated just the contrary: Visual rhetoric had received little attention over the years 1998–2002. Of the 7,067 pages reviewed from four journals for this research, only 361 pages, or 5%, of the content area dealt with visual rhetoric issues. A chi-square analysis of the page counts for the content area of each journal produced a statistically significant finding where p = 0.001. Thus, the differences in the number of pages dedicated to each content area in each journal are statistically beyond the range of sheer probability. Quantitative content analysis also revealed that articles on visual rhetoric were the shortest, with an average of 14 pages per article. This analysis also demonstrated that TC published 12 of the 18 articles on visual rhetoric/design.

A second hypothesis explored in this research considered the type of language found in the journal articles published in JBTC, JTWC, TC, and TCQ; specifically, it was postulated at the outset that very few articles referred to the theories underpinning visual rhetoric. Terms and phrases like “Gestalt theory,” “Modernist design,” or “color theory” would occur infrequently, as the field of technical communication was not yet mature in its consideration and definition of visual rhetoric.

The results supported the hypothesis that individual terms and not theories would occur most frequently. Table I shows an excerpt from the tally sheet for the top-10 coded terms.

TABLE I FREQUENCY OF TOP-10 CODED TERMS

Term or Phrase	Frequency
design	161
visual(s)	156
images	106
visual design	103
illustration(s)	100
graphics	62
photograph(s)	53
aesthetics, document design	34
visual composition	29
visual rhetoric	23

The results from this research illustrate several important facts with respect to how frequently visual rhetoric articles occur in technical communication scholarly journals and the overall nature of the language, as it conveys not only knowledge claims, but the status of the visual rhetoric and design theories in a traditionally writing-based discipline. Clearly, the preliminary grouping of articles into thematic areas demonstrates that visual rhetoric/design is an underrepresented topic in scholarly publications about technical communication. While the visual does appear to receive an inordinate amount of attention, this attention is not replicated in the discipline’s journals.

Much of the language surrounding discussions of the visual still positions visuals as occupying a subordinate position to text. The word “visual” as conjoined to terms such as “aids,” “devices,” “enhancements,” “appeal,” and “features” demonstrates that many of those creating knowledge about the visual still relegate it to a supporting role for the written word. This relegation to a supporting role works against the idea of creating a unified theory of visual rhetoric, especially within the discipline of technical communication.

Using Relational Content Analysis: Sheep and Goat Raisers’ Magazine (1920–1971)

This second case study is a relational content analysis that charts the advertising trends of an agricultural magazine in an attempt to learn more about its targeted audience. Specifically, this study explored the rhetorical appeals used to market health-related products to readers of Sheep and Goat Raisers’ Magazine. The coding sample for this investigation consisted of 369 health-related advertisements published across 51 issues of the publication from 1920 to 1971.

Background and Design

Farming remains an underexplored topic in technical communication scholarship, yet farmers need specialized, technical knowledge to turn a profit and contribute to their economies. Sheep and goat farmers mix and administer their own parasite and deworming treatment, and their successful execution of this technical knowledge directly correlates with their yearly profits.

In the early 1920s, sheep and goat farming was a relatively new enterprise that sparked interest in entrepreneurs from Scotland to New York. These farmers immigrated mainly to West Texas because the vast quantity of open prairie provided an agriculturally rich terrain for sheep and goats [27]. However, little information on these early farmers exists, particularly regarding which rhetorical appeals were the most successful in conveying the importance of the health and wellness of livestock. Sheep and Goat Raisers’ Magazine was established in 1920 to document these farmers’ tacit knowledge and promote the business aspect of the industry. The magazine was the official publication of the Texas Sheep and Goat Raisers’ Association, which was established to protect the wool and mohair industries with united action against the theft of stock and was committed to disseminating research about the health and wellness of livestock [28]. Sheep and Goat Raisers’ Magazine quickly became the leading publication on its subject matter and, by 1922, had attained a large circulation in Texas, Oklahoma, New Mexico, and Arizona [27]. Even today, Texas remains the top sheep- and goat-producing state, recently yielding 36 million pounds of wool [29] and producing 90% of the American mohair clip [30].

This case study analyzed advertisements in Sheep and Goat Raisers’ Magazine from 1920, when the magazine first went into publication, to 1971, when the magazine changed its name and content focus. A random, consecutive sample was used to narrow the data corpus to 51 issues. This sampling approach has proven effective in the content analyses of extensive and continuing publications such as newspapers and magazines [4], [31], [32]. One issue was coded from each year of the monthly publication. The sample for this study begins with the August 1920 issue (the first issue) and then advances by one month and one year (e.g., September 1921, October 1922, November 1923). From these 51 issues, 369 advertisements were coded based on their health and wellness emphasis. No duplicate advertisements were coded.

Code Book

The code book was divided into 12 categories, including the identification of the advertisement’s dominant textual and visual rhetorical appeals. Criteria for these categories were identified from advertising, technical marketing, and design texts that were also compared against a marketing text from the 1960s, one of the decades represented in this study [25], [33]–[36]. Table II provides an excerpt from the code book. This particular category evaluated the overall rhetorical appeal (logos, ethos, or pathos) of each advertisement. For example, a rater would assign a 2 if the ad’s language primarily targeted readers through testimonials and/or the company’s reputation. Authentic examples from the text collection were used to illustrate each option in every category.

The code book was created through SurveyMonkey.com, an online survey generator that offers free services. Two raters independently input their numeric data through the survey generator’s data-entry mode.

Interrater reliability was assessed by using Cohen’s kappa. Cohen’s kappa is a more rigorous means of assessing reliability than using other statistics such as an exact percent agreement because it accounts for chance when measuring the level of agreement between two raters [37]. For this study, the kappa test identified an overall agreement of 79.8%, indicating an acceptable level of consistency between raters.

TABLE II OVERALL RHETORICAL APPEAL CATEGORY IN THE CODE BOOK

Code	Appeal	Definition	Examples
1	Logos	• Compares its value to another similar product. • Emphasizes the economy of the product, either in application and (or) in its price. • Emphasizes safety features (to user or animal) or environmental concerns.	• [Thibenzole is] 10 times more potent than Phenothiazine…is easy to mix and keep in suspension. This assures a minimum of waste, a uniform dose throughout the drenching period … will not result in wool or mohair discoloration… nonirritating to human skin… It can be safely used in young or old animals.
2	Ethos	• Reiterates the company’s name or longevity. • Offers testimonials from a manufacturer representative or customers.	• Cooper’s Dipping Powder is recommended by most agricultural experiment stations and leading wool growers’ associations… Enough sold annually to dip 300,000,000 sheep. • The experience of Mr. H. P Sherman, Alfred Station, NY, a well-known breeder of Rambouillets, is of real value to other sheepmen.
3	Pathos	• Relies on guilt or scare tactics to convince the reader to purchase the product. • Insults and (or) makes the reader/farmer look inexperienced, uneducated, or careless.	• WARNING-DANGER. If your sheep show these symptoms—Look Out! • Eenie, meenie, minie, but wolves cannot be choosers! • Why don’t our boss feed Wymix? We keep sick-thin-scrawny, when we could be thrifty and growing. Our neighbor feeds it and raises prize winners.

Results, Discussion, and Conclusion

Overall (see Table III), the text of the advertisements favored ethos-based appeals (48%) compared to logos-based (41%) or pathos-based appeals (11%). Ethos-based appeals commonly used the product or company’s name as the primary selling point (e.g., “Black Leaf 50 Recommended for Killing Blowflies”).

The primary graphic used in the advertisements (see Table III) favored logos-based appeals (50%) compared to ethos-based (38%) and pathos-based appeals (12%). Logos-based appeals often included instructional graphics, such as a depiction of a farmer administering treatment to his livestock. In contrast, pathos-based appeals were primarily used in wolf-proof fence advertisements and often included some variation of a wolf licking its lips at or stalking a flock of unprotected sheep.

TABLE III OVERALL RHETORICAL APPEAL OF EACH ADVERTISEMENT’S PRIMARY TEXTUAL AND VISUAL ELEMENT

Decade	Logos Appeals — Text	Logos Appeals — Visual	Ethos Appeals — Text	Ethos Appeals — Visual	Pathos Appeals — Text	Pathos Appeals — Visual
1920s (N = 30)	10	8	17	18	3	4
1930s (N = 26)	8	8	15	5	3	13
1940s (N = 112)	58	56	52	48	2	8
1950s (N = 105)	47	56	52	37	6	12
1960s (N = 96)*	29	58	42	31	25	7
Total Mean	0.411	0.504	0.482	0.377	0.106	0.119

*The sample coded from 1970 was included with 1960s decade (N = 9 ads).

Interestingly, advertisements in the 1920s appeared to favor a combination of ethos-based appeals with pathos-based graphics. For example, an October 1922 advertisement entitled “Page Wolf-Proof Texas Fences” uses the product name to establish ethos with the reader, but the graphic depicts a wolf standing over a recently slaughtered lamb [38]. A chi-square test revealed a statistical significance (X² = 12.73, p = 0.001) between the ethos text/pathos graphic relationship in the 1920s and 1930s.

However, the early 1930s marked a dramatic increase in logos-based textual and graphic appeals that lasted until the end of this study’s timeframe in 1971. For example, a July 1931 advertisement for Colorado Wolf Proof Fence emphasized its selling point by appealing to readers’ logic. The ad’s content emphasized the importance of the fence pieces that were engineered to cross gullies or other low-terrain areas. The primary graphic demonstrated how the product is designed to weigh down fence pieces that dip into a gully, using instructional arrows to depict installation. This rhetorical shift occurred during the Great Depression, when the market for wool and mohair was low to nonexistent, declining 63% between 1929 and 1932 [28]. Rather than capitalizing on readers’ dwindling profits by preying on their emotions, these advertisers appeared to make a significant effort to appeal to readers’ logic instead. This particular ad seemed to purposely avoid appealing to readers with scare tactics or other emotionally charged themes. Other than mentioning “Wolf” in the product’s name, the advertisement never mentions wolf attacks but instead appeals to the readers’ economical sense by stressing that purchasing the fence provides “ASSURANCE against profit leaks” [39, p. 317].

This increase in logos-based appeals also suggested an increased use of quantification in the advertisement’s language. In addition to providing more instructional graphics, advertisements began providing readers instructions on administering medications and other treatments. An October 1946 advertisement for Black Leaf 40, for example, gave detailed instructions for administering their stomach worm treatment:

Dissolve 22 1/4 ounces of Copper Sulphate in 1 gallon of soft water. Dilute at the rate of one gallon of the whole solution to nine gallons of water, making 10 gallons of 1 3/4% solution. [40, p. 23]

By 1971, the use of statistics and exact measurements in advertisements had grown 37.2% since the first issue, suggesting the audience considered themselves independent agents responsible for the success of their business and the health of their animals.

Overall, the findings suggest a high level of sophistication in this magazine’s readership, preferring products whose advertisements use ethos- or logos-based appeals. Similarly, a 37% increase in advertisements that integrated measurements, statistics, and application instructions further suggested the readers’ high skill level and their desire for appeals to their logic rather than their emotions. The statistically dramatic shift away from ethos-based text and pathos-based graphics suggests that preying on readers’ emotions was an unsuccessful rhetorical appeal, especially during the Great Depression, when such an approach could have arguably been successful. Further research could code the copy for buzz words that might have enticed readers to buy the advertised product. Comparing these findings against the trends of other industries might also provide a more complete audience profile of the magazine’s readers.

Coding a Content Analysis

As a method for technical communication, quantitative content analysis provides researchers with alternatives for coding texts for conceptual or relational studies. Ranging from manual tallies of terms to using software programs to map relationships or complex themes, content analysis can be easily adapted to the size and budget of a researcher’s project. Here, the three most common methods for quantitative content analysis are examined.

Manual Coding

The simplest method for conducting a conceptual content analysis is to manually identify and tally selected terms or phrases in texts. As discussed earlier, a proper quantitative content analysis requires a researcher to identify terms or themes prior to coding any texts. This predefinition of language guides the development of the study’s hypotheses that will be refuted or confirmed with statistical analyses.

When the research questions are posed, the terms or themes to be coded are developed. These terms or themes, called “categories” in content analysis, require iterative refinement in pilot testing and interrater reliability testing. Selecting single words or word strings makes manual coding relatively easy. However, when a researcher is looking to code themes, manual coding can be more difficult. For example, if a researcher were examining kairos in a set of political documents, what concepts or phrases would measure an opportune moment? In addition, determining how coding would be mutually exclusive would be another concern important to the methodological design. This phase of development will take more time than expected.

The next phase of a manual method for quantitative content analysis is developing the coding sheet. This sheet is the first place where tallies for terms and themes are recorded. As quantitative content analysis relies on the computation of, at the very least, descriptive statistics, the sheet’s layout should match the input order for Excel, SPSS, or another statistical program. While the sheet needs room for tallies and other notes, the physical design of the sheet should replicate, for example, the order of terms and row/column headers. A best practice would involve laying out the coding sheet in Excel. This program can calculate basic statistical functions and, if Excel is insufficient, the data are easily imported into SPSS and others.

Coding, as a process, should be methodical. Marking one term at a time per article provides the most consistent results. With manual coding, several colors of highlighter pens and multiple copies of the document are necessary, especially when hired raters are used. Verifying coding in manual methods is critical; thus, the study design should make that verification easy if any postcoding questions about interrater reliability occur. If there are more terms than colors of highlighters, one good suggestion is to use multiple copies of the same document and only code as many terms per document as there are highlighters. Once an article is fully coded, tally the terms and post them to the coding sheet and into Excel or SPSS for further analysis.

The advantages of manual coding are many. This method is low-tech and easy to implement. In conceptual studies where a small number of terms or themes are being coded, manual coding is an excellent choice. This method can also be combined with the simple computer-assisted (discussed next) means to verify rater reliability. A notable problem with manual coding is the time it takes, especially if there are many terms or themes to code. Interrater reliability can also be problematic with the manual method.

Simple Computer-Assisted Method

Like the manual method, the simple computer-assisted method requires a researcher to develop the terms and/or themes for coding and create a coding sheet that will simplify data entry. The simple computer-assisted method differs in its use of software tools to find and tally categories present in a conceptual study. The speed at which this method can locate terms or phrases also makes it an excellent starting point for larger, relational studies.

In Microsoft Word 2007, the Reading Highlight function in the Find dialog box facilitates quantitative content analysis by making it easy to locate and tally individual words or word strings. If text can be pasted or imported into Word from a webpage or another file format, Word makes a very useful tool for content analysis. The key to using Word effectively is defining exactly what the program should highlight or omit. For example, the term “the” can result in the software selecting words like “their” or “theme.” Selecting “Whole word only” ensures erroneous terms are not included. Fig. 1 shows the Find function with Reading Highlight selected for “Find whole words only.” In the Find window, Word provides a total of all instances of the word selected to search. This number can then be transferred to the coding sheet.

[Figure 1: Sample of Microsoft Word’s Reading Highlight function for content analysis. — image to be added.]

Documents in Adobe Acrobat PDF format can also be searched using the program’s Find function. Like Word, Adobe can limit its searches to whole words only; also, it provides a return count of the number of words or terms the program finds. (See Fig. 2.) After searching each term or phrase, the tally is transferred to the coding sheet.

[Figure 2: Sample of Adobe Acrobat’s Find function. — image to be added.]

The computer-assisted model for content analysis trumps the manual method by reducing the coding time. In addition, this model eliminates interrater reliability problems and can reduce or eliminate the need for hired raters.

Disadvantages relate to a simple document program’s inability to judge context. A word or word string may not belong in the data; therefore, it may be necessary to review each hit the program returns. Document programs also cannot indentify themes, so again, a manual review may be necessary. Regardless, the time saved on manual coding can make this method a very worthwhile option.

Content Analysis Software

For large-scale relational studies, commercial content analysis software can be ideal. These programs can increase the depth of a study without the slowness of manual coding. With the ability to develop complex themes, word associations, and contextual settings, these programs can produce rich and detailed data. The data delivered can be counts or a variety of graphical representations, depending on the program.

There are many programs for content analysis studies. As of January 2008, 37 commercial software packages were available [41]. However, costs and complexities can deter a researcher from using software. Most content analysis programs are based on dictionaries. Lowe notes that anyone interested in using content analysis software should ask the following questions:

(1) How complex is the analysis? (2) Can it run languages other than English? (3) Is the code base or dictionary proprietary? (4) Is there an established user base? (5) Does it run only on Windows? [42, p. 17]

Any researcher who uses a content analysis program needs to factor in time to learn the program, test its operations, or customize the dictionary. In some cases, these steps may take longer than using a simpler method. Lowe also cautions researchers that the accessibility of the program’s code may influence the choice of one program over another [42, p. 4]. Where code is proprietary, viewing how the program performs complex analyses is not available [42, p. 4]. Thus, details of the analytical algorithms are unknown to the researcher. Lowe refers to some programs as “development environments”; this model vests the researcher with the task of working directly with the program at a code level [42, p. 11].

Regardless of the study and the needs of the researcher, one of the aforementioned methods for quantitative content analysis can provide a way to examine texts and produce outcomes supported by statistical measures. The manual method is excellent for coding macrolevel features of a document such as words in headings or image placement on the page. Where terms or phrases in paragraphs are defined for coding, the simple computer-assisted method maximizes a researcher’s time via its speed and accuracy. Finally, for large-scale studies that explore themes, semantic linkages, and other complex interrelationships in texts, commercial software applications are the best.

Conclusion

In studies where traditional rhetorical or textual analyses are conducted, content analysis provides a methodological approach that adds rigor and design variability. Conceptual studies, as defined by measuring terms and phrases, can be conducted easily and result in clear numerical outcomes. Relational studies, where complex themes and their interrelationships are assessed, are more strongly supported with quantitative data. These measurements—as statistical representations—give quantitative content analysis a distinct advantage over other approaches. Hypotheses can be confirmed or rejected, and research questions can be answered more definitively.

As we have shown, quantitative content analysis is adaptable to the study size and the needs of the researcher. In studies examining changes over time, quantitative content analysis—from manual to fully computer-run—provides a degree of flexibility typical of qualitative designs. This quantitative application pairs well with qualitative techniques and adds a valuable method of triangulation to these studies.

As the field of technical communication continues to stabilize as an academic discipline, strengthening the research produced is paramount. By using a diverse arsenal of methods and using best practices in research design, technical communication will produce work that is rich and methodologically sound.

Acknowledgments

The authors would like to thank Liz Watts, the IEEE reviewers, and editorial staff for their comments on earlier drafts.

References

1. D. Charney, “Empiricism is not a four-letter word,” Coll. Comp. Commun., vol. 47, no. 4, pp. 567–593, 1996.
2. A. M. Blakeslee and R. Spilka, “The state of research in technical communication,” Tech. Commun. Quart., vol. 13, no. 1, pp. 73–92, 2004.
3. K. Krippendorff, Content Analysis: An Introduction to Its Methodology, 2nd ed. Thousand Oaks, CA: Sage, 2004.
4. D. Riffe, S. Lacy, and F. G. Fico, Analyzing Media Messages: Using Quantitative Content Analysis in Research. Mahwah, NJ: Lawrence Erlbaum Associates, 1998.
5. T. Huckin, “Content analysis: What texts talk about,” in What Writing Does and How It Does It: An Introduction to Analyzing Texts and Textual Practices, C. Bazerman and P. Prior, Eds. Mahwah, NJ: Lawrence Erlbaum Associates, 2004, pp. 13–32.
6. C. Abbott and P. Eubanks, “How academics and practitioners evaluate technical texts: A focus group study,” J. Bus. Tech. Commun., vol. 19, no. 2, pp. 171–218, 2005.
7. I. Thompson, “Collaboration in technical communication: A qualitative content analysis of journal articles, 1990–1999,” IEEE Trans. Prof. Commun., vol. 44, no. 3, pp. 161–173, Sep., 2001.
8. I. Thompson, “Women and feminism in technical communication: A qualitative content analysis of journal articles published in 1989 through 1997,” J. Bus. Tech. Commun., vol. 13, no. 2, pp. 154–178, 1999.
9. E. O. Smith and I. Thompson, “Feminist theory in technical communication: Making knowledge claims visible,” J. Bus. Tech. Commun., vol. 16, no. 4, pp. 441–477, 2002.
10. K. A. Neuendorf, The Content Analysis Guidebook. Thousand Oaks, CA: Sage, 2002.
11. R. T. Barker, G. W. Rimler, E. Moreno, and T. E. Kaplan, “Family business members’ narrative perceptions: Values, succession, and commitment,” J. Tech. Writing Commun., vol. 34, no. 4, pp. 291–320, 2004.
12. N. W. Coppola, S. R. Hiltz, and N. G. Rotter, “Building trust in virtual teams,” IEEE Trans. Prof. Commun., vol. 47, no. 2, pp. 95–104, Jun., 2004.
13. R. Benbunan-Fich and S. Altschuller, “Web presence transformations in the 1990s: An analysis of press releases,” IEEE Trans. Prof. Commun., vol. 48, no. 2, pp. 131–146, Jun., 2005.
14. S. Blythe, “Exploring uses of IText in campus-community partnerships,” J. Bus. Tech. Commun., vol. 18, no. 3, pp. 270–293, 2004.
15. H. McGovern, “Not just usability testing: Remembering and applying non-usability testing methods for learning how web sites function,” Tech. Commun., vol. 52, no. 2, pp. 175–186, 2004.
16. A. Thayer, M. Evans, A. McBride, M. Queen, and J. Spyridakis, “Content analysis as a best practice in technical communication research,” J. Tech. Writing Commun., vol. 37, no. 3, pp. 267–279, 2007.
17. E. Henry, “Are investors influenced by how earning press releases are written?,” J. Bus. Commun., vol. 45, no. 4, pp. 363–407, 2008.
18. S. A. Rains and A. M. Young, “A sign of the times: An analysis of organizational members’ email signatures,” J. Comput.-Mediated Commun., vol. 11, no. 4, pp. 1046–1061, 2006.
19. D. E. Tanner, Assessing Academic Achievement. Boston, MA: Allyn and Bacon, 2001.
20. L. R. Frey, C. H. Botan, and G. L. Kreps, Investigating Communication: An Introduction to Research Methods. Boston, MA: Allyn and Bacon, 2000.
21. J. Watt and S. van den Burg, Research Methods for Communication Science. Boston, MA: Allyn and Bacon, 1995.
22. E. B. de Groot, H. Korzilius, C. Nickerson, and M. Gerritsen, “A corpus analysis of text themes and photographic themes in managerial forewords of Dutch-English and British annual general reports,” IEEE Trans. Prof. Commun., vol. 49, no. 3, pp. 217–235, Sep., 2006.
23. S. Stemler. (2001). An overview of content analysis. Practical Assessment, Research & Evaluation, [Online]. Available: http://pareonline.net/getvn.asp?v=7&n=17
24. L. Allen, “Scholarly publication as an indicator of change,” presented at the 47th Annu. Meet. Soc. Technical Communication, Arlington, VA, May, 21–24, 2000.
25. K. A. Schriver, Dynamics in Document Design. New York: Wiley, 1997.
26. D. Hopkins, After Modern Art: 1945–2000. Oxford, UK: Oxford Univ. Press, 2000.
27. Ranch & Rural Living Magazine. [Online]. Available: http://www.ranchmagazine.com/sanangelo1922.html
28. R. McSwain, Texas Sheep and Goat Raisers’ Association: 1915–1995: A History of Service. San Angelo, TX: Anchor Publishing, 1996.
29. American Sheep Industry Assoc. [Online]. Available: http://www.sheepusa.org
30. Small Farm Program. [Online]. Available: http://www.sfc.ucdavis.edu/pubs/brochures/angora.html
31. D. Riffe, C. F. Aust, and S. R. Lacy, “The effectiveness of random, consecutive day and constructed week samplings in newspaper content analysis,” Journalism Quart., vol. 70, no. 1, pp. 133–139, 1993.
32. D. Riffe, S. Lacy, and M. W. Drager, “Sample size in content analysis of weekly news magazines,” J. Mass Commun. Quart., vol. 73, no. 3, pp. 635–644, 1996.
33. S. Harner and T. Zimmerman, Technical Marketing Communication. New York: Longman, 2002.
34. S. K. Kaczmarek, Building Goodwill. New York: Longman, 1992.
35. W. Luther, The Marketing Plan: How to Prepare and Implement It. New York: American Management Assoc., 1993.
36. C. Nolan and R. Warmke, Marketing Sales Promotion and Advertising, 7th ed. Cincinnati, OH: South-Western, 1965.
37. J. Cohen, “A coefficient for agreement for nominal scales,” Educ. Psychol. Meas., vol. 20, no. 3, pp. 37–46, 1960.
38. Page Steel and Wire Co., “Page wolf-proof Texas fences will save your lambs,” Sheep and Goat Raisers’ Mag., vol. 3, no. 2, 1922, advertisement.
39. The Colorado Fuel & Iron Co., “The why of those extra wraps,” Sheep and Goat Raisers’ Magazine, vol. 11, no. 12, 1931, advertisement.
40. Tobacco By-Products & Chemical Corporation, “Dip and drench with Black Leaf 40,” Sheep and Goat Raisers’ Mag., vol. 27, no. 1, 1946, advertisement.
41. School of Social Work Doctoral Program. [Online]. Available: http://courses.washington.edu/socw580/contentsoftware.shtml
42. W. Lowe, “Software for content analysis: A review,” Resources Related to Content Analysis and Text Analysis, 2002.