An Overview of Research Methods in Technical Communication Journals (2012–2016)

Researcher’s note (June 2026). Chris Lam and I wanted to know which research methods our field actually used, so we content-analyzed 117 articles from five major technical communication journals and coded each one for method, topic, and whether it was RAD (replicable, aggregable, and data-driven). The core idea was to update earlier method overviews and document our methodological growth as the field matured. That habit of measuring how a discipline produces and validates knowledge now grounds my current work on AI in writing and assessment, where the same question recurs: what counts as replicable, data-driven evidence?

Abstract

This study reports an empirical content analysis of research methods utilized by technical communication researchers over the past five years. In the study, we coded 117 articles for their topical content, research method, and whether the research presented was RAD (replicable, aggregagble, and data-driven). We found clear patterns emerging and discuss potential implications for research in technical communication.

Index Terms – technical communication, research methods, content analysis, data.

Introduction

Scholars primarily characterize our methodological plurality as a strength [1]. Carliner et al.’s study identified a range of methods employed in the research published within technical communication journals, including experiments, case studies, document reviews, and experience reports [2]. Similarly, technical communicators identified the use of 25 different method types, primarily qualitative research focused on discourse and texts and historical research [3].

On the other hand, developing fields like technical communication tend to also have a large body of anecdotal knowledge. As early as 1992, MacNealy reported that the focus of our research from the previous 20 years was on “personal and often limited experiences and preferences” rather than the quantity, quality, and coherence of research [4, p. 533]. However, MacNealy acknowledged that anecdotal research is common for a developing discipline like technical communication. As our field matures, a longitudinal exploration of our methodological growth documents our history, reflects our present, and informs our future.

To prepare the next generation of technical communication researchers, the field must understand how we traditionally conduct research. In this paper, we explore the research methods used in a sample of articles published in the five major technical communication journals from 2012–2016. We then categorize each article as RAD (replicable, aggregable, and data-driven) as well as correlated the method types to the primary topic of the article.

Methods

Our sample included 117 articles published over a five-year period (2012–2016) within five technical communication journals. This sample was randomly selected from a population of 433 articles. Therefore, we coded 27% of the population.

We inventoried and numbered the population in an Excel worksheet and then identified the sample with a random number generator formula. The population included every peer-reviewed article in the five leading technical communication journals: Journal of Business and Technical Communication, Journal of Technical Writing and Communication, Technical Communication, Technical Communication Quarterly, and IEEE Transactions on Professional Communication. [2, 5, 6]. These journals all were previously identified as the leading scholarly forums in the field.

Once we captured the sample, we coded the 117 articles for 3 content variables: method, RAD, and topic. The codebook for method was adapted from Rainey [13].

The RAD variable related to standards of replicable, aggregable, and data-driven research [7]. This was a binary variable that noted if the article employed a RAD or non-RAD approach.

The topic was coded using a 16-variable level coding scheme utilized in previous technical communication research [8, 9] . For mutual exclusivity purposes, we coded each topic for its primary focus. For example, the topic of an article on a collaborative model for teaching international engineering students about the language and culture of a foreign research environment could be classified as collaboration, pedagogy, or intercultural communication. However, the article’s primary focus is pedagogy, so it was coded as such.

We analyzed the data through descriptive and inferential statistics as well as correspondence analysis. Correspondence analysis (or CA) is a geometric technique used to analyze two-way and multi-way tables containing some measure of correspondence between the rows and columns [10]. The approach provides results that are comparable to Principal Components Analysis or Factor Analysis but is designed for non-numeric data. The application of CA has grown in technical communication scholarship in recent years [11, 12].

Results

Table 1. Distribution of Research Methods

Method	Frequency	Expected n	Residual
Not a data-driven study	46	10.6	35.4
Text Analysis	23	10.6	12.4
Mixed Methods	12	10.6	1.4
Survey	12	10.6	1.4
Interview	8	10.6	-2.6
Experiment	4	10.6	-6.6
Usability Study	4	10.6	-6.6
Case Study	3	10.6	-7.6
Participant Observation	2	10.6	-8.6
Psychometric study	2	10.6	-8.6
Ethnography	1	10.6	-9.6

X² = 169.47, p < 0.001

Based on a chi-square goodness of fit test that assumed equal distribution among methods, there is evidence that methods are not equally distributed in our sample of 117 research articles (X² = 169.47, p < 0.001). It seems that over represented methods include non-data-driven research studies and text analysis and underrepresented methods include experiments, usability studies, case studies, participant observation, psychometric studies, and ethnographies.

Table 2. Cross Tabulation of Research Methods and RAD Research

	RAD Frequency	Not RAD Frequency
Text Analysis	13	10
Survey	12	0
Interview	8	0
Experiment	4	0
Usability Study	4	0
Case Study	2	1
Participant Observation	2	0
Psychometric study	2	0
Ethnography	1	0
Text Analysis	13	10
Survey	12	0

As can be seen from Table 2., most articles that reported data-driven studies were categorized as RAD research. However, articles using a text analysis methodology had a significantly higher percentage of research that was not coded as RAD. To supplement this finding, we also coded for a secondary method specifically for articles coded as text analysis. Of the 10 non-RAD text analysis articles in our sample, half used rhetorical analysis method while the other half employed a content or genre analysis method.

Table 3. Cross Tabulation of Topics and RAD Research

	RAD Frequency	Not RAD Frequency
Assessment	1 (33.3%)	3 (66.7%)
Collaboration	7 (100%)	0 (0%)
Communication Strategies	1 (25%)	3 (75%)
Comprehension	3 (75%)	1 (25%)
Design	2 (66.7%)	1 (33.3%)
Editing and Style	1 (33.3%)	2 (66.7%)
Gender	1 (33.3%	2 (66.7%)
Intercultural Communication	7 (70%)	3 (30%)
Knowledge and Information Management	4 (66.7%)	2 (33.3%)
Pedagogy	3 (25%)	9 (75%)
Professionalization	9 (64.3%)	5 (35.7%)
Research Design	1 (25%)	3 (75%)
Rhetoric	5 (31.3%)	11 (68.8%)
Technology	4 (40%)	6 (60%)
Usability and UX	5 (100%)	0 (0%)
Genre	6 (46.2%)	7 (53.8%)

Although some topics had a relatively small sample size, examining topics with at least 5 or more articles provides some interesting insights. For instance, 100% of articles focused on collaboration and usability and UX were all categorized as RAD research. Alternatively, a much larger percentage of rhetoric (68.8%), pedagogy (75%), and genre (53.8%) articles were coded as non-RAD research.

To supplement our findings, we conducted a CA to examine whether specific topics and methods correlated. We found significant association between collaboration articles and experiments. Also, unsurprisingly, articles about usability and UX reported and used a usability methodology.

[Figure 1: CA of topic and method — image to be added.]

Discussion

I. TC Research favors some methods over others

While this is not a surprising finding, it provides some insights into potential areas of growth for technical communication research. For instance, it’s not surprising that a large percentage of our sample’s research studies used a methodology of text analysis as researchers in our field study the artifacts that we produce. However, what is somewhat troubling is that 43.4% of text analysis articles were not replicable, aggregable, and data-driven. This means that a large percentage of these articles cannot be further validated by additional research making them ostensibly “one off” studies because no explicit method was reported in those articles. Because this study did qualitatively examine each individual research article, we cannot be sure why such a disproportionate number of text analyses were coded as RAD. However, our follow-up analysis did reveal that half of the non-RAD text analysis articles used a rhetorical analysis method. This could, perhaps, explain why such a large percentage of text analyses were not RAD. In order to build legitimacy for a discipline, it’s extremely important for researchers to build, test, and challenge each other’s research. In these cases, non-RAD studies make this impossible. It is important to note, however, that our study was a quantitative content analysis and that further qualitative research examining the non-RAD text analyses is warranted for further investigation into this finding.

Another interesting finding was the number of research articles that were not actual data-driven studies. Nearly half of all articles in the sample were coded as such. Although it is important for disciplines to be well-rounded in regards to critical theory and empirical work, the data seems to reveal a disconnect between the two. For instance, 54% of all non-data-driven studies were distributed across just four topics: rhetoric, pedagogy, technology, and professionalization. This pattern reveals a heavy concentration of theory and other commentary-based research in these four topic areas, but much less so among the other 12 topic areas.

II. Some Topics in TC Research are more balanced than others

Articles about collaboration, which accounted for about 6% of the entire sample, were all coded as RAD research studies. Further, research on collaboration has utilized a wide variety of both qualitative (interviews and observation studies) and quantitative (experiments and surveys) methods. Interestingly, however, at least in our sample, little critical work has been published on collaboration over the past five years. One potential explanation for this could be the wealth of existing collaborative frameworks from other disciplines like information science, management, and communication studies. This could, perhaps, point to a potential area for theorists in technical communication to engage and develop theories specific to collaboration on technical writing and communication projects.

On the flip side, articles about pedagogy were primarily not data-driven studies. Further, a good number of these pedagogical articles were classroom experience reports. These types of articles provide many opportunities for empirical researchers to test the findings reported in such experience reports in an effort to build a more robust and evidence-based understanding of technical communication pedagogy.

In summary, we have found several emerging patterns in regards to research method usage in technical communication. We are not suggesting that data-driven studies don’t exist in the field but merely pointing out future opportunities to bolster specific topics that might benefit from a more balanced distribution of research methods and approaches. Future research should examine these patterns and trends within the framework of a longitudinal study examining more than five years of research. Additionally, future work should also use qualitative methods to provide context and to further examine the patterns reported in this study.

References

1. A. M. Blakeslee and R. Spilka, “The state of research in technical communication,” Technical Communication Quarterly, vol. 13, no. 1, pp. 73-92, 2004.
2. S. Carliner, N. Coppola, H. Grady, and G. F. Hayhoe, “What does the Transactions publish? What do Transactions’ readers want to read? ,” IEEE Transactions on Professional Communication, vol. 54, no. 4, pp. 341-359, 2011.
3. A. M. Blakeslee, “The technical communication research landscape,” Journal of Business and Technical Communication, vol. 23, no. 2, pp. 129-173, 2009.
4. M. S. MacNealy, “Research in technical communication: A view of the past and a challenge for the future,” Technical Communication, vol. 39, no. 4, pp. 533-551, 1992.
5. P. B. Lowry, S. L. Humpherys, J. Malwitz, and J. Nix, “A scientometric study of the perceived quality of business and technical communication journals,” IEEE Transactions on Professional Communication, vol. 50, no. 4, pp. 352-378, 2007.
6. E. O. Smith, “Strength in the technical communication journals and diversity in the serials cited,” Journal of Business and Technical Communication, vol. 14, no. 4, pp. 131-184, 2000.
7. R. Haswell, “NCTE/CCCC’s Recent War on Scholarship,” Written Communication, vol. 22, no. 2, pp. 198-223, 2005.
8. R. K. Boettger and E. Friess, “Academics are from Mars, practitioners are from Venus: Analyzing content alignment within technical communication forums,” Technical Communication, vol. 63, no. 4, pp. 314-327, 2016.
9. R. K. Boettger, E. Friess, and S. Carliner, “Who says what to whom? Assessing the alignment of content and audience between scholarly and professional publications in technical communication (1996-2013),” in Proceedings of the IEEE 2014 International Professional Communication Conference, Pittsburgh, Pennsylvania, 2014.
10. M. Greenacre, Correspondence Analysis in Practice. Boca Raton: Chapman & Hall, 2007.
11. R. K. Boettger and C. Lam, “An overview of experimental and quasi-experimental research in technical communication (1992-2011),” IEEE Transactions on Professional Communication, vol. 56, no. 4, pp. 272-293, 2013.
12. C. Lam, “Where did we come from and where are we going? Examining authorship characteristics in technical communication research,” IEEE Transactions on Professional Communication, vol. 57, no. 4, pp. 266-285, 2014.
13. K.T. Rainey, “Doctoral research in technical, scientific, and business communication, 1989-1998,” Technical Communication, vol. 46, no. 4, pp. 510-531, 1999.