Jury Duty: Understanding the Process and Importance of Peer Review

There are two broad categories of reading material within the world of O&P. The easier types to access and read are articles like this, found in our industry trade publications such as The O&P EDGE and others like it. Distinct from these lighter-reading options are the academic articles published in peer-reviewed journals. These represent a different type of publication with specific governing structures and expectations. This article defines the structure of those academic articles as well as the peer-review process and the role that clinical practitioners can play in ensuring the quality of scientific publications. The Process For magazine articles, the author's work is typically reviewed by copy editors to ensure such things as clarity of content, conforming style, proper grammar, and correct punctuation. However, these editors are generally not content experts. Instead, they rely on the integrity of the individual author and experts in the industry or profession who are consulted for articles to ensure that content is accurate and consistent with modern practice habits. The primary difference between magazine articles and academic journal articles is the process of peer review, in which the author's work is placed before content experts to ensure that, in addition to grammatical clarity, the material meets certain scientific or clinical standards of accuracy and quality. The process, summarized in the flow chart below, begins when an author or team of authors prepare a manuscript, often describing the observations made during a clinical trial or the collective findings of a systematic literature review. The author then selects an appropriate academic journal and submits her or his work. The editor in chief of that journal performs an initial review of the content to determine if its substance and quality are consistent with the journal's standards. If these initial screening standards are met, the manuscript is assigned to peer reviewers for further editing and consideration. As suggested by their title, these reviewers, also called referees, are chosen by virtue of their knowledge of and experience with the manuscript's subject matter. Much like legal proceedings, which involve external scrutiny by a jury of the individual's peers, the peer-review process facilitates a thorough critique of the article's content prior to publication. The number of reviewers assigned to a given manuscript varies according to the journal's standards and the editor's preferences, but is generally two or three. Peer reviewers submit their comments and recommendations to the editor in chief, who ultimately decides the next step for a given manuscript. In rare instances, a manuscript may be accepted for publication at this early stage of review. Alternatively, the manuscript may be rejected. The prevalence of outright rejection varies rejection rates observed in more prestigious publications, but is generally around 25 percent. 1 In such cases, the author may choose to abandon his or her publication efforts or submit the manuscript to another journal. Recommendation of minor or major revision is the most common editorial decision. This is the point at which the peer-review process will often improve the quality and clarity of a submitted manuscript. As part of their process, peer reviewers are able to make discreet comments to the editor and open suggestions to the author. Such comments may include requests to clarify aspects of how a given study was performed, observations of the study's weaknesses or limitations that the author should acknowledge, or suggestions of related articles that should be considered as the reader attempts to place the immediate study findings in the broader scope of the published evidence. The comments from the peer reviewers and the editor in chief are returned to the author, who incorporates the changes and suggestions into his or her manuscript and resubmits it to the journal. The review process is then repeated. Again, according to the journal's policies and the editor in chief 's preferences, the revised manuscript may be accepted for publication or forwarded to peer reviewers for further scrutiny, feedback, and revision. Thus the peer-review process is a lengthy one, but justifiably so, as it helps to ensure that those manuscripts that become part of the published "evidence" are thoroughly reviewed and refined. Peer Reviewers Peer reviewers can be hard to find. The process is anonymous, with the identity of the reviewers withheld from both authors and readers to allow the reviewers to provide comments and feedback freely. Not surprisingly, it can be difficult to recruit experts to participate in volunteer, unrecognized efforts. In many cases, this may mean that those experts who could provide the most insightful feedback on a given manuscript choose not to do so. Just as a practicing clinician might be intimidated at the prospect of reviewing the statistical methodology incorporated within a given publication, it is equally concerning when pure academics are called upon to interpret the clinical relevance of a given study design and its observations. Particularly when studies are performed to improve clinical outcomes, there is considerable value in having clinician peer reviewers. Because the structure of academic publications can be unfamiliar to practicing clinicians, the remainder of this article introduces potential reviewers to some of the key elements of peer-reviewed articles. The Journals Authors often have a broad choice of where to seek publication for their work, as there are more than 25,000 active academic peer-reviewed periodicals. 2 However, this number is drastically reduced as an author attempts to match his or her manuscript with the content scope of existing publications. Competition exists, with some journals viewed as more prestigious than others. This status is often quantified and compared using a journal's "impact factor," as measured by the frequency with which an average article in a journal has been cited in a particular year. Thus, an author's decision to submit a manuscript to a given journal is often a product of the journal's scope of content and impact factor. [caption id="" align="alignnone" width="466"] Table 1: Impact factors of some familiar high-impact journals, as reported by www.impactfactorsearch.com.[/caption] Journals with higher distribution and containing high-quality articles that are referenced frequently by other authors will have higher impact factors. The impact factors of some familiar high-impact journals, as reported by www.impactfactorsearch.com , are shown in Table 1. Since the broader scientific community has limited interest in and awareness of the world of O&P, O&P rarely finds mention in the highest-impact journals. However, exceptions can be cited. In 2013, the New England Journal of Medicine published a case report by Hargrove et al. that describes an externally powered transfemoral prosthesis controlled using pattern recognition of myosignals from both natively innervated musculature and muscles that had undergone targeted muscle reinnervation (TMR). 3 This combination of externally powered lower-limb prostheses, pattern recognition myoelectric control, and TMR in lower-limb amputation constitutes a proud moment for the profession, but also underscores the "wow factor" necessary to penetrate publication in high-impact journals. The reality is that physical medicine constitutes a fairly small segment of the medical community, with O&P representing a still smaller segment of interest and awareness. This is reflected by the impact factors of those journals where O&P articles are most commonly published (Table 2). [caption id="" align="alignnone" width="551"] Table 2: O&P-related academic journals and their impact factors, as reported by www.impactfactorsearch.com.[/caption] Title In choosing a title for their manuscript, authors of academic articles are targeting their audience. Unlike magazine articles where a catchy or creative title may draw in prospective readers, academic readers are searching for specific content. Thus the titles of peer-reviewed manuscripts are intended to be as descriptive as reasonably possible, detailing the full scope of the treated content. So while the title "Thermal Conductivities of Commercially Available Prosthetic Materials," 4 recently published in the Journal of Prosthetics and Orthotics , may not lure in the casual reader, it clearly relates its content to the prospective academic reader. Abstract If a prospective reader finds a descriptive title that suggests that the article may be of interest, the next portion of the manuscript that he or she will turn to is the abstract. The abstract is considered by some to be the most important part of the article. For busy clinicians or interested readers who are unable to access the full article quickly and freely, the abstract is generally made available and might be the only part of the manuscript that is actually read. Alternatively, a potential reader may scan the abstract to determine whether the contents of the article warrant full attention. Given its role as both a summary and screening tool, the abstract should provide a comprehensive yet concise review of the article's content. What was the study examining and in which treatment populations? How was the study conducted? What were its key findings or observations? The abstract should not be a teaser but should instead contain a detailed summary of the study design, its findings, and the suggestions of those findings upon clinical care. Introduction If the abstract has sufficiently engaged the reader's attention, the next segment of the manuscript that he or she will encounter is the introduction. The introduction presents the purpose of the paper itself. It should present what is currently known and what is suspected in a given area, and where there appear to be voids in understanding. The findings of previous studies should be represented accurately. Ideally, the background material transitions into the presentation of a clinical problem that requires additional understanding and effectively justifies the study that will be described throughout the remainder of the manuscript. A clear hypothesis should be presented at this point, describing the observations that the author expected to see when he or she began the clinical trial. Methods The standard for the methods section of a paper in an academic publication is that it should contain enough detail that a second research team could effectively replicate the clinical trial. Thus it should contain sufficient detail regarding subjects, interventions, procedures, and data analysis. Who were the subjects and how were they recruited? What were the inclusion and exclusion criteria that were used, and were these clinically defensible criteria? How generalizable will the observations from these subjects be to other treatment populations? What were the interventions, and were these clinically defensible? Were specific components used and identified? What protocols were carried out on the subjects? Were there reasonable periods of acclimation between the introduction of a change and measurement of the effects? What outcomes were measured? Was there clinical utility to the outcomes, i.e., will these findings ultimately matter? How were data analyzed? Against what were experimental data points compared? While clinically minded reviewers may not have the acumen to design some of these studies, they are well positioned to assess the study methodology and determine if the resultant data is consistent with clinical scenarios and observations. It is not uncommon for well-intentioned researchers to commit fundamental mistakes in their methodology that compromise the clinical relevance of their observations, but such errors are only caught when the article is reviewed by clinically minded peer referees. Results Once the study's methodology has been described, the author proceeds to the results section in which he or she fully describes observations made during the trial. Data should be presented clearly and concisely, allowing a complete understanding of the observed effects associated with the experimental necessary to convey these findings. It is within the results section that the author reports the statistical analysis associated with his or her observations. For many clinician reviewers, this can be the most intimidating part of the review. Such reviewers need not feel compelled to be authorities on statistical methodologies. It is reasonable for reviewers to disclose to the editor in chief if they feel they are unable to fully assess the appropriateness of statistical manipulations. However, even if the statistical manipulations are beyond the reviewer's comprehension, the generalizable conclusions that are drawn from them should still be clear to any reviewer. In this regard, if you as the reviewer can't interpret the author's findings, then other readers will likely have the same experience. Requests for clarification and explanation are entirely appropriate during the review process. Discussion This is the section where the author interprets his or her findings within the broader body of existing published research. Were the findings consistent with those of earlier studies? If not, to what might these differences be attributed? As the author interprets the implications of his or her findings, it is important to ensure that the data adequately supports these interpretations and that the author is not outreaching the data. To the extent that there are other variables that may have explained variations in outcomes, these need to be articulated clearly at this point in the manuscript. Any limitations to the applications of the study's findings should also be presented. Finally, a clear distinction often needs to be made between data that is statistically significant and data that is clinically significant. That is, differences that create sufficient separation to satisfy statisticians may or may not equate to differences that are noticeable in clinical settings. Alternatively, data may suggest a clinically meaningful change but fall short of the standards set for statistical significance. Changes in practice patterns are justifiable when interventions produce clinically relevant alterations or improvements. Jury Duty In today's healthcare environment, the content of published evidence is becoming increasingly valued. While magazine articles have a place in the reading materials of clinical practitioners, peer-reviewed articles represent a higher level of evidence that requires additional scrutiny and revision prior to publication. However, the value of peer review is largely dependent upon the quality of peer reviewers. The clinical relevance of peer-reviewed articles is of particular interest and requires clinically minded reviewers. To that end, clinicians are encouraged to engage within the jury that oversees and improves the collective body of published evidence. Readers with additional interest in the peer-review process are referred to a webinar recently held by the American Academy of Orthotists and Prosthetists and hosted by Steven Gard, PhD, "Manuscript Peer Review: Critical Review of the Literature," which can be accessed at www.oandp/com/link/299 . Phil Stevens, MEd, CPO, FAAOP, is in clinical practice with Hanger Clinic, Salt Lake City. He can be reached at philmstevens@hotmail.com . References Calcagno, V., E. Demoinet, K. Gollner, L. Guidi, D. Ruths, and C. de Mazancourt. 2012. Flows of research manuscripts among scientific journals reveal hidden submission patterns. Science 338:1065-9. doi:10.1126/science.1227833, 2012. Simon Fraser University. Appendix C: How many active, scholarly peer reviewed journals? pages.cmns.sfu.ca/heather-morrison/appendix-c-how-many-active-scholarly-peer-reviewed-journals . Hargrove, L. J., A. M. Simon, A. J. Young, R. D. Lipschutz, S. B. Finucane, D. G. Smith, and T. A. Kuiken. 2013. Robotic leg control with EMG decoding in an amputee with nerve transfers. New England Journal of Medicine 369 (13):1237-42. Webber, C. M., M. R. Klittich, A. Dhinojwala, and B. L. Davis. 2014. Thermal conductivities of commercially available prosthetic materials . Journal of Prosthetics and Orthotics 26 (4):212-5.

Jury Duty: Understanding the Process and Importance of Peer Review

Evidence-Based Practice in O&P: Where are we now? Where are we going?

Medicare O&P Utilization: A Longitudinal Review

Implantable Fibers Could Revolutionize Neural Prosthetics

O&P JOBS

Certified Prosthetist / Prosthetist-Orthotist (CP/CPO) Gates Prosthetics + Mobility Clinic — Mountain Home, AR

Clinic Manager – Certified Prosthetist Orthotist (CPO) – Rogers, AR

CPO

Get unlimited access!

EDGE DIRECT

Welcome Back!

Retrieve your password

Not enough quota to unlock this post

Jury Duty: Understanding the Process and Importance of Peer Review

Support authors and subscribe to content.

Subscribe

Related posts:

Evidence-Based Practice in O&P: Where are we now? Where are we going?

Medicare O&P Utilization: A Longitudinal Review

Implantable Fibers Could Revolutionize Neural Prosthetics

O&P JOBS

Certified Prosthetist / Prosthetist-Orthotist (CP/CPO) Gates Prosthetics + Mobility Clinic — Mountain Home, AR

Clinic Manager – Certified Prosthetist Orthotist (CPO) – Rogers, AR

CPO

Get unlimited access!

EDGE DIRECT

Welcome Back!

Retrieve your password

Not enough quota to unlock this post