Regional Anesthesia & Pain Medicine (RAPM) is a clinical journal that aims to understand how pain medicine, both procedural and non-procedural, impact health and the well-being of patients. Fundamentally, the journal seeks to get as close to the truth as possible for a myriad of medical and procedural pain interventions. The scientific progress depends on volunteer peer reviewers — the peer review process is the vanguard of scientific excellence. ASRA Pain Medicine and RAPM are forever thankful for the work of our reviewers. Because your time is so valuable and your efforts for our journal are so critical, we have created this simple framework for reviewing manuscripts.
The most important factor for a reviewer to consider is whether the findings are important to either clinical practice or patients. The major challenge for essentially all research is that because we are NOT studying the entire population of interest, we are stuck having to infer findings based on a sample of observations. Such inferences are often wrong and may lead to erroneous clinical practices.
A strategic review should address five major criteria.
(1) If the research methodology is flawed or biased, it may lack internal validity.
Internal validity exists if the research was conducted appropriately consistent with the study design. If the study was not performed appropriately, then the relationships between the exposures and the outcomes may be erroneous. Several examples of threats to internal validity from actual manuscripts include:
- A primary investigator with a financial conflict of interest decided on group assignments for an experimental study.
- In an observational study examining the relationship between obstructive sleep apnea and postoperative morbidity, researchers used incomplete ICD-9 codes to classify sleep apnea.
- Researchers stated they conducted an intention-to-treat analysis regarding two different intrathecal local anesthetics but actually dropped all patients who did not develop surgical anesthetics.
(2) If the study investigates a new treatment, is the intervention plausible and broadly applicable to the general population?
This criterion is related to the external validity of the study (i.e., the degree to which a study’s finding applies to the entire population). There is growing concern that the findings from randomized controlled trials are not translating into improvements in health and well-being of patients. Patient morbidity can sometimes emerge when a study lacks reproducibility in the real world, as evidenced by historical examples related to glucose management in the perioperative and critical care settings.
(3) How relevant to clinical practice (and/or population health) is the primary outcome measure?
RAPM is trying to de-emphasize process/intermediate outcome measures such as 24-hour opioid consumption. Rather, we want to determine the clinical implications of improvements in pain management. For instance, does a reduction in opioid therapy actually translate into a healthier patient, a shorter length of stay, less escalation of care, less re-admission, less long-term opioid use, less health care expenditure, less chronic pain, etc.?
(4) Is the effect (or association) large enough to impact clinical practice (and/or population health)?
As a reviewer, you need to decide if the identified effect size is meaningful. Whether or not an effect size matters depends on the nature of the outcome measure. For instance, if death was the primary outcome measure, most of us would agree that a little less death is important finding. In contrast, if a new therapy is evaluating opioid consumption over a 24- to 48-hour period, we would presumably want to see a large effect given that the study is likely not powered to understand the clinical implications of the opioid reduction. Your opinion regarding the importance of the effect size estimate will be based on your expertise and clinical experience.
(5) To what of level certainty is the effect (or association) real (i.e., not due to chance)?
Characterizing precision is critical when making decisions regarding suitability for publication. For many measures, precision is reflected by the 95% confidence interval around the main effect size. The 95% confidence interval is the range of values calculated from the study that contains the true population value with a 95% probability. The precision of a study should be interpreted in the context of whether its full range includes clinically relevant values. Using the example of death, if epidurals versus systemic opioids were demonstrated to decrease 30-day postoperative mortality by 9%, but the point estimate was imprecise with a 95% confidence interval of 3% to 15%, we would still be interested in publishing the study. That is, even a 3% decrease (tail end of the 95% CI) is a big story. Precision also helps you distinguish between the absence of an effect and an underpowered study.
P-values are another way to assess the role of chance explaining observed relationships. We historically have rejected the possibility that chance explains our finding when the P-value is less than or equal to 0.05. The Editorial Office acknowledges that this threshold is controversial and discretionary. However, regardless of the threshold to define significance, once that decision is made, it must be put into the context of points 1 through 4. In other words, a P-value of less than 0.001 may be irrelevant, for example, if the primary outcome metric proves to be a process metric with a small effect size.
If there is no statistical difference among treatment groups and the precision is poor, the study should not be published because it has a high risk of a Type II error. In other words, there is too much risk that you have incorrectly concluded there is no difference when in fact there may be one. RAPM wants to embrace the publication of “negative studies” (i.e., studies that demonstrate no effect), but not underpowered studies that would likely be misconstrued as negative if published.