One of the more frequent (and frankly just) complaints) about IRBs is that they involve themselves to deeply in reviewing and often modifying the scientific design of a study, sometimes to the point of r
adically re-writing the protocol. There are circumstances where the IRB does need to evaluate the scientific design, and there are other circumstances where it should not get involved in these questions.
A look at the regulations provides little guidance. The question of scientific design really only appears explicitly in the definition of “research” at 46.102(d): “Research means a systematic investigation, including research development, testing and evaluation, designed to develop or contribute to generalizable knowledge.” We will delve into this notoriously ambiguous definition in another blog post. This is the only place the regulations appear to explicitly address scientific design, but the definition simple establishes the scope of the application of the regulations but does not tell us what good or poor scientific design is and how we should consider it in IRB reviews.
There is, however, one statement that gets to the core of the matter. In 46.111(2) we read: “Risks to subjects are reasonable in relation to anticipated benefits, if any, to subjects, and the importance of the knowledge that may reasonably be expected to result.” This statement does not explicitly invoke scientific design, but it is implied by the question of the “importance of the knowledge” to be expected from the protocol. This statement asserts that the risks to subjects not only be balance by possible benefits to those same subjects, but should also be balanced by the possible benefits of the knowledge the study may produce. The bottom line here, is that we should not put subjects at risk for the sake of unimportant scientific findings. And more to the point, it also follows that we should not put subjects at risk for the sake of invalid scientific findings. Herein lies the rationale for considering the scientific design of a study. IF we put subjects at risk, let us do so in the context of good scientific design that will actually demonstrate (or falsify) the hypothesis being tested.
It therefore follows, that if we are not putting subjects at risk in protocol, that there is therefore much less need for balancing those lack of risks against a corresponding benefit of good scientific knowledge. Thus, as risk increases, IRBs need to pay attention to the scientific design of a study to ensure that there is value to be derived from the risks to subjects. But when the risks are low to non-existent, IRBs need not be so concerned about the scientific design of a study. This is not to say, of course, that low risk research cannot yield valuable and important scientific knowledge. But the IRBs charge to ensure valuable and important scientific knowledge only extends as far as the risks to subjects extends. The IRB’s job is to protect human subjects of research and when our research subjects must bear risks of harm, it is our job to make sure that they do so for the sake of a worthy cause.
So, as risks increase, I believe it is important for IRBs to make sure that the protocol will yield valid scientific conclusions to questions that are important to the field of study. Scientific design, of course, is a complex and contextualized concept. Different scientific and clinical discipline have different methods and thresholds for validity and importance. With biomedical IRBs reviewing clinical studies, the members with clinical expertise are generally those best positioned to evaluate the scientific design, though the IRB can certainly call in outside expertise when the need arises. Clinicians in the field are also typically the best positioned to evaluate the importance of the research. A fundamental question to ask of any research study is whether its result will be sufficient to sway the community of practitioners to change or maintain their current standard of care. This is a concept that Benjamin Freedman enunciated in one of the best bioethics articles I know: the idea of clinical equipoise. Research is called for when there is disagreement about the standard of care and a well-designed scientific protocol should have the rigor and statistical power to demonstrate to the community of practitioners the superiority (or inferiority) of the treatment modalities being compared.
Not all research is clinical. A great deal of research is oriented toward understanding relationships and functions within physical, social or psychological systems. Here the determination of importance is determined by the social construction of knowledge in the discipline: what questions it values and does not value. Even what counts as good scientific design can vary widely based on the subject matter and the disciplinary practices in the field. These tend to be the harder cases to evaluate when subjects are put at some level of risk in the research study.
Here also, if subjects are not at risk, it may be very difficult for scientifically oriented members of an IRB to not insist on scientific changes to protocols. What role does the IRB have, or ought it to have, in the face of poorly designed research that presents no more than minimal risk to subjects?
A key point to keep in mind is that the regulations provide IRBs with a very unique and powerful authority: IRB decisions to deny research may not be appealed: “Research covered by this policy that has been approved by an IRB may be subject to further appropriate review and approval or disapproval by officials of the institution. However, those officials may not approve the research if it has not been approved by an IRB.” There are few committees in academia, research institutions and healthcare that are given a federal waiver of appeal for their decisions. This provides investigators with a powerful motivation to make whatever changes an IRB requests without question. But if the regulations do not specifically empower IRBs to impose their judgments about the value and validity of scientific design when risks are minimal, should IRBs be using this cudgel to extort from their investigators changes that they or some of their members believe are need for producing good science?
I would argue that as collegial peer review committees, it is certainly appropriate for IRB members to share their concerns regarding scientific design. This can be a valuable educational experience for investigators. But the IRB should not be wielding its un-appealable authority to disapprove research when risks are minimal and they find the scientific design to be wanting. On the other hand, as the risks to subjects rise, then there is indeed good (regulatory) justification for IRBs to insert themselves into questions of scientific design.
For IRBs that do insist on reviewing scientific design of minimal risk research projects, I suspect that they simply fell into this habit over time. Investigators, not knowing the regulations and practices of IRBs were more motivated to comply for the sake of ease of approval than to complain or raise objections. In this way, the mission of some IRBs has crept into areas it probably is not authorized to exercise its authority. On the other hand, there are IRBs whose institutions have explicitly authorized them to exercise oversight of scientific design. Where such policies have been promulgated through normal governance channels, I have no objection, though I think it would be important for the institution to maintain avenues of appeal, so that the unassailable authority of the IRB is not used beyond the rather narrow confines of protection of human subjection’s.
IRB members generally support the research enterprise and want to see good research in their institutions. Those members with the expertise in scientific design can provide great guidance to investigators, especially students and more junior researchers. But we have to be very careful about using the IRB’s authority to demand changes from investigators. Where subjects are at risk, this may be a necessity, but where subjects are not at risk, our counsels on scientific design should remain collegial and outside of regulatory authority.