https://immattersacp.org/weekly/archives/2017/08/15/1.htm

‘Open-book’ format does not appear to affect MOC test performance

A randomized controlled trial assigned physicians to a closed-book exam using typical time, a closed-book exam using 15 minutes of additional time, an open-book exam using typical time, or an open-book exam using 15 minutes of additional time. The open-book format allowed use of one electronic resource.


An open-book format that allowed use of one electronic resource did not appear to have adverse effects on test performance among physicians taking the Maintenance of Certification (MOC) exam, according to a study funded by the American Board of Internal Medicine (ABIM) Foundation.

Researchers conducted a randomized controlled trial to determine the effect of electronic resource availability on MOC exam performance. Physicians who were initially certified by ABIM and passed or took the internal medicine MOC exam in 2012 to 2015 were randomly assigned to a closed-book exam using typical time, a closed-book exam using 15 minutes of additional time, an open-book exam using typical time, or an open-book exam using 15 minutes of additional time. The electronic resource available during the open-book exam was UpToDate.

All physicians, regardless of study group, took the same modified version of the exam, which included 120 questions, two modules of 60 questions each. The exam did not count toward certification and was considered a trial exam. The primary outcomes were item difficulty, item discrimination (defined as how well a question differentiated between high and low abilities), and average question response time. Examination dimensionality (defined as the number of factors measured) and test-taking strategy were secondary outcomes. The study results were published online Aug. 15 by Annals of Internal Medicine.

Eight hundred twenty-five physicians were included in the trial's final data set. Study data were collected from April to September 2016. Median participant age was 43 years, and 39% were women. Physicians in the closed-book groups took less time on the test than physicians in the open-book groups (mean, 79.2 seconds [95% CI, 78.5 to 79.9 seconds] per question vs. 110.3 seconds [95% CI, 109.2 to 111.4 seconds] per question). Mean item discrimination was significantly higher for physicians in the open-book groups versus the closed-book groups (0.35 per question [95% CI, 0.32 to 0.35] vs. 0.39 [95% CI, 0.37 to 0.41] per question). Participants accessed the electronic information resource for 45% of the questions. Seven hundred sixty participants (92%) completed surveys after the trial exam; of these, 84% said that they would feel comfortable using an electronic resource during an exam and 87% said they felt it reflected the way they practice. Fifty-four percent of survey respondents said they had adequate time to look up information during the trial exam.

The authors noted that the exam was not official and that participants may not have been as motivated as under actual test conditions. In addition, they pointed out that they looked at only one electronic resource, that the exam was not full-length, that adding time beyond 15 minutes may have affected the results, and that the results may not be generalizable to other specialties. However, they concluded that adding an electronic resource to an exam with time constraints did not appear to adversely affect performance or change the skills or factors that the exam targeted.

The authors stated that ABIM would begin to allow access to an electronic resource for certain MOC programs, a decision first announced in March 2017. “We will continue to study the open-book format and make adjustments accordingly,” the authors wrote. “This and future work should help credentialing organizations understand the implications of enhancing testing programs with resources while maintaining defensibility by using appropriate time constraints and content.”

In an accompanying editorial, Steven E. Weinberger, MD, MACP, ACP's Associate EVP and EVP/CEO Emeritus, noted that it isn't clear whether an open-book examination should be considered a major advance or a minor step in addressing concerns about MOC.

“I believe that an ideal MOC process should identify knowledge gaps that are important and have some relationship to a physician's practice, allow him or her to close those knowledge gaps, and then demonstrate that the gaps have been closed,” Dr. Weinberger wrote. “This puts the priority for MOC reform on its relevance and value to both patients and physicians, rather than on such outcomes as the ability of a question to discriminate between high-and low-performing examinees.”

He called for ABIM to work with internal medicine and subspecialty organizations “to ensure that MOC is not just a hoop to jump through, but a process that ultimately improves the knowledge and skills of physicians and the quality of the health care they provide.”