Assessment Innovations

In 2016, ABIM conducted a study to examine ways ABIM Board Certified physicians might have access to online resources during a portion of an MOC assessment. Over 800 physicians who had passed the Internal Medicine Certification exam or had taken the Internal Medicine MOC exam within the last five years participated in the study.

The study provided crucial insight into the feasibility of using online resources and how those resources impact both physician experience and performance. 

In 2015, ABIM redesigned new score reports with increased detail, providing better explanation of assessment results and giving more feedback about individual performance to better identify potential areas of weakness. The score report was distributed in a new, electronic format. Learn more about these improvements to the score report.

Unlike single-best-answer items, multiple-response (MR) items require a candidate to select two or three correct responses from a list of answer options. MR items are similar to the traditional single-best-answer items in many ways, but can be useful in areas of medical content where multiple correct responses may naturally occur (e.g., symptoms that occur simultaneously, treatments that are applied concurrently or outcomes that are equally as common or likely). Decisions in medical practice can be complex, and MR items can address this complexity and could make assessments more reflective of physician practice.

ABIM has reviewed ways to improve the design of the screens in assessment to make them more user friendly, focusing on navigation and accessibility. Our current test vendor replaced all their test center monitors with new 24-inch models that display at a much higher resolution. These monitors are almost twice as wide as previously used monitors. 

As part of our co-creation effort, ABIM conducted a usability study in May 2015 to test new designs of the widescreen interface with diplomates and gather feedback that would help improve the final widescreen design. The final design was modified based on diplomate feedback and debuted on the Fall 2016 Cardiovascular Disease Certification and MOC exams. In 2017, other individual assessment programs were converted to make full use of the larger display area and higher resolution of the 24-inch monitors with the goal of delivering all assessments on the widescreen in the near future.

Including media (e.g., images, audio clips or video) in a test question can improve the question’s ability to convey the necessary information to candidates. ABIM continues to review the quality of its image bank and has made it a priority to update images. Audio exhibits (most commonly, heart and lung sounds) provide actual or mimicked sounds for physicians to identify to diagnose specific medical conditions. 

ABIM has committed to implementing a process of reviewing audio sounds to ensure all audio in assessments are high quality. Some previous assessments have included diagnostic imaging videos, such as ultrasounds. 

ABIM recently expanded its video initiative to include professionally filmed videos of real or simulated encounters or scenarios. These videos could be used to assess both listening and observational skills when meeting with a patient, and also other skills including communication, teamwork and patient safety. Video items, when compared to a more traditional written question, convey necessary information to answer the question in a way that is more reflective of common encounters in practice.

In 2014, ABIM made modifications to lab report presentation in its secure assessments. The new display included an array of normal ranges on the screen, results marked with an “H” or “L” when outside of the normal range, and findings tailored to patient characteristics when laboratory test reference ranges vary by demographics. By providing all of the information for the test question on the item screen, this modification could not only improve fidelity to practice, but also improve the usability of the computer test interface.

An example of the more realistic lab report appears below:

Blood urea nitrogen          45          H         [8-20 mg/dL]

Serum creatinine               1.3                      [0.7-1.5 mg/dL]

Serum sodium                   135         L          [136-145 mEq/L]

Serum potassium              3.5                      [3.5-5.0 mEq/L]

Serum digoxin                    1.8                      [therapeutic: 1.0-2.0 ng/mL]

ABIM is in the process of researching and evaluating computer-based clinical simulations to determine their validity, usability and added value above current forms of assessment. 

Recent years have seen computer-based simulation assessment as an innovative testing format for licensure and certification programs. In comparison to traditional multiple choice questions (MCQs), these dynamic performance assessments could provide a powerful tool to measure candidate knowledge and skills with greater depth and breadth, especially for skills not easily measured by MCQs. 

With rapid advancement of computer technology and measurement models, computer-based case simulations are feasible and psychometrically sound. They could provide an innovative way to assess diagnostic decision-making and therapeutic intervention skills in a way which mimics the processes physicians might follow in daily practice. The sequence of responses given in the simulated clinical scenario could be scored to characterize a physician’s performance and provide feedback for improvement. 

ABIM is currently developing a high-fidelity simulation for interventional cardiologists to self-assess their procedural skills. Interventional cardiology simulations use Mentice VIST technology and ABIM-developed catheterization lab simulation scenarios to help physicians identify their strengths and weaknesses in interventional cardiology.

A clinical calculator was developed that allows candidates to access different formulas utilized in everyday medical practice. 

As access to technology has improved, the ability to look up certain types of information, like risk calculators, has become common in practice. The clinical calculator provides candidates access to a type of external resource, normally accessed on the internet, in the secure assessment environment. Candidates are able to select different calculations, such as Anion Gap, based on the question in order to calculate a score or piece of diagnostic data that will inform the correct response. 

The clinical calculator was tested with physicians in a usability study in 2015, and physician feedback was utilized to improve the calculator before it was first implemented on the 2016 Internal Medicine Certification exam and appeared on additional Certification and MOC exams in Fall 2017. A sample clinical calculator screen appears below:

ABIM investigated the demonstration of calipers in current practice and found that there were some differences when compared to calipers used on the Cardiovascular Disease, Critical Care Medicine and Clinical Cardiac Electrophysiology exams. Some functionality issues were identified by physicians in survey feedback. ABIM has since modified the current caliper tool to function as closely as possible to those used by physicians in their current practices. An image of the newly developed caliper appears below:

Feedback from diplomates has informed the development of a new media player with the help of an external developer to play videos, view images and listen to audio clips. The new media player is more user friendly, and includes new features, such as frame-by-frame advance and changing contrast, which more closely resemble tools available to physicians in the field. The new media player debuted on the Interventional Cardiology Certification and MOC exams in Fall 2017. A sample image of the new media player prototype appears below: