The L&D Relevance of Certifying Medical Professionals

---

In this Powered by Learning podcast, Dr. Keith Mann, Vice President, Continuing Certification at The American Board of Pediatrics, shares what’s involved in certifying medical professionals and what L&D professionals can learn from it. You'll find out how providing maintenance of certification activities to pediatricians delivers meaningful and sustainable change in healthcare.

Show Notes:

Dr. Keith Mann shares how the American Board of Pediatrics has evolved the way in which it does assessments and certifications to benefit the physician, the healthcare practice and the patient. His experiences and findings reach beyond the medical world and are relevant for any training department looking to take learning, assessment and certification to a new level. Dr. Mann shares many points including the following:

• Integrating learning with the assessment to give greater value to the training and greater impact to the improvement in your organization.
• Using spaced learning and micro learning to engage the learner and make the training more impactful. ABP accomplishes this with their Question of the Week training activity.
• Seek out learners who can also help create the assessments and the activities. Tap into their expertise and help increase the credibility of the training.
• Assessment based activities with built in learning components can improve outcomes in your organization and beyond. ABP uses research gained from assessments to address the disparity in healthcare to underserved populations.

Learn more about d'Vinci's work with The American Board of Pediatrics:

Question of the Week

Decision Skills Assessment

MOCAM

---

VIEW TRANSCRIPT

Announcer 1: This is Powered by Learning, a podcast designed for learning leaders to hear the latest approaches to creating learning experiences that engage learners and achieve improved performance for individuals and organizations.

Announcer 2: Powered by Learning is brought to you by d'Vinci interactive. For more than 25 years, d'Vinci has provided custom learning solutions to government agencies, corporations, medical education and certification organizations, and educational content providers. We collaborate with our clients to bring order and clarity to content and technology. Learn more at dvinci.com.

Susan Cort: Hello, and welcome to Powered by Learning, I'm your host, Susan Cort. With me today is d'Vinci's CEO, Luke Kempski. Today, we're going to talk with Keith Mann, Vice President, Continuing Certification at the American Board of Pediatrics about what's involved in certifying medical professionals and what L and D professionals can learn from it. Welcome, Keith.

Keith Mann: Thank you so much, Susan and Luke, for having me, it's great to be here and I look forward to our conversation today.

Luke Kempski: Yes. Great to see you again, Keith,

Susan: Keith, start off by telling us a little bit about your background and also your current role at the American Board of Pediatrics.

Keith: I am a pediatrician and, for many years was involved in graduate medical education, then subsequently, quality improvement and patient safety at two different Children's Hospitals. Based upon that background, I was interested in a job at the American Board of Pediatrics several years ago. There was an opening for the Vice President of Continuing Certification, and the job fit, I guess, was good on both sides. I moved from my quality and safety role out in Kansas City to become the vice president for Continuing Certification at the American Board of Pediatrics here in Chapel Hill.

Luke: Oh, that's great. I know, Keith, at d'Vinci, we're involved in learning solutions in technology across a number of clients. We've always found it interesting how the medical professions approach, the ongoing certifications, and especially with specialized physicians. In a broad way, can you tell us about what a pediatrician needs to do to maintain board certification? What activities do they need to complete, and what is the experience like for them?

Keith: Absolutely. A physician becomes initially certified after they complete medical school, and then subsequently, their residency training in pediatrics. Then can get additional credentials or certifications by doing sub-specialty training, neonatology, for example. Then they enter into a life-long program for Continuing Certification. That's the program I oversee at the American Board of Pediatrics. In order to maintain their certification, or continue their certification, a physician has to maintain their professionalism and professional standing. That's a little hard for us to measure in a central way, so we rely on physicians maintaining a valid and unrestricted license at the state level, knowing that the professionalism understanding at that local level is better than more macro level.

Then there are three other components to continuing certification. There's lifelong learning and self-assessment, the assessment of knowledge, judgment, and skills, and the improvement in medical practice. Those are the other three components that we kind of actively work with physicians on maintaining over time. For lifelong learning and self-assessment, there are learning activities that align with their professional practice, whatever their area of interest is. They complete a certain amount of those learning activities over the course of continuous five-year cycles. Their assessment of knowledge, judgment, and skills is assessed through what used to be a secure exam every 7 to 10 years, but now we've moved into something that we're calling MOCA-peds, it's a longitudinal assessment of learning. I'll spend a little more time on that in a second.

Then the last part is improvement in medical practice, which is around the idea that knowledge acquisition is one thing, but knowledge application is another. We want to make sure that physicians are not just learning the right content, but applying that to their practice. As an example, a physician might know that there's a certain age and population of physicians to give the influenza vaccine to, but translating that into practice, so 100% of people get that vaccine is harder. The idea is improvement in medical practice is taking that knowledge and making sure we're applying it to constantly improve patient care. I mentioned a second ago about MOCA-peds, and I'm happy to spend a little bit more time talking about that as well.

Luke: All right, great. We'll come back to that. I think when you talk about those MOC learning activities with a maintenance of certification, kind of the learning activities, I imagine there's a whole range of complexity.

Maybe you could talk about one that's maybe not too complex and is one of your favorites.

Keith: Yes, that's easy. Actually our least complex is probably my favorite. I would say my favorite and probably our most popular activity is something called a question of the week, we call it QOW for short, to keep it simple. QOW gives participants a pediatric case study, an abstract from the literature, an in-depth commentary on a specific topic that's referenced in that abstract, hyperlinked references, and as you can probably intuit from the name, a question, every week for 50 weeks of the year. The activity helps us work with physicians to keep them up to date on core content knowledge because the commentary goes over common topics in general pediatrics, but it also helps us support physicians in keeping up with emerging literature because we're linking the question and the commentary to a recent abstract in the medical literature.

That activity helps us do both of those well, and it occurs in a way that diplomates can space their learning on a weekly basis and can do it relatively quickly. The activity takes between 15 and 30 minutes at the most, depending on how much effort you put into it. And so it can be done in quick periods of time in between lunch or in between a patient, even if you had a few extra minutes. It's a popular activity, I think, for a number of reasons.

Luke: That's excellent. I know that we developed that probably, oh, it's probably been five-plus years ago now, and it's great to see that it increased in popularity.

Keith: We have about 80,000 certified pediatricians of which, over the years, about 20,000 in some way, have engaged with Question of the Week. We have about 7,000 regular users and about a quarter of the pediatricians that are certified with us have engaged with Question of the Week, at least once.

Luke: That's great. I know that the quality improvement, special part of the overall maintenance of certification, and the continuing to stay continuously certified, and how the participation in that component like actually positively impacts the diplomate's knowledge and their practices overall, and the care that they provide. Can you talk a little about that?

Keith: Sure. As you're both aware, healthcare is complex, and unfortunately physicians, pediatricians, and physicians of all specialties don't always deliver the absolute best care to patients. And it's not because there are bad physicians out there, or because there are major knowledge gaps, it's because knowing something and doing something are vastly different things. In healthcare, this concept of quality improvement is a framework that we use to help physicians systematically analyze and subsequently improve the care that is delivered to patients, and then, along with that, outcomes begin to improve. We try to help physicians down that journey by providing different online improvement templates for them to utilize.

We also allow physicians to submit their own quality improvement projects directly to either a sponsor organization that verifies that project on our behalf or directly to us, and we verify their participation in that project.

I think what physicians learned through that exercise is often two-fold. One is they learn about the content with which they are trying to make improvements. In order to make the improvements, you have to delve into the content area to make sure you are up to date in the most recent literature, but more commonly what they're learning is how to apply that content to their practice in a systematic way. So they're learning about analyzing data, measurement, and improvement methodology that allows them to both make a meaningful change in their practice, and then sustain that change over time, so patients ultimately get better care.

Luke: That's really great that they can see the progress that they're making as they're learning and fulfilling their requirements. I know that the continuing certification program, it seems like it'd be fairly difficult and involve a lot of really talented people to come up with the activities and the assessments on an ongoing basis. What does it take to develop a program in terms of the types of roles and backgrounds of the participant and the time it takes to develop?

Keith: I think it takes a lot. It takes a lot of people, and it takes partnerships. I think that's equally as important. We don't really develop anything without some degree of partnerships with either specialty societies that we work with, pediatricians who volunteer for us in a number of different ways. As an example, we have a Continuing Certification Committee with 10 pediatricians from different walks of life who give me and the ABP input on the program continuously. We have focus groups with pediatricians to understand their needs. All the way down to the exams that we administer, it's pediatricians who help us with the analysis of practice that subsequently leads to the percentages of questions on different contents in the exam. Every step of the way, we are engaging pediatricians, either directly or through their societies, to help us build a continuing certification program that our goal is for it to be meaningful, relevant, and valuable to pediatricians. We can't do that without the volunteer work of well over 300 pediatricians that support the American Board of Pediatrics through their efforts.

Luke: That's great. We're certainly hearing a lot about partnerships in a lot of the learning and development organizations outside of the medical field as well in terms of a way to connect, to involve the people who will ultimately participate in the activities, and then also to really get subject matter expertise from a lot of different directions as well. What types of subject matters and types of learning experiences are most popular right now, and why do you think that is?

Keith: It's interesting. Subject matter is a little hard because we have general pediatrics and then 15 pediatric subspecialties, and so the content is different by subspecialty. Now, there are some content areas, which clearly overlap all subspecialties, more generic topics. An example of one we just worked on with you all is around diagnostic bias and diagnostic error. So knowing that a physician can miss a diagnosis or have a delay in a diagnosis, we put together a module with some content experts to help physicians understand and learn about diagnostic bias in a way that makes the likelihood of getting the correct diagnosis at the correct patient at the correct time, more likely. That's the content that could apply to everybody. Then we'll also work specifically with groups on content that is relevant directly to them.

There are also some other areas that we try to focus on, like motivational interviewing, be another example of a topic that can apply to multiple specialties. There is a lot of interest right now in behavioral and mental health. We've seen increasing rates of adolescent depression and suicide and autism, and physicians across multiple domains have to care for patients. General pediatrician, I think, it makes sense that you'd be caring for patients with different mental health concerns, but a specialist who manages a patient with a chronic disease, that patient is likely to have mental health concerns over time.

So a specialist who might not be as trained in developmental pediatrics as a generalist also has to know some of that content. There are some of these overlapping content fields.

From a way-to-learn perspective, we're seeing a lot of interest in micro-learning. Give me something quick that I can consume in a digestible chunk when I have time to do so. I think that's one of the reasons the Question of the Week is popular. We had a decision skills activity, which is a case-based, question-based format that we work with you on. We used to have a 50-question decision skills form that would release. And we're still using that form, but we're trying to now break it up into content. Rather than have to go through 50 questions in order to get credit, we're looking to get 10 questions on cardiology, 10 questions on endocrinology, 10 questions on behavior and developmental medicine. So you could learn more in bite-sized chunks. I think we're hearing a lot more about that over the past couple of years.

Luke: That idea of micro-learning and also spaced out over a time period is definitely popular throughout the learning and development field across a lot of industries as well. That's interesting to hear how that's coming in. I'm sure with students, they're starting to get more of that through their education, even prior to getting to either medical school or to graduate school.

Keith: Absolutely true. We're even seeing the same with our approach towards assessment, where the exam that a physician used to take was seven years apart in continuing certification. They would get 200 questions delivered in a single setting. We've moved towards MOCA-Peds, which is our longitudinal assessment, where we deliver 17 to 20 questions per quarter to physicians, and you release one question at a time. The physician gets, once they answer the question, they actually rate the relevance to their practice and their comfort level with their answer, and then see the answer and a rationale. So they learn immediately after answering the question, and then we can use their confidence and relevance ratings to actually feed them back questions over the course of the exam that they rated as relevant to their practice that maybe they got wrong.

It gives us an opportunity to reinforce that learning at a later time in the year while still maintaining the validity of our assessment product. A physician, instead of getting a single point-in-time exam every seven years, learns and continuously, and then over the course of four years, we take those cumulative questions and give an assessment passing score to them.

Susan: That's really a neat idea, Keith. I love the fact that it's not assessment for assessment sake, but there's actually learning coming through those assessments.

Keith: Absolutely. It's assessment for learning, which I think we hear from pediatricians anyway, that they prefer this model drastically over the point-in-time, traditional exam, and that more importantly, they are learning and applying that learning to their practice. So we get examples for every time we roll out a new specialty of physicians that learned something from one of the questions and applied it to their practice in relatively short order.

Luke: That definitely makes sense. I could see that trend, again, flowing across a lot of learning and development and certification kinds of professionals that are out there. Keith, what are some of your biggest challenges in developing and administering specific activities that are both meaningful and relevant to both the pediatricians and pediatric subspecialists?

Keith: I think the biggest challenge is probably the volume of content and the rapidly evolving medical literature. One of the ways we're tackling that is by integrating assessment and learning. Through MOCA-peds, we're able to both add rationales to the question and introduce articles during the course of the year, either through an emerging topic concept, we were able to introduce a measles question last year, a COVID question this year, or through guidelines that are introduced in the beginning of the year that we subsequently asked questions on. One way we're tackling this is through the MOCA-Peds product. Other ways are partnering with other organizations. We can't possibly produce enough content for general pediatrics and pediatric subspecialties to meet all of the needs, so we partner with other educational organizations.

The American Academy of Pediatrics is a great example, they are the educational home for many pediatricians. A lot of the work that they do, for example, they put on a national conference and exhibit every year that thousands of pediatricians attend, and they have aligned the ability to get a continuing medical education for attending that conference with the ability to also get credit for the continuing certification of a physician for that self-assessment component. A physician can leave the NCE with both CME credit and Maintenance of Certification Part Two credit, which is that self-assessment. So we partner with other educational organizations to try to-- that are already working with physicians on their educational needs to make sure that's aligned with our program.

Luke: That's really great. To wrap us up today, what's next in continuing certification, from your perspective for the ABP, and what's motivating these new approaches?

Keith: I'll mention two things. One is really around just the core components of continuing certification. We'd like to better align assessment and learning even more than we're currently doing. As an example, being able-- for a physician to be able to utilize cumulative scores on their assessments over time, tied to the content domain of the questions they are most commonly getting incorrectly, and then aligning that with what they know about their own practice, to better drive learning. Right now it's difficult for a physician to look into their portfolio and choose learning topics that align most with the gaps that we've seen in their assessments over time. We'd like to make it easier for them to do that.

We'd like it to make it easy for a physician to see their knowledge gaps by displaying them in a way that's visible and then being able to eventually link those to learning activities, which can help close those gaps. That's related to the program itself. Related to what's going on in society right now, we have a big emphasis this year on helping physicians close health disparity gaps. There are clearly differences in many outcomes for children based upon race. Black children have a higher mortality rate and infancy than White children, an infant mortality rate. That's independent of socioeconomic status and other variables that come into play. At the same time, there are other social determinants of health, like the zip code that you live in, your household income.

And so we are looking at helping physicians collect meaningful data on the race or ethnicity of children and collect meaningful data on the social determinants of health that impact outcomes. And would like to help facilitate that so that physicians can look at data stratified by, as an example race or ethnicity, and look at their practice and try to determine are the influenza rates different based upon the race of a child or are the immunization rates different based upon other variables that come into play with zip code the family lives in? If we can help physicians identify that, we may be able, over time, to tackle some of the healthcare disparities that exist in our society. That's some active work that we're really interested in forwarding over the next year or two.

Luke: Yes, it's really fascinating and interesting how you can take data from the local practices in all different kinds of area, and then have it be shared through your platforms, and then applied both back at the local practice, and then also from a national viewpoint.

Keith: A lot of what quality improvement is driving the intrinsic motivation that physicians have to be better. If you ask a group of physicians, do they like quality improvement, those who have never engaged in it will almost automatically say no. It's difficult if you haven't learned it systematically. I never learned it growing up as a physician, I learned it 15, 20 years into my career. So now we're teaching it in training, but for all those physicians practicing over years, the idea of quality improvement is difficult and many are frustrated with the requirement. But the idea is if we can stimulate physicians to get that inherent drive to improve, then that intrinsic motivation takes over. And whether there's a requirement or not is less concerning to the physician, they're doing it because they know it makes a difference in their practice.

So that's the goal of introducing this concept of looking at stratified data on a more holistic way, is that once physicians see the topic might look different by race or ethnicity, there'll be automatically driven to try to understand and close that gap.

Luke: Well, that's really great. Keith touched on such a wide range of topics and so many different things that we could dive into deeper if we only had more time.

Susan: I love how you stress though, Keith, the benefits to the physicians and ultimately the patients coming out of the training and the assessments. It's making it so meaningful for the pediatricians who participate in and ultimately for the people that they take care of. So just some great insights, really interesting. As Luke said, it certainly applies way beyond the medical field. Thank you.

Luke: Thanks so much for spending your time with us and sharing with the Powered by Learning audience what's going on with the American Board of Pediatrics and what's happening in the general world of medical certification and how that can be relevant across a lot of different learning and development applications.

Keith: Well, I truly appreciate the invitation. I've enjoyed spending time with both of you today and let you know that we apply the same continuous improvement principles internally at the American Board of Pediatrics. Hopefully, you'll have me back in a couple of years and I can share the improvements that we've made in that timeframe.

Susan: Consider it an invitation, Keith, thank you. Luke, it was great to talk with Keith. He certainly had some interesting insights to share. What are some of the key takeaways that you got from the interview?

Luke: The integration of the learning and assessments, really getting to that higher level of assessment, and making it really relevant to the practices, to improve the practice. He talked about the question of the week activity, a great example of learning and micro-learning and case-based learning all wrapped up in one. Then it's also, again, a way they can meet their certification requirements. Talked about some of the interesting topics most recently with diagnostic bias, interviewing skills, and mental health concerns. Also, talked about the partnerships that they've established with both their member pediatricians and along with other associations that have special knowledge so that they can really have a broad subject matter and a lot of the learners actually involved in the creation of the assessments and the activities.

Then last, he talked about closing those disparity rates and using the activities as a way to measure where those disparity rates are and where they can make improvements to, to make them better in the future.

Susan: It was so interesting to talk with him, and he certainly so passionate about the topic. It was great to hear how they have evolved their training over the years. I think it sounds like it's even more relevant and meaningful for the physicians, and also more impactful on healthcare in general.

Luke: Yes, lots of good stuff there.

Susan: Thanks Luke, and many thanks to Keith Mann from the American Board of Pediatrics for joining us today. If you have any questions about what we talked about today, you can reach out to us on d'Vinci's social channels, through our website, dvinci.com, or by emailing us at poweredbylearningthatdvinci.com.

Announcer 2: Powered by Learning is brought to you by d'Vinci Interactive. For more than 25 years, d'Vinci has provided custom learning solutions to government agencies, corporations, medical education and certification organizations, and educational content providers. We collaborate with our clients to bring order and clarity to content and technology. Learn more at dvinci.com.