The Role of Integrative Educational Intervention Package (Monthly ITE, Mentoring, Mocked OSCE) in Improving Successfulness for Anesthesiology Residents in the National Board Exam

AUTHORS

Ali Dabbagh ORCID 1 , 2 , * , Hedayatollah Elyassi 3 , A. Sassan Sabouri 1 , 4 , Kourosh Vahidshahi 5 , Seyed Amir Mohsen Ziaee 6 , Anesthesiology Department 1

1 Anesthesiology Department, School of Medicine, Shahid Beheshti University of Medical Sciences, Tehran, Iran

2 Anesthesiology Research Center, Shahid Beheshti University of Medical Sciences, Tehran, Iran

3 Anesthesiology Department, Anesthesiology Research Center, School of Medicine, Shahid Beheshti University of Medical Sciences, Tehran, Iran

4 Anesthesiology Department, Massachusetts General Hospital, Harvard Medical School, Boston, United States

5 Pediatric Cardiology Department, School of Medicine, Shahid Beheshti University of Medical Sciences, Tehran, Iran

6 Urology Research Center, Shahid Beheshti University of Medical Sciences, Tehran, Iran

How to Cite: Dabbagh A, Elyassi H, Sabouri A S, Vahidshahi K, Ziaee S A M, et al. The Role of Integrative Educational Intervention Package (Monthly ITE, Mentoring, Mocked OSCE) in Improving Successfulness for Anesthesiology Residents in the National Board Exam, Anesth Pain Med. 2020 ; 10(2):e98566. doi: 10.5812/aapm.98566.

ARTICLE INFORMATION

Anesthesiology and Pain Medicine: 10 (2); e98566
Published Online: April 23, 2020
Article Type: Brief Report
Received: October 2, 2019
Revised: November 12, 2019
Accepted: November 14, 2019
Crossmark
Crossmark
CHECKING
READ FULL TEXT

Abstract

Background: National Board of Anesthesiology (NBA) pass rate is an important and critical step in clinical residency programs.

Objectives: This study was designed to assess the relationship between an integrative educational intervention (IEI) and the relative annual pass rate (RAPR). RAPR is defined as ratio of NBA pass rate of Shahid Beheshti University of Medical Sciences (SBMU) to the NBA pass rate of all the anesthesiology residency programs across Iran.

Methods: In a descriptive-analytic retrospective study from 2012 to 2019, RAPR was calculated. IEI was implanted in the latter 4years period of this time interval includes: (1) individualized mentorship for residents by faculty members; (2) monthly in-training examination (ITE) in written; and (3) periodical mocked OSCE exam. Spearman’s correlation coefficient was used to assess correlation between integrative educational intervention and RAPR results. P value less than 0.05 was considered statistically significant.

Results: There was a statistically significant relationship between “integrative educational intervention program” and the RAPR results: Spearman’s correlation coefficient = 0.655 (P value = 0.039).

Conclusions: The IEI package of Anesthesiology Department, SBMU showed a significant relationship with improvements in successfulness for anesthesiology residents in the National Board Exam (RAPR trend). More prolonged studies could prevail further aspects of these interventions.

1. Background

Trainee assessment is one of the main steps in medical education and training process; both for undergraduate and postgraduate trainees (1-3). Passing the National Board of Anesthesiology certifying exam is among the criteria for assessment of residency programs (4-6). In addition, it is a major challenge for each resident, as a main step in being certified for clinical residents in medical specialties (7). Though it is a main success to pass the exam, the time and effort needed to be prepared for the exam is a major stress-provoking factor for residents. Add to this stress, the possibility that there is a chance for failing the exam, which affects especially those who have failed. This stress is among the main factors for resident burnout (8, 9). Anesthesiology is among the residency programs with a significant degree of resident burnout in some studies, being in top of the resident burn out list (9-12). Institutional support could improve well-being in residents (11). In addition, assessments of trainees are among the fields necessitating research that is more sophisticated; mandating more collaborative research to improve assessment methods (13).

There have been a number of methods to increase the success rate and to improve the capacity of the residents in passing the exam. However, none has proved to be perfect yet and research in this field is still active. In this study, we assess the trend of pass rates in National Board Exam after implementing an integrative educational intervention (IEI) consisted of monthly written exam, periodical mocked objective structured clinical exams (OSCE) and individualized mentorship program in Anesthesiology residents, Shahid Beheshti University of Medical Sciences (SBMU), Tehran, Iran (14).

2. Objectives

The trend was assessed over the 8-year time interval to see the relationship between period for interventions and the pass rates in National Board Exam.

3. Methods

This is a descriptive-analytic retrospective study performed through an 8-year period (2012 - 2019). In this study, the trend of pass rate was assessed. This time interval from 2012 to 2019 included one 4-year interval before the implementation of the IEI and a 4-year interval in which the following package of IEI program was performed throughout the whole time interval:

1) Individualized mentorship for residents by faculty members; described in detail in another study (14);

2) Monthly in-training examination (ITE) in written multiple choice questions (MCQ) format;

3) Periodical mocked OSCE exam.

Anesthesiology Department Examination Committee (ADEC), SBMU, performed these interventions from A to Z with participation of all faculty members, based on a standard and peer reviewed departmental protocol which is described before (14). To make the annual pass rate comparison more standardized, the annual pass rate of Anesthesiology Department, SBMU was compared with the nationwide annual pass rate of the National Board exam using the following ratio:

Relative annual pass rate (RAPR) = pass rate of SBMU anesthesiology residents/pass rate of the National Board of Anesthesiology.

Spearman’s correlation coefficient was used to assess the possibility of correlation between the integrative educational intervention and the RAPR results. One-tailed correlation coefficient was considered statistically significant at the 0.05 level. All statistical analyses were done using SPSS (version 11.5; SPSS Inc, Chicago, IL, USA). P values < 0.05 were considered statistically significant.

4. Results

RAPR results are demonstrated in Table 1. These results showed that RAPR increased annually during the years that the IEI (i.e. mentorship, monthly written exams and periodical mock OSCE) was done. The assessment of relationship between IEI and the RAPR results demonstrated a significant relationship: Spearman’s correlation coefficient = 0.655 (P value = 0.039).

Table 1. Annual Results of RAPRa
YearRAPR
20121.00
20131.01
20141.10
20151.15
20161.06
20171.20
20181.36
20191.35

aRelative annual pass rate (RAPR) = pass rate of SBMU anesthesiology residents/pass rate of the National Board of Anesthesiology.

5. Discussion

This descriptive-analytic retrospective study demonstrated that there was a statistically significant relationship between the IEI (i.e. package of the three aforementioned interventions) and the annual RAPR index. In addition, there was an increasing trend in the annual RAPR. These results are indicative of global improvements in residency training based on a programmed educational intervention associated with feedback and continuity over a couple of years.

Medical knowledge has been quoted as one of the main six core competencies of the Accreditation Council for Graduate Medical Education (ACGME) in training clinical residents; including anesthesiology residents (2, 15-17). More importantly, there is an integral relationship between Miller's Pyramid framework (18, 19) and ACGME core competencies. For example, Williams et al. (20), demonstrated this relationship between Miller’s Pyramid framework on one side and the ACGME core competencies, on the other side; both sides of this relationship are cornerstones of medical education. Many others have similar results confirming the latter study by Williams et al. (21-25). These findings stress the importance of knowledge as one of the main competencies for each clinical resident mandating efforts to improve outcome of educational interventions, which is in concordance with the results of our study.

Among the wide variety of approaches, MCQ’s have an old history and are frequently designed with varying degrees of validity and reliability (26-28). Albeit these drawbacks, MCQ exams are still among the main armamentarium in assessment of clinical residency programs; focusing but not limited to the knowledge domain (29, 30). In another study, Lingenfelter et al. (31) demonstrated that ITE could provide “formative assessments” regarding “medical knowledge” of the residents. These studies support our findings regarding the relationship between performing an integrated and continuous program based in part on MCQ exams and improvement in RAPR trend.

Clinical faculty members are the mainstay of trainee assessment. However, their role is much more than just taking exams. In other words, performing formative exams supported by regular mentoring function improves their impact significantly (14, 32-35). There is significant evidence demonstrating the role of mentorship in training of anesthesiology residents, not only improving their knowledge and clinical performance of the trainee but also affecting professionalism and patient care characteristics; all being among the main ACGME core competencies (33, 34, 36-38). These studies are in favor of our findings regarding the relationship between regular mentoring and improvement in RAPR index.

Mock oral examination especially if being standardized could detect “areas of relative strength and weakness” which are different between residency programs (39, 40). In another study, it was demonstrated that multi-institutional mock oral examination can improve performance in American Board of Surgery Certifying Examination and identify the residents at risk of board exam failure (41).

Resident burnout is a main consideration in each residency program. Among the many factors that could be involved in burnout of residents, emotional factors are in the top of the list (42, 43). There is possibility that the integrative educational intervention assessed in this study could have some impacts regarding resident burnout because of improving the success rate and the possible emotional effects of mentorship. However, this needs more studies to be accomplished to see if such relationship really exists.

5.1. Conclusions

Our study demonstrated that the integrative educational intervention (IEI) package of Anesthesiology Department, SBMU had a significant relationship with improvements in RAPR trend; which means this package could be beneficial in improving the results of successfulness for anesthesiology residents in passing the National Board Exam. More prolonged studies could prevail further aspects of these interventions.

Contributor Information

Anesthesiology Department:

Dariush Abtahi, Homayoun Aghamohammadi, Narges Sadat Ahmadizadeh, Seyed Hossein Ardehali, Noor Mohammad Arefian, Hamidreza Azizi-Farsani, Faranak Behnaz ORCID , Shideh Dabir, Payman Dadkhah, Mastaneh Dahi, Masih Ebrahimy Dehkordy ORCID , Ahmad Eghbali, Lida Fadaizadeh, Kamal Fani, Mohammad Fathi, Behrooz Farzanegan, Mehdi Ghahremani, Babak Gharaei, Mahshid Ghasemi ORCID , Reza Goharani, Seyed Masoud Hashemi, Seyed-Mohammadreza Hashemian, Alireza Jaffari, Alireza Jahangiri Fard, Hossein Ali Jelveh-Moghadam, Mohammadreza Hajiesmaeili, Behnam Hoseini ORCID , Mohammadreza Kamranmanesh, Shayesteh Khorasanizadeh ORCID , Mehran Kouchek, Homeyra Kouzekanani, Nilofar Massoudi, Alireza Mahdavi, Elham Memary, Alireza Mirkheshti, Mir-Mohammad Miri, Gholamreza Mohseni, Sirous Momenzadeh, Faramarz Mosaffa ORCID , Mohammadreza Moshari, Kamran Mottaghi, Masoud Nashibi ORCID , Navid Nooraei, Hossein Nouri, Tahereh Parsa, Mahtab Poor Zamany Nejat Kermany, Badiozaman Radpay, Seyed Sajad Razavi, Fatemeh Roodneshin, Afsaneh Sadeghi, Farhad Safari ORCID , Sara Salarian, Alireza Salimi, Sohrab Salimi, Shahram Sayyadi ORCID , Parissa Sezari ORCID , Seyed Pouzhia Shojaei, Soodeh Tabashi, Mehrdad Taheri, Ardeshir Tajbakhsh, Houman Teymourian, Maryam Vosoghian

Footnotes

References

  • 1.

    Yang SC, Tsou MY, Chen ET, Chan KH, Chang KY. Statistical item analysis of the examination in anesthesiology for medical students using the Rasch model. J Chin Med Assoc. 2011;74(3):125-9. doi: 10.1016/j.jcma.2011.01.027. [PubMed: 21421207].

  • 2.

    Tetzlaff JE. Assessment of competence in anesthesiology. Curr Opin Anaesthesiol. 2009;22(6):809-13. doi: 10.1097/ACO.0b013e3283326958. [PubMed: 19773650].

  • 3.

    Newble DI, Jaeger K. The effect of assessments and examinations on the learning of medical students. Med Educ. 1983;17(3):165-71. [PubMed: 6865814].

  • 4.

    Atsawarungruangkit A. Residency program characteristics that are associated with pass rate of the American Board of Pediatrics certifying exam. Adv Med Educ Pract. 2015;6:517-24. doi: 10.2147/amep.s90022. [PubMed: 26316837]. [PubMed Central: PMCPmc4542559].

  • 5.

    Atsawarungruangkit A. Relationship of residency program characteristics with pass rate of the American Board of Internal Medicine certifying exam. Med Educ Online. 2015;20:28631. doi: 10.3402/meo.v20.28631. [PubMed: 26426400]. [PubMed Central: PMCPmc4590350].

  • 6.

    Holt KD, Miller RS, Vasilias J, Byrne LM, Cable C, Grosso L, et al. Relationships Between the ACGME Resident and Faculty Surveys and Program Pass Rates on the ABIM Internal Medicine Certification Examination. Acad Med. 2018;93(8):1205-11. doi: 10.1097/acm.0000000000002228. [PubMed: 29596081].

  • 7.

    Schwed AC, Lee SL, Salcedo ES, Reeves ME, Inaba K, Sidwell RA, et al. Association of General Surgery Resident Remediation and Program Director Attitudes With Resident Attrition. JAMA Surg. 2017;152(12):1134-40. doi: 10.1001/jamasurg.2017.2656. [PubMed: 28813585]. [PubMed Central: PMCPmc5831434].

  • 8.

    Thomas NK. Resident burnout. Jama. 2004;292(23):2880-9. doi: 10.1001/jama.292.23.2880. [PubMed: 15598920].

  • 9.

    Rodrigues H, Cobucci R, Oliveira A, Cabral JV, Medeiros L, Gurgel K, et al. Burnout syndrome among medical residents: A systematic review and meta-analysis. PLoS One. 2018;13(11). e0206840. doi: 10.1371/journal.pone.0206840. [PubMed: 30418984].

  • 10.

    Wolpaw JT. It Is Time to Prioritize Education and Well-Being Over Workforce Needs in Residency Training. Acad Med. 2019. doi: 10.1097/acm.0000000000002949. [PubMed: 31425182].

  • 11.

    Sun H, Warner DO, Macario A, Zhou Y, Culley DJ, Keegan MT. Repeated Cross-sectional Surveys of Burnout, Distress, and Depression among Anesthesiology Residents and First-year Graduates. Anesthesiology. 2019;131(3):668-77. doi: 10.1097/aln.0000000000002777. [PubMed: 31166235].

  • 12.

    de Oliveira GJ, Chang R, Fitzgerald PC, Almeida MD, Castro-Alves LS, Ahmad S, et al. The prevalence of burnout and depression and their association with adherence to safety and practice standards: a survey of United States anesthesiology trainees. Anesth Analg. 2013;117(1):182-93. doi: 10.1213/ANE.0b013e3182917da9. [PubMed: 23687232].

  • 13.

    Holmboe ES, Sherbino J, Long DM, Swing SR, Frank JR. The role of assessment in competency-based medical education. Med Teach. 2010;32(8):676-82. doi: 10.3109/0142159x.2010.500704. [PubMed: 20662580].

  • 14.

    Dabbagh A, Massoudi N, Vosoghian M, Mottaghi K, Mirkheshti A, Tajbakhsh A, et al. Improving the Training Process of Anesthesiology Residents Through the Mentorship-Based Approach. Anesth Pain Med. 2019;9(1). e88657. doi: 10.5812/aapm.88657. [PubMed: 30881915].

  • 15.

    Schmit EO, Wu CL, Khodadadi RB, Herrera LN, Williams WL, Estrada CA. What Defines an Honors Student? Survey of Pediatric and Internal Medicine Faculty Perspectives. South Med J. 2019;112(8):450-4. doi: 10.14423/smj.0000000000001005. [PubMed: 31375843].

  • 16.

    Tetzlaff JE. Assessment of competency in anesthesiology. Anesthesiology. 2007;106(4):812-25. doi: 10.1097/01.anes.0000264778.02286.4d. [PubMed: 17413920].

  • 17.

    Soto RG, Cormican DS, Gallagher CJ, Seidman PA. Teaching systems-based competency in anesthesiology residency: development of an education and assessment tool. J Grad Med Educ. 2010;2(2):250-9. doi: 10.4300/jgme-d-09-00078.1. [PubMed: 21975630]. [PubMed Central: PMCPmc2930315].

  • 18.

    Miller GE. The assessment of clinical skills/competence/performance. Acad Med. 1990;65(9 Suppl):S63-7. doi: 10.1097/00001888-199009000-00045. [PubMed: 2400509].

  • 19.

    Etezadi F, Najafi A, Pourfakhr P, Shariat Moharari R, Reza Khajavi M, Imani F, et al. An Assessment of Intubation Skill Training in Novice Anesthesiology Residents of Tehran University of Medical Sciences With the Use of Mannequins. Anesth Pain Med. 2016;6(6). e39184. doi: 10.5812/aapm.39184. [PubMed: 28975071]. [PubMed Central: PMCPmc5560575].

  • 20.

    Williams BW, Byrne PD, Welindt D, Williams MV. Miller's Pyramid and Core Competency Assessment: A Study in Relationship Construct Validity. J Contin Educ Health Prof. 2016;36(4):295-9. doi: 10.1097/ceh.0000000000000117. [PubMed: 28350312].

  • 21.

    Miller N, MacNew H, Nester J, Wiggins JB, Shealy C, Senkowski C. Jump starting a quality and performance improvement initiative to meet the updated ACGME guidelines. J Surg Educ. 2013;70(6):758-68. doi: 10.1016/j.jsurg.2013.06.016. [PubMed: 24209652].

  • 22.

    Carney PA, Palmer RT, Fuqua Miller M, Thayer EK, Estroff SE, Litzelman DK, et al. Tools to Assess Behavioral and Social Science Competencies in Medical Education: A Systematic Review. Acad Med. 2016;91(5):730-42. doi: 10.1097/acm.0000000000001090. [PubMed: 26796091]. [PubMed Central: PMCPmc4846480].

  • 23.

    Collichio F, Muchmore EA. The American Society of Hematology and ASCO Curricular Milestones for Assessment of Fellows in Hematology/Oncology: Development, Reflection, and Next Steps. Am Soc Clin Oncol Educ Book. 2018;38:887-93. doi: 10.1200/edbk_201773. [PubMed: 30231329].

  • 24.

    Cruess RL, Cruess SR, Steinert Y. Amending Miller's Pyramid to Include Professional Identity Formation. Acad Med. 2016;91(2):180-5. eng. doi: 10.1097/acm.0000000000000913. [PubMed: 26332429].

  • 25.

    Ten Cate O. Nuts and bolts of entrustable professional activities. J Grad Med Educ. 2013;5(1):157-8. doi: 10.4300/jgme-d-12-00380.1. [PubMed: 24404246]. [PubMed Central: PMCPmc3613304].

  • 26.

    Amouei A, Barari R, Naghipour D, Mortazavi Y, Hosseini SR. Evaluation of Multiple Choice Questions Quality Trend as Structure and Taxonomy. Future of Medical Education Journal. 2014;4(3):26-30. doi: 10.22038/fmej.2014.3282.

  • 27.

    Lee AJ, Goodman SR, Banks SE, Lin M, Landau R. Development of a Multiple-Choice Test for Novice Anesthesia Residents to Evaluate Knowledge Related to Management of General Anesthesia for Urgent Cesarean Delivery. J Educ Perioper Med. 2018;20(2). E621. [PubMed: 30057932]. [PubMed Central: PMCPmc6055537].

  • 28.

    Al-Rukban MO. Guidelines for the construction of multiple choice questions tests. J Family Community Med. 2006;13(3):125-33. [PubMed: 23012132]. [PubMed Central: PMCPmc3410060].

  • 29.

    Rangel RH, Moller L, Sitter H, Stibane T, Strzelczyk A. Sure, or unsure? Measuring students' confidence and the potential impact on patient safety in multiple-choice questions. Med Teach. 2017;39(11):1189-94. doi: 10.1080/0142159x.2017.1362103. [PubMed: 28799435].

  • 30.

    Quraishi MK, Khateeb Hanif U, Parmar R. Improvement in Confidence Levels for the Management of Paediatric Cardiac Arrests in Medical Students Following a Training Course. Anesth Pain Med. 2018;8(2). e14867. doi: 10.5812/aapm.14867. [PubMed: 30214880]. [PubMed Central: PMCPmc6119218].

  • 31.

    Lingenfelter BM, Jiang X, Schnatz PF, O'Sullivan DM, Minassian SS, Forstein DA. CREOG In-Training Examination Results: Contemporary Use to Predict ABOG Written Examination Outcomes. J Grad Med Educ. 2016;8(3):353-7. doi: 10.4300/jgme-d-15-00408.1. [PubMed: 27413437]. [PubMed Central: PMCPmc4936852].

  • 32.

    Zakus P, Gelb AW, Flexman AM. A survey of mentorship among Canadian anesthesiology residents. Can J Anaesth. 2015;62(9):972-8. doi: 10.1007/s12630-015-0418-8. [PubMed: 26087911].

  • 33.

    Gonzalez LS, Donnelly MJ. A survey of residency program directors in anesthesiology regarding mentorship of residents. J Clin Anesth. 2016;33:254-65. doi: 10.1016/j.jclinane.2016.03.004. [PubMed: 27555175].

  • 34.

    Ergun S, Busse JW, Wong A. Mentorship in anesthesia: a survey of perspectives among Canadian anesthesia residents. Can J Anaesth. 2017;64(4):402-10. doi: 10.1007/s12630-017-0816-1. [PubMed: 28092066].

  • 35.

    Perdana A, Nugroho AM, Ariadi A, Nari Lastri D. Cognitive and Psychomotor Function Changes Among Anesthesiology Residents After 12 Working Hours in Elective Anesthesia Service. Anesth Pain Med. 2016;6(1). e33071. doi: 10.5812/aapm.33071. [PubMed: 27047793]. [PubMed Central: PMCPmc4780391].

  • 36.

    Boysen P2, Daste L, Northern T. Multigenerational Challenges and the Future of Graduate Medical Education. Ochsner J. 2016;16(1):101-7. [PubMed: 27046415]. [PubMed Central: PMCPmc4795490].

  • 37.

    Hu YY, Mazer LM, Yule SJ, Arriaga AF, Greenberg CC, Lipsitz SR, et al. Complementing Operating Room Teaching With Video-Based Coaching. JAMA Surg. 2017;152(4):318-25. doi: 10.1001/jamasurg.2016.4619. [PubMed: 27973648].

  • 38.

    Crosby E, Lane A. Innovations in anesthesia education: the development and implementation of a resident rotation for advanced airway management. Can J Anaesth. 2009;56(12):939-59. doi: 10.1007/s12630-009-9197-4. [PubMed: 19847588].

  • 39.

    Meyerson SL, Lipnick S, Hollinger E. The Usage of Mock Oral Examinations for Program Improvement. J Surg Educ. 2017;74(6):946-51. doi: 10.1016/j.jsurg.2017.05.003. [PubMed: 28529196].

  • 40.

    Soleimanpour M, Rahmani F, Naghizadeh Golzari M, Ala A, Morteza Bagi HR, Mehdizadeh Esfanjani R, et al. Comparison of Electronic Learning Versus Lecture-based Learning in Improving Emergency Medicine Residents' Knowledge About Mild Induced Hypothermia After Cardiac Arrest. Anesth Pain Med. 2017;7(4). e57821. doi: 10.5812/aapm.57821. [PubMed: 29226111]. [PubMed Central: PMCPmc5712137].

  • 41.

    Fingeret AL, Arnell T, McNelis J, Statter M, Dresner L, Widmann W. Sequential Participation in a Multi-Institutional Mock Oral Examination Is Associated With Improved American Board of Surgery Certifying Examination First-Time Pass Rate. J Surg Educ. 2016;73(6):e95-e103. doi: 10.1016/j.jsurg.2016.06.016. [PubMed: 27663083].

  • 42.

    Busireddy KR, Miller JA, Ellison K, Ren V, Qayyum R, Panda M. Efficacy of Interventions to Reduce Resident Physician Burnout: A Systematic Review. J Grad Med Educ. 2017;9(3):294-301. doi: 10.4300/jgme-d-16-00372.1. [PubMed: 28638506]. [PubMed Central: PMCPmc5476377].

  • 43.

    Dewa CS, Loong D, Bonato S, Trojanowski L, Rea M. The relationship between resident burnout and safety-related and acceptability-related quality of healthcare: a systematic literature review. BMC Med Educ. 2017;17(1):195. doi: 10.1186/s12909-017-1040-y. [PubMed: 29121895]. [PubMed Central: PMCPmc5680598].

  • Copyright © 2020, Author(s). This is an open-access article distributed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International License (http://creativecommons.org/licenses/by-nc/4.0/) which permits copy and redistribute the material just in noncommercial usages, provided the original work is properly cited.
    COMMENTS

    LEAVE A COMMENT HERE: