The I.C.E. Exchange brings together credentialing professionals of all career stages and industries to connect and learn together. Access 16 select sessions from 2021 Exchange.
Remote Proctoring and Security: What Have We Learned?
Presenters: Carl Bowman, CompTIA; Corina Owens, Alpine Testing Solutions
Remote proctoring is a very relevant and timely topic of concern, both from certification validity and content security perspectives for many certification bodies. Further, remote proctoring will be used for the foreseeable future and will, at a minimum, continue to be a topic of research and ongoing concern throughout 2021. Hear high-level comparative results from the four psychometric studies, lessons learned from internal secret shops of the remote proctoring service, and recommendations to enhance future innovations in remote proctoring delivery. Remote proctoring services provide most of the essential security features required for high-stakes testing; however, there is still plenty of opportunity for growth and improvement.
Debating and Discussing 5 Controversial Credentialing Topics
Business of Certification, Level: Intermediate
Presenters: Cynthia Allen and Janice Moore, SeaCrest Consulting; Morgean Hirt, ATD Certification Institute; Shannon Carter, CAE, American College of Veterinary Internal Medicine; Mickie S. Rops, MAEd, FASAE, CAE, Mickie S. Rops Consulting LLC
Certification programs provide value to professionals, employers, associations, and the public. The certification business has evolved and meeting the needs of a diverse stakeholder base poses both strategic challenges and potential opportunities. In this session, a panel of certification leaders will discuss and debate 5 questions certifiers face including remote proctoring, micro-credentials, training & certification, accreditation, and struggling programs.
Going the Way of Blockbuster: Will Test Centers Become Extinct?
Test Development and Administration, Level: Intermediate
Presenters: Rachel R. Watkins Schoenig, Esg., Cornerstone Strategies LLC; Bill West, Examity; Jeff M. Marsh, Ascend Learning; Linda Adler, National Board of Medical Examiners; Stephanie Dille, ProctorU
We’ve spent years debating whether online proctoring is a viable delivery option for high-stakes assessments, including certification and licensure exams. The pandemic moved several organizations down that path more quickly than anticipated. Now that programs, and test-takers, know what it takes to test remotely, will it further alter our testing model to the point where test centers become used so seldom that they go the way of Blockbuster? Join testing professionals with both test center and online testing experience as they debate the future of test delivery
Hey! Why Doesn’t Anybody Look Like Me?
Business of Certification, Level: Introductory
Presenters: Mihaiela Ristei Gugiu, National Registry of Emergency Medical Technicians; Dawn Morton-Rias, EdD, PA-C, National Commission on Certification of Physician Assistants; Robin Rone, APRA International; Moderator: Beverly Black, National Commission on Certification for Physician Assistants
This session will focus on strategies for enhancing equity, diversity, and inclusion among testing and credentialing professionals. There are qualified professionals from minority groups such as black, indigenous, and people of color (BIPOC), people with disabilities, lesbian, gay, bisexual, queer, and transgender (LGBTQ+) etc.), but they are not equally represented in the credentialing profession.
Do those involved in examination development and administration currently reflect the diversity of the candidates? What organizational strategies would open the pipeline so that the professionals developing and administering credentialing exams reflect the diversity of the overall population? How can the industry attract more diverse people into the credentialing profession? Is there a systemic bias that creates a barrier to entry? How can the obstacles that interfere with obtaining the education and experience needed to qualify for positions in the credentialing industry be removed?
Success in the profession requires a range of knowledge and skills, including communication, research, data management, collaboration, team building, networking, project management, and leadership. Learn about organizational assessments to determine: the diversity and inclusiveness of the current staff and board; the organizational attitude about equity, diversity, and inclusion; and the commitment to diversity in recruitment to fill available positions. Learn how forging new alliances with historically black colleges and universities can be part of a deliberate effort to diversify the pool of candidates.
Testing Accommodations through Different Exam Delivery Modalities: Challenges and Opportunities
Test Development and Administration, Level: Introductory
Speakers: Ashley Norris, PhD, ProctorU; Carlie Davies, American Medical Certification Association; Dana Kelly, National Association of Student Financial Aid Administrators; Greg Pope, APEGA
Credentialing organizations are faced with increased volumes of requests for testing accommodations and must carefully balance the access needs of candidates with disabilities with their mission to ensure safe and effective practice. With the increased use of online proctoring as an exam delivery modality, as an alternative or in conjunction to test center delivery, there are a number of considerations to review and evaluate in providing testing accommodations.
The panel will discuss challenges and opportunities as it relates to testing accommodations through different modalities, including online proctoring and in test center, or from a hybrid offering. Key topics to be addressed include:
Examples will be discussed along with key applications from a program perspective, leveraging the experience from the three individuals representing different testing or regulatory organizations. This discussion will provide an opportunity to outline options for testing accommodations and appropriate parameters for reviewing accommodations requests, communicating decisions, and implementing accommodations while preserving the integrity of the testing program.
Lead change, audit yourself: a process review of content development DEI and bias reduction.
Test Development and Administration, Level: Intermediate
Presenters: Christine Mills, Ph.D.,and Kelly Von Lunen, Ascend Learning; Grace DeLee, MA
For decades test sponsors have employed bias review committees and DIF analysis to address bias in test content. With recent events in the United States infusing energy into how our culture addresses Diversity, Equity, and Inclusion, test sponsors are prompted to take a fresh look at their processes and ask, “How can we do better”?
In this session attendees will hear about the audit process and lessons learned from one organization’s effort to audit their diversity, equity, inclusion, and bias reduction (DEI-BR) processes.
Exploring the Return on Investment with Automated Item Generation: ROI of AIG
Presenters: Ada Woo, PhD, Ascend Learning; Xin Li, PSI Services; Christine Mills, Ascend Learning; Saad Khan, FineTune Learning
The COVID-19 pandemic has accelerated the push towards remote delivery of formative and summative assessments, further exacerbating longstanding security concerns of item exposure. One approach to mitigate these concerns would be to maintain a dynamically updated item pool that is continuously replenished with new items. Unfortunately, this is challenging to implement and scale up as traditional processes of creating assessment content rely in large part on manual labor and highly dependent on subject matter expertise.
In this session we explore the promise of artificial intelligence (AI) and automated item generation (AIG) to help address these challenges in assessment content creation and potentially opening up new opportunities for real-time and personalized assessments creation. We provide an overview of the current state of the art in AIG and how recent advances in AI are revolutionizing this domain. We discuss the potential return on investments with such tools and technologies in increasing item writer productivity and throughput.
We know we need a public member but why exactly? And how do we go about it?
Presenters: Karen Plaus, Ph.D., CRNA, FAAN, CAE, Consultant; Rebecca (Becky) LeBuhn, MA, Citizen Advocacy Center; Kevin Keller, CAE, Certified Financial Planner Board of Standards; Denise (Denny) Voigt Crawford, JD; Moderator: Catherine Dower
I.C.E. member boards usually need at least one public member, in part to meet accreditation requirements (such as those for ABSNC and NCCA). Some boards have long had public members and want to ensure their success. For other organizations, public board membership is a new concept. This session will explore the reasons behind encouraging or requiring public members on boards and also look at the pragmatic issues of:
This panel includes individuals with experience in the C-suite or as public members in organizations in health care and financial services that have strong onboarding and support processes. The panelists will be interviewed by a member of the Public Member Committee.
Demonstrating value: Efficacy research and proven results
Presenters: Elisa Kahn, PMP, American Management Association; Carla Caro, MA, ACT, Jason D. Way, PhD, ACT
We all believe in the value of credentials—or else we wouldn’t be here! It is not always the case, however, that organizations, industries, and practitioners around the world recognize and appreciate how credentialing can improve outcomes and advance a profession. Before investing time, energy, and funds in credentials stakeholders want to know—does the credential make a difference? What’s the benefit? This session will focus on how credentialing organizations can PROVE the value of their offerings and communicate this value to stakeholders. The implementation and integration of value proposition studies and efficacy research in credential development, maintenance, and marketing, can help support and substantiate organizations’ claims regarding their certification offerings.
The session will provide an introduction to the conduct of value proposition studies and efficacy research, including a discussion of strategic considerations and methodological approaches. The results of a recently completed efficacy and outcomes research study for the AMA Certified Professional in Management™ (AMA-CPM™) will then serve as an illustrative example of how credentialing organizations can identify, support, and prove the value of their credentials.
If we survive COVID, political unrest, racial bias and climate change, What will 2030 look like?!
Primary Presenter: Grady Colson Barnhill, MEd, National Commission on Certification of Physician Assistants; Co-Presenter 1: Denise Roosendaal, CAE, Institute for Credentialing Excellence; Co-Presenter 2: Tom Granatir, American Board of Medical Specialties; Co-Presenter 3: Marc Seale, A Prestigious UK University; Co-Presenter 4: Catherine Dower, JD, Catherine Dower Consulting
Going about our daily lives it is easy to focus on tasks at hand and what is required to complete them, but as members of the credentialing landscape, it is important that we sometimes step back to consider and evaluate how our programs intersect with the consumer, the public and society, and to anticipate how we might make improvements. Panelists have extensive experience in certification, licensure and regulation, but also in law, legislation and policy will use a discussion-style format to explore key credentialing, workforce and societal issues:
Re-credentialing Improvements: Through increased education and consumer awareness, the public, employers and other stakeholders may begin to demand more of credentialing processes. A more informed consumer may demand more meaningful recertification/renewal processes to provide practitioners with evidence-based tools to support better patient safety or other outcomes. Increased stakeholder awareness may help strengthen credentialing bodies’ credentialing maintenance processes, but only for activities that are relevant, cost-effective, meaningful and minimally burdensome. Longitudinal Assessment is likely to be a part of this landscape, which will represent a new trend – the marriage of assessment and education to not only document competence but to improve it at the same time.
Regulatory Reform: Increasing consumer awareness and a focus on public safety could fuel more interest in regulatory process, perhaps leading to improvements such as more public members on licensing or credentialing boards. Evaluation of risk could lead to improved regulatory structures with varying rigor reflective of public risk involved (e.g., more relaxed requirements for cosmetologists relative to nurses). Increased rigor could also be applied for individual practitioners shown to be at greater risk for harming the public or having poor outcomes, such as practitioners in isolated practice, those re-entering practice or those who have been in practice for a large number of years.
Credentialing Restructure: The drive for meaningful recertification may lead to more individualized processes that document expertise in specialty or areas of practice focus. Microcredentialing, stacked credentials and so on may facilitate increasingly diverse credentialing schemes within professions. As technology, knowledge and skills develop and expand, practitioners, by necessity may know more and more about less and less. Increased access to big data and AI may make it possible to “automatically” observe practitioner behaviors and help document practitioner competence. For example, access to Electronic Health Records could make it possible to verify physician compliance with best practices or positive health outcomes for a number of patients.
Covid Legacies: What will remain of credentialing changes implemented to survive 2020? More online proctoring, almost certainly, but what are the trade-offs in terms of risk to the public? Clinical rotations, internships or on-the-job training are hard to complete during a pandemic and are costly and time-consuming for both students and educators. Will these requirements be increasingly replaced by simulations? Will there be more regulatory flexibility with regard to scope of practice and reciprocity? Will state borders be more “fluid,” particularly during times of acute need? How will certification and licensing bodies improve their flexibility and responsiveness?
Diversity Issues: 2020 was a year of great unrest and increasing focus on racial bias, systemic racism and cultural issues. With the increased awareness one hopes we will have an increasing understanding of how these issues intersect with credentialing and how we can make improvements. In the credentialing testing space we are likely to see an increased focus on sensitivity reviews of test questions, more studies of bias (differential item functioning) and ways to meaningfully address testing bias. In the education space we will address admission strategies, recruitment and incentives to not only improve diversity in the professions but also improve access to care across the country.
New Measurement Considerations in the Era of Online High Stakes Testing
Presenters: John Weiner, PSI Services; Isabelle Gonthier, PhD, PSI Services; Timothy Joseph Muckle, NBCRNA
Online remote proctored testing is a growing trend that was accelerated by the pandemic, with social distancing and remote learning/work becoming predominant to the point that even accredited high-stakes exams began utilizing this approach. With this surge to remote online testing, concerns have been amplified regarding security and the psychometric soundness and comparability of test scores obtained in different modalities (e.g., traditional bricks and mortar testing centers vs. online testing at home). Psychometric methods have been in place for decades to evaluate measurement quality and we can assume many of these are still suitable for today’s new test delivery modalities. However, new challenges require taking a fresh look at these time-tested methods to ensure that they remain suitable for the task at hand. A review is needed to see whether new methods and models are required for modern testing environments that yield new types of information about candidate test taking behavior, especially in the era of “Big Data” collection and AI-assisted proctoring.
In this session, a panel of seasoned experts in credentialing testing will discuss the challenges posed by online high stakes testing, the suitability of proven psychometric methods and models, and potential new approaches and frameworks. This diverse panel is comprised of senior leaders and psychometricians from several certification organizations (medical, pharmacy, language testing), as well as a global assessment services company responsible for multimodal delivery of many thousands of high stakes examinations. Session attendees will be encouraged to participate in the discussion to share perspectives and engage with the credentialing community.
Revenue is not a Four Letter Word: Pricing and Revenue Stability for Financial Sustainability
Presenters: Terri Hinkley, EdD, MBA, BSN, RN, CAE, Medical-Surgical Nursing Certification Board; Carlos Restrepo, Agilutions; Michael Tatonetti, Pricing for Associations
Certification boards have not traditionally diversified their revenue models. 2020 saw many certification boards financially impacted because of global testing suspension. Further, increasing competition from for-profit entities are challenging certification programs. Associations have long embraced non-dues revenue streams for financial sustainability and provide a model for certification boards to examine ways to expand their revenue through existing product offerings. This session will explore how certification boards can evaluate existing products/opportunities and coalesce them into revenue streams for the organization.
Tell me how you really feel about it: What’s trending in recertification/renewal?
Business of Certification, Level: intermediate
Presenters: Kevin M. Bradley, Human Resources Research Organization; Michael R. Peabody, National Association of Boards of Pharmacy
Over the past few years, the presenters have been exploring recertification and renewal practices. At a previous I.C.E. Exchange we shared the results of a survey administered during the first phase of our work. When the presenters conducted a live poll during the session, they noted significant differences between survey and session responses. Here, we present qualitative results of a follow-up survey to highlight the changing landscape of recertification and renewal requirements, especially in light of changes triggered by the COVID-19 pandemic. Specific organizations were targeted to represent diverse industry sectors.
Virtual Standard Setting Models for Success
Presenters: Charis Walikonis, Prometric; Jillian Nelson, Prometric; Stephen Fletcher, American Board for Certification in Orthotics, Prosthetics & Pedorthics; Natalya Mytareva, Certification Commission for Healthcare Interpreters
No matter the context or complexity of your standard setting, you can successfully undertake a virtual process! This session will discuss several models of virtual standard setting from a variety of perspectives, including customized processes for unique situations. You will learn tips for overcoming common challenges and pitfalls that you can apply to your program, as well as key best practices to help you maintain the validity, reliability, or fairness of the process.
Benefits and Risks of Innovation to the Validity Argument of Credentialing Examinations
Credentialing Innovations, Level: Intermediate
Presenters: James P Henderson, PhD., Credentialing Examination Consulting, LLC; Chad W. Buckendahl, ACS Ventures, LLC; Mark Carlos Franco, Esq., Whiteford, Taylor & Preston, L.L.P.; Sarah Carroll, National Board for Certification in Occupational Therapy, Inc.
With the current emphasis on innovation in credentialing, many organizations feel compelled to try new methods as they work to build the validity argument for their high stakes credentialing examination. An important question concerns how to weigh the benefit of innovation against the risk it may pose to the outcome of key projects and the case for validity. This session examines this question from the vantage point of law, accreditation, and research.
UPDATE: CERTIFICATION IN THE EMERGING NORMAL
Presenters: Denise Roosendaal, CAE, Institute for Credentialing Excellence; Tom Granatir, American Board of Medical Specialties; David Swankin, Esq., Citizen Advocacy Center; Sally Irvin,
This session builds upon well-received “Updates” at prior ICE Exchanges which raised awareness of legislation and litigation affecting certification and recertification. This year, our expert panel will also explore how COVID19 and the election are impacting the regulatory and legislative climate and disrupting established models for training and assessment. They will look at ways in which a rekindled focus on public health affects certification, business decisions, and employment. They will comment on how changing dynamics shape the strategic choices confronting ICE members’ boards of directors and how attention to diversity, equity, and inclusion challenge business-as-usual.