2021 R&D Virtual Summit
Research Snapshot: A Look at I.C.E. Research
Wednesday, October 20th | 1:30 pm - 3:00 pm ET
Join I.C.E.'s Research and Development Committee to discuss topics, projects, and activities the committee has been pursing through the year. This is your opportunity to ask questions, provide real-time feedback, and influence the research I.C.E. publishes. This year’s virutal session will be comprised of two parts: a presentation followed by roundtable discussions for each topic. The roundtables provide a deeper dive into the topics that resonate best with your interests.
Current research projects, presenters and roundtable descriptions are listed below.
A recording of the R&D Summit will be posted.
Presenters:
- Chris Beauchamp, Yardstick Assessment Strategies
- Kirk Becker, Pearson VUE
- Misty Bloom, Behavior Analyst Certification Board
- Kevin Bradley, HumRRO
- Susan Davis-Becker, ACS Ventures
- Kari Hodge, The American Board of Radiology
- Pat Muenzen, ACT
- Denise Roosendaal, I.C.E.
- Priya Santhanam, SAP Concur
- Patricia Young, Kryterion
- Tony Zara, Pearson VUE
Eligibility Criteria
Most credentials have eligibility criteria or prerequisites that candidates must meet as part of the certification process, in addition to passing a required examination/assessment. Common eligibility criteria include education or training, work experience, and a commitment to upholding the integrity of the profession. Eligibility criteria are commonly used to measure attributes, competencies, or background experiences that are central to the certification construct but are not efficiently or effectively assessed by the certification examination/assessment. All eligibility criteria are important in determining whether or not a person should be certified, and there is (and should be) a great deal of thought that goes into establishing the eligibility criteria. The procedures and processes for establishing eligibility criteria vary widely, and organizations might struggle to determine the appropriate methodology for setting eligibility criteria.
This I.C.E. R&D project will:
- Identify common eligibility criteria and rationales for eligibility criteria for certifications in different industries/professions
- Identify and describe common procedures organizations follow for establishing, reviewing, and updating eligibility criteria
- Identify best practices for establishing, reviewing, and updating eligibility criteria (including methods for validating criteria and alternate eligibility criteria)
Roundtable Session Description
Representatives of the Eligiblity Criteria task force will share preliminary survey results as context for facilitated discussion with attendees. Task force members will explore topics in greater detail with participants with a particular focus on topics that do not lend themselves well to a survey questionnaire format.
Job Analysis
Job Analysis is an important part of the validity argument for credentialing programs. As Myers, Morral, & Sares (2019) point out, a mixed-methods approach over a 3 to 7-year time frame is most useful for establishing content validation and currency. We are interested in documenting the approach of credentialing organizations to job analysis, evaluating common industry practices, and exploring new technologies that may be applied to job analysis.
This study will survey current practices of the credentialing field regarding frequency, format, and operational considerations, as well as targeted and open-ended questions about new approaches being explored. For survey-based job analyses, we will solicit information (specifically data or specific analyses) on the rating categories used, their independence, and the utility of collecting SME ratings.
Roundtable Session Description
We have completed our survey of current JTA practices of the credentialing field. Questions were asked regarding frequency, format, operational considerations, and new methods and technologies implemented within JTAs. The JTA R&D committee will present the initial results of the survey, and attendees are encouraged to discuss how the results compare with their processes, as well as the implications of the future technologies raised.
Forensic Data Analysis
Forensic data analysis refers to the examination of structured test data for the purpose of identifying any fraudulent behaviors (e.g., collaboration, testing with prior knowledge of the content, stealing content). Our goal is to conduct research that can be used by professionals within credentialing to better understand the purpose of forensic data analysis, evaluate and select analysis options based on the needs of their program, and determine how the findings from these can inform decision making.
Roundtable Session Description
This task force will present their research on forensic data analysis techniques and practices that were identified through a review of the literature and a comprehensive survey of members of the credentialing community. This research resulted in a proposed framework linking various FDA techniques to suspicious or aberrant candidate behaviors. Session attendees will be asked to share feedback as to how the framework captures the security concerns and analysis practices of their organization.
Subject Matter Expert (SME) Practices in Credentialing
It would be impossible to develop or maintain credentialing examinations without the expertise of subject matter experts (SMEs). Their expertise is required in every step of the examination development process. Selecting an SME group that is representative of the target audience for the credential is required to ensure the quality and defensibility of credentialing examinations.
This task force is studying all practices related to subject matter experts (SMEs) involved in the development or maintenance credentialing examinations (e.g., recruitment, selection, training, evaluation, engagement, succession planning).
This R&D project will:
- Identify common selection criteria (qualifications, demographics, personality?) and rationales for the selection criteria for SMEs in various industries/professions
- Identify evidence to demonstrate that SMEs are representative of target audience to accreditation bodies and make-up of SMEs is appropriate for exam development activity
- Identify and describe common policies and procedures organizations follow for recruiting and selecting SMEs (includes NDAs [Standard 3], forms, on-boarding)
- Identify best practices for selecting SMEs
- Identify common methods/components for training SMEs
- Identify best practices for managing and evaluating SMEs (include duration of term, compensation)
- Identify tips and tricks for working with SME groups during various activities
Roundtable Session Description
SME Practices in Credentialing breakout session will be a focus group discussing a study topic in greater depth (e.g., practices for evaluating SME, changes being made to SME practices that will continue post-pandemic). Participants will also have an opportunity to provide input on what they find most useful in the project work product (e.g., best practices, common practices, case studies or examples of interesting or successful programs).
Remote Proctoring
So, you’re going remote? Here is what you should consider about the candidate's perspective and experience. While test security, psychometrics and validity are all important aspects to consider when deciding if Live Remote Proctoring (LRP) is a viable option for your organization, another important consideration is the candidates themselves.
This ICE R&D project will
- Identify the test security/experience considerations for candidates
- Identify what information candidates need to know
- Identify what the organization should consider
- types of exams- written computer-based exam, standardized conditions that don't require the physical presence of another person.
- item type- with large files or images
- system requirements- should be set to accommodate the items and assets, system check should check that the exam will be delivered properly.
Roundtable Session Description
The Live Remote Proctoring project aims to examine the candidates’ experiences and perspectives with the emerging delivery modality. These perspectives include new security and privacy considerations as well as additional rules and protocols to ensure outcome comparability with other delivery modes. Please join our session to hear our progress, participate in the discussion, and share your perspectives and experiences!
Test Disruptions and Impacts on Test Validity
The purpose of this project is to examine the role of test disruption on the validity and interpretation of credentialing assessment results. Disruptions can come in many forms and can include many ranges of impact. Some disruptions are intended (e.g., new test delivery model) while others are unintended (e.g., network failure during examination administration). Nevertheless, the methodologies and tools required to establish test validity should be consistent and based upon generally accepted principles
Roundtable Session Description
Test disruptions comes many forms and includes a range of impacts. The taskforce built a framework and conducted interviews to learn more. Five main categories were identified with classifications for planned and unplanned disruptions. Join our taskforce discussion to hear our progress, participate in the discussion, and share your insight and experiences.
Thank you to the 2021 I.C.E. Research and Development Committee!
Chair
Pat Muenzen, PhD, ACT
|
Vice Chair
Tony Zara, PhD, Pearson VUE
|
Committee Members
|
Chris Beauchamp, PhD, Meazure Learning |
|
Christine Mills, PhD, Ascend Learning
|
Kirk Becker, PhD, Pearson VUE
|
|
Tim Muckle, PhD, NBCRNA
|
Kevin Bradley, PhD, Human Resources Research Organization
|
|
Lisa Nepi, EdM, SeaCrest Company |
Daniel Breidenbach, PhD, PSI Services
|
|
Nicole Risk,PhD, American Medical Technologists
|
Susan Davis-Becker, PhD, ACS Ventures LLC
|
|
Priya Santhanam, SAP Concur
|
Tony Ellis, MSEd, CAE, Oncology Nursing Certification Corporation
|
|
Pooja Shivraj, MS, PhD, American Board of Obstetrics and Gynecology
|
Matthew Ferris, MA, MBA, ELS, NBCRNA
|
|
John Weiner, MA, PSI Services |
Michelle Gross, MBA, MEd, NCC, NCSC, LPC, Metacred
|
|
Linda Waters, PhD, Prometric
|
James Hellrung, MA, National Center for Competency Testing
|
|
Patricia Young, MA, Kryterion Inc. |
Kari Hodge, PhD, NACE International
|
|
|