2021 R&D Virtual Summit

Research Snapshot: A Look at I.C.E. Research

Wednesday, October 20th | 1:30 pm - 3:00 pm ET

Join I.C.E.'s Research and Development Committee to discuss topics, projects, and activities the committee has been pursing through the year. This is your opportunity to ask questions, provide real-time feedback, and influence the research I.C.E. publishes. This year’s virutal session will be comprised of two parts: a presentation followed by roundtable discussions for each topic. The roundtables provide a deeper dive into the topics that resonate best with your interests.

Current research projects, presenters and roundtable descriptions are listed below.


A recording of the R&D Summit will be posted.


  1. Chris Beauchamp, Yardstick Assessment Strategies 
  2. Kirk Becker, Pearson VUE
  3. Misty Bloom, Behavior Analyst Certification Board
  4. Kevin Bradley, HumRRO
  5. Susan Davis-Becker, ACS Ventures 
  6. Kari Hodge, The American Board of Radiology
  7. Pat Muenzen, ACT
  8. Denise Roosendaal, I.C.E.
  9. Priya Santhanam, SAP Concur
  10. Patricia Young, Kryterion
  11. Tony Zara, Pearson VUE

Eligibility Criteria

Most credentials have eligibility criteria or prerequisites that candidates must meet as part of the certification process, in addition to passing a required examination/assessment. Common eligibility criteria include education or training, work experience, and a commitment to upholding the integrity of the profession. Eligibility criteria are commonly used to measure attributes, competencies, or background experiences that are central to the certification construct but are not efficiently or effectively assessed by the certification examination/assessment. All eligibility criteria are important in determining whether or not a person should be certified, and there is (and should be) a great deal of thought that goes into establishing the eligibility criteria. The procedures and processes for establishing eligibility criteria vary widely, and organizations might struggle to determine the appropriate methodology for setting eligibility criteria.

This I.C.E. R&D project will:

  1. Identify common eligibility criteria and rationales for eligibility criteria for certifications in different industries/professions
  2. Identify and describe common procedures organizations follow for establishing, reviewing, and updating eligibility criteria
  3. Identify best practices for establishing, reviewing, and updating eligibility criteria (including methods for validating criteria and alternate eligibility criteria)

Roundtable Session Description

Representatives of the Eligiblity Criteria task force will share preliminary survey results as context for facilitated discussion with attendees. Task force members will explore topics in greater detail with participants with a particular focus on topics that do not lend themselves well to a survey questionnaire format.

Job Analysis 

Job Analysis is an important part of the validity argument for credentialing programs. As Myers, Morral, & Sares (2019) point out, a mixed-methods approach over a 3 to 7-year time frame is most useful for establishing content validation and currency. We are interested in documenting the approach of credentialing organizations to job analysis, evaluating common industry practices, and exploring new technologies that may be applied to job analysis.

This study will survey current practices of the credentialing field regarding frequency, format, and operational considerations, as well as targeted and open-ended questions about new approaches being explored. For survey-based job analyses, we will solicit information (specifically data or specific analyses) on the rating categories used, their independence, and the utility of collecting SME ratings.

Roundtable Session Description

We have completed our survey of current JTA practices of the credentialing field. Questions were asked regarding frequency, format, operational considerations, and new methods and technologies implemented within JTAs. The JTA R&D committee will present the initial results of the survey, and attendees are encouraged to discuss how the results compare with their processes, as well as the implications of the future technologies raised.  

Forensic Data Analysis 

Forensic data analysis refers to the examination of structured test data for the purpose of identifying any fraudulent behaviors (e.g., collaboration, testing with prior knowledge of the content, stealing content). Our goal is to conduct research that can be used by professionals within credentialing to better understand the purpose of forensic data analysis, evaluate and select analysis options based on the needs of their program, and determine how the findings from these can inform decision making.

Roundtable Session Description

This task force will present their research on forensic data analysis techniques and practices that were identified through a review of the literature and a comprehensive survey of members of the credentialing community. This research resulted in a proposed framework linking various FDA techniques to suspicious or aberrant candidate behaviors. Session attendees will be asked to share feedback as to how the framework captures the security concerns and analysis practices of their organization.   

Subject Matter Expert (SME) Practices in Credentialing

It would be impossible to develop or maintain credentialing examinations without the expertise of subject matter experts (SMEs). Their expertise is required in every step of the examination development process. Selecting an SME group that is representative of the target audience for the credential is required to ensure the quality and defensibility of credentialing examinations.

This task force is studying all practices related to subject matter experts (SMEs) involved in the development or maintenance credentialing examinations (e.g., recruitment, selection, training, evaluation, engagement, succession planning). 

This R&D project will:

  • Identify common selection criteria (qualifications, demographics, personality?) and rationales for the selection criteria for SMEs in various industries/professions
  • Identify evidence to demonstrate that SMEs are representative of target audience to accreditation bodies and make-up of SMEs is appropriate for exam development activity
  • Identify and describe common policies and procedures organizations follow for recruiting and selecting SMEs (includes NDAs [Standard 3], forms, on-boarding)
  • Identify best practices for selecting SMEs
  • Identify common methods/components for training SMEs
  • Identify best practices for managing and evaluating SMEs (include duration of term, compensation)
  • Identify tips and tricks for working with SME groups during various activities

Roundtable Session Description

SME Practices in Credentialing breakout session will be a focus group discussing a study topic in greater depth (e.g., practices for evaluating SME, changes being made to SME practices that will continue post-pandemic). Participants will also have an opportunity to provide input on what they find most useful in the project work product (e.g., best practices, common practices, case studies or examples of interesting or successful programs).

Remote Proctoring 

So, you’re going remote? Here is what you should consider about the candidate's perspective and experience. While test security, psychometrics and validity are all important aspects to consider when deciding if Live Remote Proctoring (LRP) is a viable option for your organization, another important consideration is the candidates themselves.

This ICE R&D project will

  • Identify the test security/experience considerations for candidates
  • Identify what information candidates need to know
  • Identify what the organization should consider
    • types of exams- written computer-based exam, standardized conditions that don't require the physical presence of another person.
    • item type- with large files or images
    • system requirements- should be set to accommodate the items and assets, system check should check that the exam will be delivered properly.

Roundtable Session Description

The Live Remote Proctoring project aims to examine the candidates’ experiences and perspectives with the emerging delivery modality. These perspectives include new security and privacy considerations as well as additional rules and protocols to ensure outcome comparability with other delivery modes.  Please join our session to hear our progress, participate in the discussion, and share your perspectives and experiences!

Test Disruptions and Impacts on Test Validity 

The purpose of this project is to examine the role of test disruption on the validity and interpretation of credentialing assessment results. Disruptions can come in many forms and can include many ranges of impact. Some disruptions are intended (e.g., new test delivery model) while others are unintended (e.g., network failure during examination administration). Nevertheless, the methodologies and tools required to establish test validity should be consistent and based upon generally accepted principles

Roundtable Session Description

Test disruptions comes many forms and includes a range of impacts. The taskforce built a framework and conducted interviews to learn more. Five main categories were identified with classifications for planned and unplanned disruptions. Join our taskforce discussion to hear our progress, participate in the discussion, and share your insight and experiences.

Thank you to the 2021 I.C.E. Research and Development Committee!


Pat Muenzen, PhD, ACT 

Vice Chair

Tony Zara, PhD, Pearson VUE

Committee Members
Chris Beauchamp, PhD, Meazure Learning  

Christine Mills, PhD, Ascend Learning

Kirk Becker, PhD, Pearson VUE


Tim Muckle, PhD, NBCRNA

Kevin Bradley, PhD, Human Resources Research Organization

  Lisa Nepi, EdM, SeaCrest Company

Daniel Breidenbach, PhD, PSI Services


Nicole Risk,PhD, American Medical Technologists


Susan Davis-Becker, PhD, ACS Ventures LLC


Priya Santhanam, SAP Concur

Tony Ellis, MSEd, CAE, Oncology Nursing Certification Corporation


Pooja Shivraj, MS, PhD, American Board of Obstetrics and Gynecology

Matthew Ferris, MA, MBA, ELS, NBCRNA

  John Weiner, MA, PSI Services

Michelle Gross, MBA, MEd, NCC, NCSC, LPC, Metacred


Linda Waters, PhD, Prometric

James Hellrung, MA, National Center for Competency Testing

  Patricia Young, MA, Kryterion Inc.

Kari Hodge, PhD, NACE International