Making Your Experience Requirement Meaningful
Track: Test Development and Administration
This e-poster session focuses on a large-scale, international credentialing body’s efforts to operationalize the hours of experience requirement established for its certification program – moving the requirement from simple quantification of time spent on the job to the qualitative evaluation of actual performance on the job within each of five domains of practice as defined and validated through a role delineation study.
Through the implementation of a panel review process, trained, certified assessors systematically review and rate candidate essays describing their experience to authenticate the candidate’s eligibility as an experienced practitioner. The assessors use a comprehensive rubric, including indicators, attributes, and behavioral descriptors, to inform their ratings of the candidates. A panel of subject-matter experts, including more- and less-experienced practitioners from around the world, developed the rubric and set performance standards.
Because the performance of the role varies as a function of the size of the organization, the nature of the industry and the geographic region in which the organization operates, the specific actions of candidates may seem quite dissimilar. The structured evaluation can be used to determine whether the candidate is truly performing tasks at the level of responsibility required in the role. Each of the candidate essays captures unique and non-overlapping behaviors and is judged independently by the assessors. Only after passing the experience summary hurdle is the candidate determined to be eligible to take the written certification examination.
The e-poster will include a description of: the standard setting process used to set the bar for performance on each essay and overall and the procedures used to calibrate the ratings of assessors.
The e-poster will provide attendees with insights from several perspectives including:
- the credentialing sponsor who needed to determine whether potential candidates from around the world, in different industries, and with different job titles and levels of experience were actually qualified to sit for the certification examination; and
- a testing services partner who developed a psychometrically sound and defensible decision-making system.
Objectives— Attendees will learn how to
- implement a method to meaningfully operationalize experience requirements that are typically described as years/hours of experience in the role
- create a rubric that permits the standardized and objective assessment of a candidate’s eligibility to sit for an examination
- create a standard for making go/no decisions regarding candidates' eligibility to sit for an examination.
- provide your organization with a methodology to evaluate its eligibility requirements, including experience and education.