Validation and Fairness Threats When Evaluating Programs Against Standards
Track: Test Development and Administration
Over the past few years, the presenter has conducted a number of evaluations for various credentialing programs to determine whether they meet validity and fairness standards. Often credentialing organizations often evaluate their programs to determine their readiness of meeting accreditation standards such as ISO/IEC 17024:2012 or NCCA’s Standards for the Accreditation of Certification Program. While these accreditation standards are useful, the accreditation standards are often general and vague leaving credentialing organizations to determine the detailed methodologies and activities to meet these standards. Consequently, these credentialing organizations often focus on meeting these standards while unintentionally ignoring other resources such as evidence based materials (Handbook of Test Development) and other testing and research standards (The Standards for Educational and Psychological Testing, AAPOR’s Standards Definition, NCES Statistical Standards). Therefore, in an attempt to meet these accreditation standards (ISO and NCCA) and ignoring others related to improving a validity and fairness claim, credentialing organizations often share many common threats to the validity and fairness.
In this presentation, the presenter will discuss his own experience of evaluating these programs and discuss the common threats to validity and fairness across the psychometric/test development lifecycle (including maintenance) and program and management processes of credentialing programs. Examples of the common threats will include SME representation, the required number of SMEs for various activities, defining a detailed scope, lack of survey based job analysis, and lack of policies such as security and appeals policies. It is the hope that the audience members will have an opportunity to learn about these common validation and fairness threats to their own program to potentially strengthen their validity and fairness claim of their program.