An Examination of Reliability and Validity Claims of a Foreign Language Proficiency Test

Abstract

This dissertation study examined the reliability and validity claims of a modified version of the Spanish Modern Language Association Foreign Language Proficiency Test for Teachers and Advanced Students administered at George Mason University (GMU). The study used the 1999 computerized GMU version that was administered to 277 test-takers via WebCT, (a web course management tool), and not the original 1961 pencil-and-paper edition. The GMU examination retained five sections from the test’s original seven, namely Speaking, Listening, Reading, Writing, and Culture and Civilization. The original response formats, i.e. multiple-choice and true-false, were preserved for the most part. The Writing section was changed from the original error correction and text editing format to a fill-in-the-blanks format. The Culture and Civilization section was re-written due to obsolescence; however it retained the original answer format, to which a matching component was added. The study’s design was framed by the investigative steps proposed by Messick (1989) and Nitko and Brookhart (2007). An ex post facto approach was used, which mixed the confirmatory factor analysis and multiple regression analyses with descriptive interpretations. The quantitative analyses examined the impact of nationality, gender and average time spent on completing the tasks on the five subscales of the Spanish GMU MLA examinations. The analyses provided mixed results related to the predictors’ ability to explain the subscale scores. The descriptive analyses revealed numerous validity flaws and gaps in the test construction. Such inadequacies overshadowed the positive aspects and eventually led to my questioning of the hypothesis that the judgments based on data from the Spanish GMU MLA examinations are reliable and valid. Recommendations for possible revisions were made, along with suggestions for future research avenues.