Developing and validating listening and speaking tests of English for tourism and hospitality in Indonesia
Loading...
Date
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
Universitas Islam Internasional Indonesia
Publisher DOI
Volume
Issue
Resources
Total Views: 0Total Downloads: 0
download count data not available for this item.
Abstract
This dissertation presents the development and validation of an English listening and speaking test for tourism and hospitality workers in Indonesia. This study is motivated by the growing communicative needs of tourism and hospitality professionals, where a fluent English language is essential for service delivery as well as for guest satisfaction and work interaction in multilingual and multicultural settings. However, English assessment practices in Indonesia have been based on general or academic-purpose proficiency tests such as TOEFL, TOEIC, IELTS, and locally-run TOEFL-like prediction tests that do not fit workplace communication and are not consistent with construct relevance, validity, and fairness. To address the three research questions, the study addresses the identification of important constructs of workplace communication in tourism and hospitality, the validity of context in the design of the tests and how well psychometric evidence supports defensible score interpretation. A unified theory of Weir socio-cognitive model (2005), BEAR assessment system (2023), and Bachman's Assessment Use Argument (2010) are applied to the study in an exploratory sequential mixed method design. The construct is developed with literature review and expert judgment from experts in language assessment, English for Specific Purposes and tourism practice who are used to help us develop task design, response formats and scoring criteria. I consider listening ability as identification, inference and reflective judgment while speaking ability as description, handling and simulation tasks with increasing complexity of communication. I use semi-direct and technology-based test formats and evaluate them by Rasch analysis and Many-Facet Rasch Model. I find that I have a well-functioning Rasch scale and the listening test has item difficulty ranging from -3.39 to +3.38 logits, item separation of 13.37, reliability of .99 and fit are generally acceptable, with mean infit and outfit MNSQ values of 0.98 and 1.05. The speaking test also has a well-functioning Rasch scale and has 59.4% variance explained by the Rasch dimension, it has item separation of 4.54, reliability of .95 and criterion difficulties ranging from -0.60 to +0.58 logits. Strategic competence was the most difficult criterion for speaking, with task completion being the easiest. The differences in severity were small (ranging from -0.18 to +0.18 logits) and were statistically estimated by the Many Facet Rasch Model (MFRM). DIF and bias analyses showed that fairness concerns were in fact mainly limited to a few packages, item formats and strategic skills rating. In the context of the Assessment Use Argument (AUA), I believe that the results support cautious but reasonable assertions about the quality of workplace communication measurement in tourism and hospitality settings. The study concludes with limitations regarding sample size, occupational scope and construct boundaries, and acknowledges the need for further research. And the dissertation shows that a construct-driven theory-informed and empirically grounded approach can be a valid and fair alternative to general-purpose English proficiency tests for specialized professional domains.
Description
Citation
Collections
Endorsement
Review
Supplemented By
Referenced By
License
Except where otherwised noted, this item's license is described as All Rights Reserved
