Catálogo de publicaciones - libros

Measurement involves the processes of description and quantification. Questionnaires and test instruments are designed and developed to measure conceived variables and constructs accurately. Validity and reliability are two important characteristics of measurement instruments. Validity consists of a complex set of criteria used to judge the extent to which inferences, based on scores derived from the application of an instrument, are warranted. Reliability captures the consistency of scores obtained from applications of the instrument. Traditional or classical procedures for measurement were based on a variety of scaling methods. Most commonly, a total score is obtained by adding the scores for individual items, although more complex procedures in which items are differentially weighted are used occasionally. In classical analyses, criteria for the final selection of items are based on internal consistency checks. At the core of these classical approaches is an idea derived from measurement in the physical sciences: that an observed score is the sum of a true score and a measurement error term. This idea and a set of procedures that implement it are the essence of Classical Test Theory (CTT). This chapter examines underlying principles of CTT and how test developers use it to achieve measurement, as they have defined this term. In this chapter, we outline briefly the foundations of CTT and then discuss some of its limitations in order to lay a foundation for the examples of objective measurement that constitute much of the book.

The Rasch model is described as fundamental to objective measurement and this chapter examines key issues of conceptualising variable, especially in education. The notion of inventing units for objective measurement is discussed, and its importance in developmental assessment is highlighted.

This Chapter explains the Rasch model for ordered response categories by demonstrating the latent response structure and process compatible with the model. This is necessary because there is some confusion in the interpretation of the parameters and the possible response process characterized by the model. The confusion arises from two main sources. First, the model has the initially counterintuitive properties that (i) the values of the estimates of the thresholds defining the boundaries between the categories on the latent continuum can be reversed relative to their natural order, and (ii) that adjacent categories cannot be combined in the sense that their probabilities can be summed to form a new category. Second, two models at the level of a single person responding to a single item, the so called and , have been portrayed as being different in the response structure and response process compatible with the model. This Chapter studies the structure and process compatible with the Rasch model, in which subtle and unusual distinctions need to be made between the and of response probabilities and between and relationships. The Chapter demonstrates that the response process compatible with the model is one of classification in which a response in any category implies a latent response at every threshold. The Chapter concludes with an example of a response process that is compatible with the model and one that is incompatible.

This paper is concerned with the analysis and scaling of mathematics achievement data over time by applying the Rasch model using the QUEST (Adams & Khoo, 1993) computer program. The mathematics achievements of the students are brought to a common scale. This common scale is independent of both the samples of students tested and the samples of items employed. The scale is used to examine the changes in mathematics achievement of students in Australia over 30 years from 1964 to 1994. Conclusions are drawn as to the robustness of the common scale, and the changes in students’ mathematics achievements over time in Australia.

Users of statistical software are frequently unaware of the calculations underlying the routines that they use. Indeed, users—particularly in the social sciences—are often somewhat adverse towards the underlying mathematics, Yet, in order to appreciate the thrust of certain routines, it is beneficial to understand the way in which a program arrives at a particular solution. Based on data from the Economic Literacy Study conducted at year 11 and 12 level across Queensland in 1998, this article renders explicit the steps involved in calculating growth and gain estimates in student performance. To this end, the first part of the article describes the Omanual calculation of such estimates using the Rasch estimates of item thresholds of common items at the different year levels produced by Quest (Adams & Khoo, 1993) as a starting point for the subsequent calibrating, scoring and equating. In the second part of the chapter, we explore the extent to which estimates of change in performance across year levels that are calculated with ConQuest (Wu, Adams & Wilson, 1997).

The article shows that the manual and automatic way of calculating growth and gain estimates produce nearly identical results. This is not only reassuring from a technical point of view but also from an educational point of view as this means that the reader of the non-mathematical discussion of the manual calculation procedure will develop a better understanding of the processes involved in calculating growth and gain estimates.

This study attempted to evaluate outcome of foreign language teaching by measuring the reading and writing proficiency achieved by students studying Japanese as a second language in six different year levels from year 8 to the first of university in the classroom setting. In order to measure linguistic gains across six years, it was necessary, firstly, to define operationally what reading and writing proficiency was; and secondly, to create measuring instruments, and, thirdly, to identify? suitable statistical analysis procedures.

The study sought to answer the following research questions: Can reading and writing performance in Japanese as a foreign language be measured?; and Does reading and writing performance in Japanese form a single dimension on a scale?

The participants of this project were drawn from one independent school and two universities, while the instruments used were the routine tests produced and marked by the teachers. The estimated test scores of the students calculated indicated that the answers to all research questions are in the affirmative. In spite of some unresolved issues and limitations the results of the study indicated a possible direction and methods to commence an evaluation phase of foreign language teaching. The study also identified the Rasch model as not only robust measuring tools but also as capable of identifying grave pedagogical issues that should not be ignored.

The Rasch model is employed to measure students’ achievement in leading Chinese as a second language in an Australian school. Comparison between occasions and between year levels were examined. The performance in Chinese achievement tests and English word knowledge tests are discussed. The chapter highlights the challenges of equating multiple tests across levels and occasions.

In this study, two common techniques for detecting biased items based on Rasch measurement procedures are demonstrated. One technique involves an examination of differences in threshold values of items among groups and the other technique involves an examination of fit of item in different groups.

Focus groups conduced with undergraduate students revealed general concerns about marker variability and the possible impact on examination results. This study has two aims: firstly, to analyse the relationships between student performance on an essay style examination, the questions answered and the markers; and, secondly, to identify and determine the nature and the extent of the marking errors on the examination. These relationships were analysed using two commercially available software packages, RUMM and ConQuest to develop the Rasch test model. The analyses revealed minor differences in item difficulty, but considerable inter-rater variability. Furthermore, intra-rater variability was even more pronounced. Four of the five common marking errors were also identified.

Four sets of analyses were conducted on the 1996 Course Experience Questionnaire data. Conventional item analysis, exploratory factor analysis and confirmatory factor analysis were used, Finally, the Rasch measurement model was applied to this data set. This study was undertaken in order to compare conventional analytic techniques with techniques that explicitly set out to implement genuine measurement of perceived course quality. Although conventional analytic techniques are informative, both confirmatory factor analysis and in particular the Rasch measurement model reveal much more about the data set, and about the construct being measured. Meaningful estimates of individual students’ perceptions of course quality are available through the use of the Rasch measurement model. The study indicates that the perceived course quality construct is measured by a subset of the items included in the CEQ and that seven of the items of the original instrument do not contribute to the measurement of that construct. The analyses of this data set indicate that a range of analytical approaches provide different levels of information about the construct. In practice, the analysis of data arising from the administration of instruments like the CEQ would be better undertaken using the Rasch measurement model.