What does a score of 78 mean? How does a percentile rank differ from a percentage? And why does the psychologist write two more numbers on either side of a score? In this chapter we provide the keys for reading assessment reports of every kind, and we show what a number can tell us and where its meaning stops.
Umm Majed sits in the school meeting with a six-page report in front of her. She reads: “Full-scale score 72, percentile rank 3, confidence interval 65 to 79.” She looks up and asks the teacher: “Does this mean my son passed with three percent? Does he have a disability or not?”
The teacher hesitates. He knows the number is not a pass rate, but he does not know how to explain that, nor why the report lists two additional numbers around the score.
We return to Umm Majed at the end of the chapter to give her two questions a precise answer. The concepts presented here recur throughout the category chapters, for they are the shared language in which every diagnostic report is written.
After studying this chapter, the reader should be able to:
Assessment results reach teachers and families as numbers: a standard score, a percentile rank, a confidence interval and a descriptive classification. The percentile rank is the percentage of the student’s peers in the norming sample (the representative group from which the instrument’s norms were derived) who scored lower than the student did; a percentile rank of 84 means the student outperformed 84% of them. Each of these numbers carries a specific meaning and has limits it cannot exceed, and misreading it leads to a mistaken decision: a percentile rank taken as the proportion of correct answers, as Umm Majed understood it in the opening scene; a score treated as an exact figure free of error; or a firm decision made when the confidence interval straddles the cut-off. In most categories, the diagnostic judgment rests on where the student stands relative to peers, on how precisely that position is estimated, and on whether the instrument suits its purpose and the student to whom it was administered.
We open the chapter by distinguishing norm-referenced from criterion-referenced measures (Section 2.1), then present the mean and standard deviation (Section 2.2), the normal curve, standard scores and their descriptive classifications (Section 2.3), and percentile ranks and their pitfalls (Section 2.4). We then take up reliability/precision and the standard error of measurement, together with the related matters of score differences, regression toward the mean and floor and ceiling effects (Section 2.5), followed by validity, sensitivity, specificity and fairness (Section 2.6), and then norms, standardization and test adaptation (Section 2.7). We apply all this to reading a realistic score table (Section 2.8), and close with ethics, professional responsibility and levels of qualification (Section 2.9).
Before we ask “How much did the student score?” we must ask “Compared with what?” A score on its own has no meaning. Eighty points may be excellent on a hard test and weak on an easy one. Measures are therefore divided into two main types according to what the score is compared with:
The student’s score is compared with the scores of a large group of same-age or same-grade peers, called the norming sample. The result tells us the student’s position among them: average, above average or below average.
The student’s score is compared with a predetermined level of performance. The result tells us what the student can and cannot do, regardless of peers.
The two types complement each other. Diagnosis usually calls for norm-referenced measures, because its question is “Is the student’s performance unusually far from that of peers?” Instructional planning, by contrast, calls for criterion-referenced measures, because its question is “What skill should we teach next?”
A single tool may combine both logics. One example is curriculum-based measurement, a set of brief, repeated probes used to track a student’s reading or mathematics week by week (defined and discussed in detail in Chapter 4). It serves as a general indicator of growth: its scores are interpreted normatively when compared with seasonal grade-level norms, such as fall, winter and spring percentile ranks, and at the same time against a specific growth goal set for the student. It thus draws on both frames of reference at once.
Saad, a fourth grader, obtained a percentile rank of 50 on a standardized reading test, placing him exactly in the middle of his peers. This is norm-referenced information, and it does not point to a problem. On a criterion-referenced test, however, his teacher noticed that he could not distinguish between letters that share the same shape and differ only in their dots, such as ب (bāʾ), ت (tāʾ) and ث (thāʾ). The first test does not reveal this, yet it is precisely what the teacher needs to know in order to decide what to teach tomorrow.
Every standard score is built on two numbers: the mean (Mean), the center of the norming sample’s scores, and the standard deviation (Standard deviation, SD), the degree to which scores are spread around that center. A small standard deviation means people’s scores lie close together; a large one means they are widely dispersed.
To make the idea of standard deviation concrete, imagine two classes of twenty students each, both with an average height of 140 centimeters. In the first class, heights range from 135 to 145; in the second, from 120 to 160. The mean is the same, but the spread differs. A student who is 150 centimeters tall would be considered very tall in the first class and fairly ordinary in the second. It is the standard deviation that tells us so.
This gives us the rule on which diagnosis rests: we measure how far a student lies from the mean in standard deviations rather than in points. A distance of two standard deviations from the mean signifies the same thing on any test, however its scores are calculated.
When we measure a trait such as intelligence or height in a large number of people, the scores usually fall into a bell shape: most people cluster near the mean, and their numbers dwindle the further we move from it in either direction. This shape is the normal curve (Normal curve), and among its properties are fixed proportions that do not change:
This last point is the origin of thresholds that recur throughout this book. The criterion of “two standard deviations below the mean” is used for intellectual disability, and “two standard deviations above the mean” appears in many definitions of giftedness. By this statistical criterion alone, then, each of the two groups represents about 2% of people at either end of the distribution.
Tests convert the raw score, the number of correct answers, into a standard score with a fixed mean and standard deviation. The benefit is that standard scores can be compared across ages and tests. The most widely used types are:
| Type | Mean | — | Where you will encounter it |
|---|---|---|---|
| z-score (z) | 0 | 1 | In calculations and research; it is the basis from which all other types are derived |
| Standard score | 100 | 15 | Composite and index scores on intelligence and achievement tests and adaptive behavior scales |
| Scaled score | 10 | 3 | Subtests of intelligence scales, and the subscales of the GARS-3 |
| T-score (T) | 50 | 10 | Behavioral and emotional rating scales, including many measures of attention-deficit/hyperactivity disorder (Attention-Deficit/Hyperactivity Disorder, ADHD) |
To convert any score to another type, we first convert it to a z-score and then from the z-score to the desired type:
On ability tests, a higher score means better performance. On many behavioral rating scales, however, a higher score means more problems. A T-score of 70 is excellent on a reading test and worrying on a hyperactivity scale. Always check the test manual or the report for what a high score means.
Tests translate the standard score into a verbal description that is easy to understand. The following example shows the classifications used by the Wechsler Intelligence Scale for Children, Fifth Edition (Wechsler Intelligence Scale for Children, Fifth Edition, WISC-V), together with the official English label for each category, since specialists frequently read reports written with these labels (Wechsler, 2014):
| Standard score range | Descriptive classification | Approximate percentage of people |
|---|---|---|
| 130 and above | Extremely high | 2% |
| 120–129 | Very high | 7% |
| 110–119 | High average | 16% |
| 90–109 | Average | 50% |
| 80–89 | Low average | 16% |
| 70–79 | Very low | 7% |
| 69 and below | Extremely low | 2% |
Three cautions apply to this table. First, the labels differ across tests, and even across editions of the same test. The fourth edition of this very scale called the 70–79 range “borderline” (Borderline) and anything above 130 “very superior” (Very Superior). So if you read the phrase Very High in a foreign report describing a score of 125, bear in mind that a higher category sits above it. Second, the “average” category here (90–109) is a descriptive classification specific to the Wechsler scales; it is narrower than the 85–115 range, which spans one standard deviation on either side of the mean and is the range used in this book’s figures. Third, and most important, a descriptive classification is not a diagnosis. The label “extremely low” describes only where the score falls; a judgment of intellectual disability requires other criteria, such as adaptive behavior, that is, the everyday living skills the student actually uses at home and at school (Chapter 7 addresses this in detail).
The percentile rank is, on the surface, the easiest score for families to understand, and it is also the one most prone to misunderstanding. It means the percentage of peers in the norming sample who scored lower than the student. A percentile rank of 30 means the student outperformed about 30% of his peers, and that about 70% of them outperformed him.
The percentile rank has nothing to do with the percentage of correct answers. A student who answered 80% of the questions may have a percentile rank of 20 if the test was easy and most of his peers answered more than that. This is the trap Umm Majed fell into when she thought her son had “passed with three percent.”
Because most people cluster around the mean, moving a few percentile ranks near the mean takes only a small difference in performance, whereas moving the same number of ranks at the extremes takes a large one. Compare these two moves, each spanning a similar number of ranks:
For this reason, percentile ranks should not be subtracted from one another or averaged, nor used to measure a student’s progress. If a student moves from the 5th to the 10th percentile, his progress may be far greater than the two numbers suggest.
Age equivalents (Age equivalent) and grade equivalents (Grade equivalent) express performance in terms of an age or a school grade. A teacher may read in a report that “the student’s reading age equivalent is 7 years, 3 months,” while the student is actually ten, and conclude that he reads like a seven-year-old. That reading is inaccurate. An age equivalent means only that the student’s raw score equals the average raw score of children of that age; it does not mean that his skills or errors resemble theirs. These scores are also unevenly distributed, and they do not show whether the gap is large relative to the student’s peers. This is why the Standards for Educational and Psychological Testing urge great caution in their use (AERA et al., 2014), and why standard scores and percentile ranks are preferred.
Reliability (Reliability) is the consistency of a measure’s results. If the test were given to the same student twice under similar conditions, would he obtain two similar scores? A scale that shows a different weight each time is useless, however accurate it may be on average. The Standards for Educational and Psychological Testing use the compound term “reliability/precision” (Reliability/precision) to cover traditional reliability coefficients as well as every other index of score precision, such as the standard error of measurement and the information function, which we present at the end of this section (AERA et al., 2014).
| Type | Question it answers | How it is calculated |
|---|---|---|
| Test–retest reliability | Does the score remain stable over time? | The test is given twice with a short interval between administrations, and the correlation between the two is calculated |
| Internal consistency | Do the test items measure the same thing? | Calculated from the item intercorrelations in a single administration; the best known index is Cronbach’s alpha (Cronbach's alpha) |
| Interrater reliability | Do two different raters assign the same score? | Two people rate the same student and their ratings are compared; this is especially important for observation and rating scales |
| Alternate-forms reliability | Do two forms of the test yield similar results? | Two equivalent forms are administered, and the correlation between them is calculated |
Reliability is expressed as a coefficient ranging from zero to one. Salvia and colleagues (Salvia et al., 2017) proposed widely used benchmarks: at least 0.60 when scores are used to describe groups, at least 0.80 for screening decisions, and at least 0.90 for important decisions about an individual, such as diagnosis and classification.
No measure is perfectly reliable. A student’s score on any given day is affected by fatigue, attention, guessing and the conditions in the room. The score we obtain is therefore an estimate of the true score, the score the student would have obtained had he been measured without any error. The standard error of measurement (Standard Error of Measurement, SEM) is a number that tells us how much the student’s score would be expected to fluctuate if we measured him many times over:
From it we build a confidence interval (Confidence interval): a range of scores within which we can say, with a given degree of confidence, that the true score lies. For a 95% confidence interval, we multiply the standard error by approximately 1.96, then add the result to the score and subtract it from the score.
A cut-off score (Cut-off score) is the score that separates two categories in a decision, such as “refer” versus “do not refer” on a screening tool, or “meets the criterion” versus “does not meet it” for a diagnostic criterion. We use this term throughout the book.
It is precisely for this reason that the American Association on Intellectual and Developmental Disabilities (American Association on Intellectual and Developmental Disabilities, AAIDD) calls for the standard error of measurement to be taken into account when determining significant limitations in intellectual functioning, so that 70 is not treated as a hard dividing line (Schalock et al., 2021). A score of 72 or 74 does not automatically rule out intellectual disability, and a score of 68 does not automatically establish it. What settles the matter is the convergence of evidence, especially evidence of adaptive behavior.
Every score in a diagnostic report is a point inside a band. The less reliable the measure, the wider the band. And when the band falls on both sides of the cut-off score, the decision becomes a professional judgment that rests on other evidence besides the number.
The way we calculated Majed’s interval is a simplified method that centers the interval on the observed score itself. Many modern tests, however, including the Wechsler Intelligence Scale for Children, center the interval on the estimated true score (Estimated true score) and use the standard error of estimate (Standard error of estimation, SEE) to do so (Sattler, 2018):
Because reliability is less than one, the estimated true score always lies closer to the mean than the observed score does. Applied to Majed, his estimated true score would be 100 + 0.95 × (72 − 100) ≈ 73.4 and the standard error of estimate about 3.3, so the interval would run from about 67 to about 80. The interval is thus skewed toward the mean rather than symmetrical around 72, yet it straddles the cut-off score of 70 under both methods. This is why readers see intervals in report tables such as “84: 78–92” and take them for a typo, when they are nothing of the kind (see Table 2.8). When you read a report, check the test manual for how its intervals were constructed before comparing them with the intervals of another test.
The Special Education Organizational Guide stipulates that among the conditions for a student’s eligibility for intellectual disability programs is “that the student’s IQ not exceed 75 on the Wechsler test, or 69 on the Stanford-Binet (Fifth Edition), or the equivalent of either,” together with limitations in at least two areas of adaptive skills. It also requires “not relying on the results of a single scale” and that instruments be administered and interpreted “by qualified specialists” (Ministry of Education, 1436 AH [Arabic]). The earlier Regulations had defined the “educable” range as an IQ between 55 and 75 on the Wechsler or between 52 and 73 on the Stanford-Binet (Ministry of Education [Al-Ma'arif], 1422 AH [Arabic]). The different figures in the Regulations can be understood in light of the different standard deviation of the older editions of the Stanford-Binet (16 rather than 15); both fall at roughly the same distance from the mean. In the current Guide, however, the current editions of the two scales share the same standard deviation (15), and we found no rationale in the text for the two different figures (75 and 69). Nor did we find any provision directing that the standard error of measurement or the confidence interval be taken into account when these figures are applied.
A question for reflection: A student scores 77 on the Wechsler scale, with a confidence interval of 72–83, and shows clear limitations in three adaptive domains. How should the team handle the cut-off score of 75 in light of what you have learned in this section?
Many diagnostic decisions rest on a difference between two scores: between ability and achievement in the discrepancy model, between two indexes on an intelligence test, or between a student’s scores before and after an intervention. Each of the two scores carries its own error, and the two errors combine in the difference. A difference is therefore less precise than either score, and it must be relatively large before we can be confident that it is real.
When we find a difference between two scores, we ask two distinct questions. The first concerns statistical significance: is the difference too large to be attributed to measurement error alone? The second concerns rarity, or the base rate (Base rate): what proportion of the norming sample itself shows a difference of this size or larger? A difference may be significant, that is, real rather than a product of error, and yet common among typically developing children, in which case it cannot be treated as a sign of a disorder. Good test manuals present the two in separate tables.
When the two scores share the same standard deviation, the standard error of the difference (Standard error of the difference, SEdiff) is calculated as follows:
The reliability of the difference score itself has a formula that shows the problem from another angle: rdiff = [(r₁ + r₂) ÷ 2 − r₁₂] ÷ (1 − r₁₂), where r₁₂ is the correlation between the two scores. If the two indexes had reliabilities of 0.94 and 0.92 and correlated at 0.60, the reliability of the difference would be (0.93 − 0.60) ÷ 0.40 ≈ 0.83, lower than that of either index. The higher the correlation between two scores, the lower the reliability of the difference between them. This is the root of the criticism leveled at decisions based on differences alone, as in the ability–achievement discrepancy model and in patterns of strengths and weaknesses models (to which Chapter 6 returns).
When people who obtained extreme scores are measured again, their average score the second time tends to move closer to the population mean. This statistical phenomenon is called regression toward the mean (Regression toward the mean). It occurs because an extreme score usually combines genuinely high (or low) ability with measurement error that happened to fall in the same direction, and that error does not necessarily recur the second time. The estimated true score we met in the previous “For the specialist” box is a direct application of this idea.
The phenomenon has practical consequences in two places. The first is giftedness: if we select students who scored above 130 on a test with a reliability of 0.90, the expected score of a student who obtained 135, on a retest with the same instrument, is about 100 + 0.90 × 35 ≈ 132, and it drops further if the second test is a different one that correlates with the first at only about 0.80 (to about 128). A decline in the scores of some selected students at reevaluation is therefore to be expected; it means neither that the original decision was wrong nor that the student has “lost” his giftedness (Lohman & Korb, 2006). The second is disability and learning disabilities: some of those selected for their very low scores will score higher when remeasured even without any intervention, so this gain cannot be credited entirely to the program in the absence of a comparison group.
A test’s floor (Floor) is its capacity to discriminate among very low levels of performance, and its ceiling (Ceiling) is its capacity to discriminate among very high levels. If a test has few easy items, a child with very low performance and a child with low performance will both receive the same minimum score, and the test can no longer tell them apart. Likewise, if there are few difficult items, high performers bunch up at the highest possible score. On many individual intelligence tests, the full-scale score range bottoms out at about 40 and tops out at about 160. That suffices for most children, but not in two situations: distinguishing levels of severity among the more severe cases of intellectual disability, to which Chapter 7 returns, and distinguishing among exceptionally gifted students, to which Chapter 12 returns. For this reason, some publishers have issued extended norms (Extended norms) for both tails of the distribution.
Closely related is the item gradient (Item gradient), the amount by which the standard score changes when the raw score increases by one point. If one additional correct answer moves a child from a scaled score of 4 to 7, the test is coarse in that range, and a small scoring error or a momentary lapse can change the whole classification. Bracken proposed minimum standards of technical adequacy that have since become widespread in the literature, among them that a raw score of 1 on a subtest should place the child more than two standard deviations below the mean if the floor is to be adequate, and that a one-point increase in raw score should not produce a change of more than a third of a standard deviation (Bracken, 1987). The problem is most pronounced with young children and with children at the youngest or oldest ages a test covers. If a child is at the edge of a test’s age range, examine the norm tables before trusting the score.
Everything we have presented so far belongs to classical test theory, which assumes a single standard error for all scores on a test. Most modern tests, however, were built, wholly or in part, within another framework: item response theory (Item Response Theory, IRT). Teachers need only two ideas from it: that a test’s precision varies across ability levels and is highest where its items are concentrated; and that this is what makes computerized adaptive testing possible, in which the computer selects for each student the items suited to his level.
The theory describes each item by an item characteristic curve (Item characteristic curve), which links the student’s ability, denoted θ and expressed on a scale resembling the z-score, to the probability of answering the item correctly. The curve’s most important features are two parameters: difficulty (b), the ability level at which the probability of a correct answer reaches 50%, and discrimination (a), the slope of the curve at that point. A third parameter for guessing may be added for multiple-choice questions (Embretson & Reise, 2000).
From the curve we derive the information function (Information function), which shows how much the item contributes to the precision of the ability estimate at each level. What all the items contribute is summed in the test information function, and the standard error of the ability estimate at any level equals the inverse of the square root of the information at that level. A test whose items are concentrated around the mean therefore measures average students with high precision and the two tails with less, which is a more precise formulation of the floor and ceiling problem discussed above. A computerized adaptive test (Computerized adaptive testing, CAT) begins with an item of moderate difficulty, estimates the student’s ability after each response, and selects the next item from the item bank so as to yield the most information at that estimate. It thus reaches acceptable precision with fewer items, and each student faces items close to his level. The validity of this approach, however, depends on calibrating the item bank on a sample resembling the students we are measuring, a requirement that brings us back to Section 2.7.
If reliability asks, “Does the scale measure consistently?”, validity asks, “Does it actually measure what it is meant to measure? And are the decisions we base on its scores sound?” In the modern view, validity is a property of the interpretation of a scale’s scores for a particular purpose, and it changes with the purpose and with the person tested. A scale that is valid for screening three-year-olds may not be valid for diagnosing an adolescent. The Standards for Educational and Psychological Testing treat validity as a single, unified concept, sometimes called construct validity: the degree to which the evidence supports interpreting the score as representing the intended construct. The construct is what all the evidence, taken together, attests to. The Standards draw this evidence from five sources (AERA et al., 2014):
| Source of evidence | Question | Example |
|---|---|---|
| Content | Do the items cover the facets of the construct adequately, and do they suit the setting in which they are used? | An autism spectrum disorder scale whose items cover the two core domains in the diagnostic manual: difficulties in social communication and interaction, and restricted, repetitive behaviors and interests. For the Arabic version, local experts review each item: does it describe a behavior one would expect to see in a Saudi home or school? |
| Response processes | Do test takers or raters actually engage the processes the construct assumes when they respond? | Saudi children are asked to think aloud while solving the items of a reasoning test, or parents are asked what they understand by a statement on a rating scale. If it turns out that a child fails because he does not know a Standard Arabic word in the question, the item is measuring vocabulary, not reasoning |
| Internal structure | Do the items and dimensions relate to one another as the theory predicts? | Factor analysis of the Arabic version: does it retain the structure of the original? A factor analysis of the Arabic version of the Childhood Autism Rating Scale arrived at a three-factor model of 14 items (Alotaibi & Alotaibi, 2021), and an Omani study of the Adaptive Behavior Assessment System (ABAS-3) supported a single general-factor model among children with intellectual disability (Emam et al., 2020) |
| Relations to other variables | Do scores correlate with what they are expected to correlate with, predict the criterion, and distinguish between groups? | The Saudi version of the Wechsler Preschool and Primary Scale of Intelligence, Third Edition, correlated with the Saudi version of the Stanford-Binet at 0.81 (Ejbara, 2024). In a validation of the Arabic version of the Modified Checklist for Autism in Toddlers across nine Arab countries, sensitivity reached 0.86 and specificity 0.80 (Seif Eldin et al., 2008) |
| Consequences | What are the effects, intended or unintended, of using the scores in decisions? | Does using a scale for nomination to gifted programs lead to lower-than-expected representation of students from villages or of a particular group? (Chapter 14 returns to this question) |
The older division into “content validity,” “criterion validity” and “construct validity” still appears in many test manuals and Arabic textbooks. In that scheme, “criterion validity” corresponds to part of the fourth source, while “construct validity” corresponds to validity as a whole in the modern view. The second and third sources, however, are the most important questions we ask when evaluating an Arabic adaptation: Does the Arab child understand the item as intended? And has the adaptation retained the structure of the original?
For screening and diagnostic tools, validity evidence based on relations to other variables is usually reported through two concepts: sensitivity (Sensitivity), the proportion of those who have the condition whom the tool correctly detects, and specificity (Specificity), the proportion of those who do not have the condition whom the tool correctly rules out. Yet two other figures matter more to teachers and families. The first is the positive predictive value (Positive predictive value): if a student's result is positive, how likely is it that he actually has the condition? The second is the negative predictive value (Negative predictive value): if his result is negative, how likely is it that the condition is truly absent? This second figure is the most important question in screening, because a negative result means the student will not be followed up.
These two probabilities depend jointly on the tool and on how prevalent the condition is in the group being tested. Consider a hypothetical example with illustrative figures: a screening tool with 85% sensitivity and 95% specificity is administered once to 1,000 children in a universal screening where the condition has a prevalence of 2%, and once to 200 children referred because of concerns, among whom the prevalence is 30%:
| Universal screening (1,000 children, prevalence 2%) | Referred group (200 children, prevalence 30%) | |
|---|---|---|
| Have the condition | 20 | 60 |
| True positive (detected by the tool) | 17 | 51 |
| False negative (missed by the tool) | 3 | 9 |
| Do not have the condition | 980 | 140 |
| False positive | 49 | 7 |
| True negative | 931 | 133 |
| Positive predictive value | 17 ÷ 66 ≈ 26% | 51 ÷ 58 ≈ 88% |
| Negative predictive value | 931 ÷ 934 ≈ 99.7% | 133 ÷ 142 ≈ 94% |
In universal screening, roughly three-quarters of the positive results are false, yet a negative result is highly trustworthy. In the referred group the picture is reversed: most positive results are correct, but about one in every sixteen children with a negative result actually has the condition. This is why a positive screening result is not a diagnosis, and why a negative result in a high-risk group is not sufficient reason to close the file.
The Standards for Educational and Psychological Testing make fairness (Fairness) a third foundation of measurement alongside validity and reliability/precision, devoting their third chapter, “Fairness in Testing,” to it (AERA et al., 2014). Fairness in this sense is at heart a question of validity: Does a score mean the same thing for a student from a different linguistic, cultural or geographic group, or for a student with a disability? A failure in this respect is called measurement bias, and it takes several forms:
| Form | What it means | How is it examined? |
|---|---|---|
| Construct bias | The construct itself is not defined, or does not manifest, in the same way in the two groups, as when “appropriate social behavior” is measured by the standards of a single culture | Expert review from both cultures, and comparison of factor structures |
| Differential item functioning (Differential item functioning, DIF) | Two students of equal ability from different groups do not have the same probability of answering a particular item correctly | Comparing the two groups' performance on each item after matching them on total score or estimated ability; experts then review the items on which a difference appears |
| Measurement non-invariance | The relationship between the items and the construct differs across the two groups, so their scores cannot be compared directly | Sequential tests in multigroup confirmatory factor analysis: equivalence of structure, then of loadings, then of intercepts |
| Predictive bias | The same score predicts different later performance in the two groups | Comparing the regression lines of criterion on score in the two groups: a difference in slope or intercept is evidence of bias |
A difference in group means may be real; the two groups may genuinely differ in the measured skill for reasons related to opportunities to learn, and the scale may be describing that difference accurately. Bias is when the same score means something different depending on the group. Nor is examining the instrument enough: fairness also includes making the construct accessible to every test taker, as when a student with low vision is given the test in large print, provided this does not change what it measures. Fairness in the referral and identification stages, and indices of proportional representation, are taken up again in Chapter 14.
A standard score means “the student's position among his peers in the norming sample.” The norming sample (Norm sample) is the group to which the scale was administered to derive its norms. If the norming sample does not resemble the student, the score loses its meaning. We therefore ask three questions of any instrument before using it:
To simplify the path an instrument travels from one culture to another, this book distinguishes three levels. The division is the author's own teaching simplification, built on what international guidelines require (see the next box):
| Level | What happens | What can be done with the scores |
|---|---|---|
| Translation | Items are rendered into Arabic literally or nearly so | The original norms cannot properly be used to judge the child |
| Test adaptation | Items and examples are modified to suit the culture, with back-translation (translating the Arabic version back into the original language and comparing it with the original), expert review, item analysis, and verification of measurement invariance across the two versions and of the absence of differential item functioning (see 2.6.1) | Better than translation, but the original norms remain unsuitable |
| Local standardization | The adapted version is administered to a representative sample of the local population, new norms are derived, and reliability and validity are studied in that population | Standard scores become meaningful for children in that population |
The second edition of the International Test Commission Guidelines for Translating and Adapting Tests consists of eighteen guidelines grouped into six categories. Unlike the transfer levels in our table, they follow the stages of the work (International Test Commission, 2017):
Local standardization in our table thus corresponds roughly to the completion of the “confirmation” and “score scales” stages, together with the derivation of new norms. For a version that has not gone through the confirmation stage, a well-crafted translation is not enough.
The instruments used in the Kingdom vary widely in where they stand among the three levels. Among the published examples we found:
A question for reflection: Which of these examples reaches the level of “local standardization” in Table 2.7? And what does each one lack to be suitable for a student in your city?
If the answer to most of these questions is no, use the scale's results with great caution, confine them to description rather than diagnosis, and state this explicitly in the report.
Intelligence test reports usually present a table summarizing the composite scores. Table 2.8 presents hypothetical values for a ten-year-old student, modeled on the indexes of the Wechsler Intelligence Scale for Children, Fifth Edition:
| Index | Standard score | Percentile rank | 95% confidence interval | Descriptive classification |
|---|---|---|---|---|
| Verbal Comprehension | 84 | 14 | 78–92 | Low average |
| Visual Spatial | 102 | 55 | 94–110 | Average |
| Fluid Reasoning | 98 | 45 | 91–105 | Average |
| Working Memory | 76 | 5 | 70–85 | Very low |
| Processing Speed | 80 | 9 | 73–91 | Low average |
| — | 86 | 18 | 81–92 | Low average |
The confidence intervals in the table are built around the estimated true score, as in the original scale, which is why they are not symmetrical around the score (see the “For the specialist” box in 2.5). We assumed the following reliability coefficients: 0.94 for Verbal Comprehension and Fluid Reasoning, 0.92 for Visual Spatial, 0.93 for Working Memory, 0.90 for Processing Speed, and 0.96 for the Full Scale IQ (Full Scale IQ, FSIQ). How, then, should a teacher read this table?
School psychology has an ongoing debate about the value of profile analysis, that is, interpreting the pattern of strengths and weaknesses across indexes. On one side, critics argue that the indexes add little to the full scale score in predicting achievement, that the reliability of differences between them is much lower than that of the full scale score, and that patterns have not proven useful in selecting a particular instructional intervention (McGill et al., 2018). On the other side, its defenders argue that disciplined interpretation of the indexes, grounded in differences that are significant and rare and supported by evidence from outside the test, helps in understanding the student and in guiding subsequent assessment, particularly with twice-exceptional learners, whose uneven abilities the full scale score may conceal (Kaufman et al., 2016). The position this book takes lies between the two: begin with the full scale score and the limits of what it tells you, move to interpreting the pattern only after the differences have been examined statistically, and treat the pattern as a hypothesis requiring further evidence. Chapter 6 returns to this debate in discussing patterns of strengths and weaknesses models.
Those who assess children answer to two kinds of references that should not be confused. The first is a technical reference, chief among them the Standards for Educational and Psychological Testing, issued by three American associations (AERA et al., 2014). The Standards specify the validity, reliability/precision and fairness that tests and their use should meet, and devote their third chapter to fairness in testing. In international practice they are complemented by the International Test Commission's guidelines on test use, which set out the competencies needed by test users (International Test Commission, 2001). The second is the codes of ethics that professional bodies issue for their members, among them the American Psychological Association's Ethical Principles of Psychologists and Code of Conduct (American Psychological Association, 2017), the National Association of School Psychologists' professional standards, which include its principles for professional ethics (National Association of School Psychologists, 2020), and the Council for Exceptional Children's ethics and professional practice standards (Council for Exceptional Children, 2015). Among the principles on which these references agree, several bear directly on the teacher's work:
Among these principles lies a tension that recurs in practice: the family's right of access versus test security. The family has the right to know everything related to their child's assessment and program, and to request re-diagnosis if they believe he has not been placed in the appropriate program (Ministry of Education [Al-Ma'arif], 1422 AH [Arabic]). On the other hand, copying and handing over test items exposes the test to circulation and may violate the publisher's rights. The solution toward which professional codes lean is to explain the results to the family in detail and to show them their child's responses and how they were scored in the presence of the specialist, without handing over the test booklet or copying its items.
Many publishers classify their instruments into qualification levels that determine who may purchase and use them. The details vary from one publisher to another, but the general idea is the same: a first level for tools that teachers and others can administer after reading the manual; a second level requiring training in measurement or a professional license; and a third level for complex tools such as individual intelligence tests and diagnostic instruments, which requires graduate study in a specialized field or an advanced professional license. The Wechsler Intelligence Scale for Children is an example of this last level. What the Saudi documents we reviewed stipulate is summarized in the following table:
| Role | What the documents stipulate | What we found no provision for |
|---|---|---|
| General education teacher | Member of the school's assessment team (Ministry of Education, 1436 AH [Arabic]) | The tools he is entitled to administer. His practical role is referral, observation and informal tools |
| Special education teacher | Member of the assessment team (Ministry of Education [Al-Ma'arif], 1422 AH [Arabic]; Ministry of Education, 1436 AH [Arabic]) | A list of the standardized scales he administers |
| Psychologist in schools and institutes | The Regulations, under the title “behavioral training teacher (psychologist),” assign him assessment and diagnosis using standardized intelligence scales, adaptive behavior scales and informal methods, as well as re-diagnosis when needed (Ministry of Education [Al-Ma'arif], 1422 AH [Arabic]). The Guide requires that tools be administered and interpreted by “qualified specialists” and that they be “approved by the Ministry of Education” (Ministry of Education, 1436 AH [Arabic]) | A published list of approved scales, and the qualification criteria required for each scale |
| Psychologist in the health sector | Classified and licensed by the Saudi Commission for Health Specialties. In the edition of the classification manual we reviewed, a bachelor's degree holder is classified as an “assistant specialist” working under supervision, and is classified as a “psychology specialist” after no less than three years of clinical experience (Saudi Commission for Health Specialties, 2009 [Arabic]). The psychology licensing examination assesses, among other things, the application of knowledge in psychological assessment and diagnosis (Saudi Commission for Health Specialties, 2024 [Arabic]) | The current edition of the classification manual |
| Physician | Participates in the school's assessment team “if available” (Ministry of Education, 1436 AH [Arabic]) | A general provision in education regulations specifying the diagnoses reserved for physicians alone |
We found no published provision in the Kingdom corresponding to the qualification levels publishers apply to the purchase and administration of standardized psychological tests. Until such a provision is issued, the practical reference remains the publisher's own requirements, together with the principle of boundaries of competence described above.
The first dilemma: consent by silence. The Organizational Guide states that if the guardian does not respond within a period not exceeding fourteen days from the date of written notification, “this shall be considered permission and consent” (Ministry of Education, 1436 AH [Arabic]). The provision protects the student's right not to have his assessment delayed because a notice was ignored. Yet the principle of informed consent assumes that the family has understood what it is consenting to. What should the team do if it learns that the father cannot read, or that the notice was sent home with the student and never arrived? The provision permits the team to proceed; ethics requires attempting to reach the family by other means, and documenting that attempt, before settling for silence.
The second dilemma: the mother who asks for the items. A mother asks for a copy of the booklet of the intelligence test administered to her son, because she wants to “train him on the questions he got wrong” before his reassessment next year. The mother has a right to access and understanding, and the test has a security that protects the validity of her own son's results next time. How would you word a reply that respects both rights?
The teacher can now answer Umm Majed's two questions along these lines: “The number 3 is not a passing grade. It means that Majed's performance on this test is higher than that of about 3% of his peers of the same age. In other words, he needs clear support. As for your question about disability, the test alone cannot answer it. His score is close to the score used in making the decision, and the band around it extends on both sides, so the team will gather information about his everyday skills at home and at school before reaching any judgment.”
What we have presented in this chapter rests on a single unifying idea: a score is an approximate estimate of the student's position among his peers. Standard scores of every kind are different languages for one position on the normal curve; a percentile rank is the proportion of peers the student outperformed; every score is surrounded by error whose extent the confidence interval shows; and the difference between two scores is less precise than either one. It follows that a decision in which the confidence interval crosses the cut-off score is a professional decision requiring other evidence; that a difference between two scores supports interpretation only if it is both significant and rare; and that an extreme score can be expected to regress toward the mean on retesting.
Precision alone is not enough. Whether a score suits its purpose is a question of validity, whose evidence is gathered from multiple sources, and whether it suits the student is a question of norms and fairness: norms are appropriate only if their sample resembles the student and is recent enough, and an accurate translation carries over neither the validity nor the norms of the original. Whenever an assessor uses a translated tool or norms from another setting, one question remains that no calculation can settle: Does the norming sample resemble the student in front of me? The answer depends on the technical evidence available and on the limitations the report states openly.
Noura is thirteen and in the first year of intermediate school. Her teacher referred her because of marked underachievement over several years. She was given an Arabic-adapted individual intelligence test whose norms were collected about eighteen years ago on a sample from a region other than her own. Her full scale score was 74, with a full scale reliability coefficient of 0.96. Her Verbal Comprehension index was 86 (reliability 0.94) and her reasoning index 72 (reliability 0.90). An adaptive behavior form completed by the teacher alone indicates a deficit in one domain. One team member holds that “the score is below 75, so the student is eligible for the intellectual disability program.”
Calculate the confidence interval for the full-scale score and the standard error of the difference between the two indexes. Then reach a judgment: is the evidence sufficient for an eligibility decision? Support your judgment with at least three points you have learned in this chapter, and name the missing evidence you would request before deciding. (Objectives 4, 6, and 7)