Regarding my statement: "All tests are biased."
I say this because I feel that there are too many factors and variables that can contribute towards an unaccountable adjustment (difference) in ability, knowledge, or performance when comparing two individuals, or two groups results. To elaborate, factors include but are not limited to, environment, cultural background, language, and/or socio-economic status (I'm hesitant to use that one).
Glen made an interesting point: "All tests should be biased. They should be biased towards the people who have the knowledge." This seems to have an effect on the reliability of the test, though.
Subscribe to:
Post Comments (Atom)
Your first paragraph is correct. There is no such thing as a perfectly unbiased assessment. Even if there was a perfect assessment protocol (or a perfectly unbiased test), bias could still be introduced via the interpretation of the results.
ReplyDeleteConsider our tape-measure activity. While the instrument was free from bias (it measured males and females with equal accuracy), a woman standing 5'8" could be interpreted as "tall" while a male of that same height would be seen as “average.” So even an unbiased assessment can become biased when we interpret the result.
As I said in class, it's not that we will ever totally eliminate bias, but we must take reasonable action to reduce it to an acceptable level. Saying that “no bias” is the only acceptable level of bias is like saying 1.0 reliability is the only acceptable level: It ain’t gonna’ happen.
In your second paragraph, I think I see how you connected bias to low reliability: Bias is systematic error. (Error being the difference between the true and observed score.) More bias = more error = less reliability.
But Glen slightly misspoke. (He was paraphrasing something I said in class.) What he should have said is that every test should *discriminate* between those who know the material (or can complete the tasks) and those who don't (or can't).
Discrimination is not the same as bias. ← I think this is the main sticking point.
The more *consistently* that an assessment can sort those two groups, the *higher* the reliability. Remember error is the difference between true and observed scores. If the test perfectly discriminates between high and low performers, the observed score *is* the true score, which means zero error and perfect reliability.