Testing. It’s become so much part of the life of a learner or a teacher, at any age. And it’s a fascinating topic.
Okay, I’m one of those weird people who thinks of test-taking myself as a sort of competitive athletic event, one at which I’m really quite good even while thinking that the vast majority of tests I’ve ever taken were nearly completely pointless. No, that’s not my impostor syndrome kicking in. It has to do with a central concept in test design, which I’ll explain below.
What I love most about assessment is how useful it can be when done well. One of my colleagues says that testing doesn’t bring out the best in people, it doesn’t bring out the worst in people, but it brings out the most in people. We put you in a situation where your normal compensatory strategies for getting along in the world aren’t going to work. As Peter Ossorio says, when you ask a person to do something they can’t do, they’ll do something they can do. You’ll figure out something to do, the best you can, and what you do will be a reflection, in some way, of you. It’s like science — each test is an experiment that you and I do together. No one bit of data proves anything by itself, but when we put things together and look for themes, consistencies, divergences, a story begins to emerge, and it often does so surprisingly quickly.
But what bugs me is how little most folks understand about tests of all stripes — most importantly, how they’re built, how they work, what they’re good for… and what they aren’t. So what I’d like to do is to kick off a random-access series of posts on various aspects of assessment, including ordinary classroom tests, high-stakes testing for No Child
Left Behind Allowed Ahead (also known as No Teacher Left Standing) and other similar “accountability” movements, bubble tests like the dreaded SAT and its ilk, and, of course, my favorite, the one-on-one kinds of tests used for special education and other diagnostic work, the kind that seriously geeky people like me give. Those include cognitive tests, neuropsychological tests, academic tests, psychological tests, behavioral questionnaires, and other fun stuff. I’ll start there because, well, because I like them and I think they’re really pretty interesting. I’ll try to chew off manageable chunks to talk about, and over time, I hope people learn something.
The most serious and popular misconception I encounter is a fundamental misunderstanding of what tests can do. They’re not magic, and neither are those of us who give them magicians. We’re just very observant (or at least we’re supposed to be!), and we’re using them to make a series of structured observations.
Again, this is like science. When I was training as a molecular biologist, one of the things I had thwacked into my head (through reading in the literature some of the truly impressively weird things that happened when people didn’t remember it) was that no experiment ever tells you anything about the real world. It tells you what happened on that day when that person did that experiment in that way. You might use that information to conjecture about the nature of the real world based on your data, and over time, as you build up more data, you can get a better and better sense of what the real world might be like. But you might see a different experiment, claiming to answer the same question, where you get different results. Uh, oh. Where do you look, to figure out what was going on to find the difference that made the difference? In the Materials and Methods, the specifics of how the experiment was designed and constructed. Very often, that’s where the difference lies. You cannot separate data from the experiment that generated it.
Same with assessment. No test, no matter how beautifully it’s designed, how skillfully it’s administered, and how insightfully it’s interpreted, can possibly tell you anything incontrovertibly true about the real human being. The test tells you what that person did on that day on that test with that tester in that environment. It might reflect something probably true about the person, but you have to stay humble with your interpretation.
Since you will always value what you measure, it makes sense to think very carefully about how to measure what you actually value. In education, we talk about the idea of “alignment” — we’d say that this test is or is not well-aligned to the skills we want the student to be able to demonstrate. That’s what I was talking about above, why I don’t respect the very bubble tests that I tend to be able to blow out of the water. They typically test what is easy to measure, but not what a thoughtful professional would consider all that valuable. At the conclusion of many thousands of hours of clinical training, psychologists in most states have to take a detailed fact-recall bubble test covering basically the entire field. We to prove that we know which classic theorist suggested that you were running from the bear because you were afraid, versus which one suggested that you were afraid because you were running from the bear. But we don’t have to demonstrate the capacity to actually manifest any clinical competencies with actual, oh, I dunno, human beings in distress. In test design, we talk about the very-closely-related concept of “validity,” which comes in many flavors. In this case, the construct validity of the test — how it defines what it is that it’s trying to measure — is awful. Fact knowledge within a domain is a useful thing, and might be a good prerequisite to beginning clinical work. But the public is not protected from incompetent psychologists by choosing only those who can remember the facts printed in their textbooks.
I think the best-aligned test I ever took was the qualifying exam for the Ph.D. I didn’t get in cancer biology. I was required to dive in to fields I was unfamiliar with, learn about the prior research in those fields, and propose new lines of research that would answer important unanswered questions. Minus the speed with which I had to do it (three of these, in completely different fields, within a single week!), this test was testing very much what I would need to do if I became a principal investigator running my own lab someday. Of course, the alignment/construct validity of that test wasn’t perfect either. What it didn’t explore was the personality traits which set me up to be a very sad and bored and frustrated person in the lab, the precise difference between thinking about science, which I love and am good at, and doing bench science on a day-to-day basis, which I don’t and am not.
What I find most concerning about the high-stakes testing (aka “accountability”) movement in education is that it tends to use tests with poor validity in a variety of domains (construct validity, content validity, and predictive validity being the most notable), and that it tends to ignore other underlying methodological differences between comparison groups (most notably, differences in the populations being served and the resources available to teachers and administrators to serve them, but also differences in how various jurisdictions define their goals and standards). When science teachers teach kids about experimental controls, we start with the idea of a “fair game.” But there’s no way on earth that these “games” are fair. There’s nothing truly “standardized” about these experiments, and almost every interpretation that is made of them is a massive overinterpretation from inadequate data. Gives serious testing a bad name. Harrumph.
Okay, so my plans for this series of posts right now involve topics like the various types of validity and reliability (the twin pillars of assessment for people who actually want usable data!), and a sort of overview of each of the major types of clinical testing (e.g., cognitive, academic, neuropsychological, behavioral, projective) and what they are and aren’t good for. I’ll do classroom and educational and high-stakes stuff later, but I’d rather start with what I do the most of. If there are specific ideas or questions you’d like me to address, feel free to drop them in the comments area here.
I’m not usually political on this blog, but… (thoughts on mental illness and culpability of public figures)
Oh, look, here we go again.
Public figures, folks in positions of leadership and authority, present their violent fantasies, talking about their political enemies as evil, dangerous, deserving of death, etc. They often suggest methods that would be way-cool, too, and talk about how great it would be to “take decisive action” or somesuch. Over and over again. In loud voices, using all of their charisma.
(I will refrain from calling out specific speakers and specific incidents here, because it is wrong no matter who does it, and because I do not want this blog to degenerate into a pointless debate about the minutiae of precisely who said precisely what and whether that counts. The specific tragedy that occurred today is just the latest instantiation.)
Lots of folks say, “Oh, that’s awful! Don’t go around inciting violence!”
The speakers respond, “Oh, come on, can’t you take a joke? I was only speaking figuratively! That’s just political rhetoric! I’m not really telling people to go out and do those things! And besides, other people are doing it, too! And I’m not really an authority figure anyhow, because, after all, I’m just an entertainer, or a candidate, or a humble religious leader, or a Citizen Just Like You.” Right, because we all happen to have hundreds, thousands, or millions of people listening to us on the TV/radio/internet/lectern/pulpit.
Then someone does something awful: blows up a building, shoots a bunch of people, or otherwise takes violent action that looks rather like what that authority figure fantasized about.
And everyone is shocked! shocked, I say! Everyone, especially including the public figures themselves, decries the violence and does their best to distance themselves from the person who actually did the bad thing. The good news is that when we learn more about that person, we find out that they were a “crazy” “lone wolf.” They’ve often posted rambling and incoherent monologues on the internet or left other clear evidence of serious and persistent mental illness. Everyone titters and points and does their level best to say, “That person is nothing like me,” because thought disorders are scary. (I’m not being sarcastic. The idea that your own brain might turn on you is legitimately terrifying. But when we’re scared of something, one of our normal and natural defenses is to try to make it be as separate from us ourselves as possible.)
So hooray! The public figures are off the hook! They couldn’t possibly have predicted that some whack-job would have taken them seriously and done that awful thing. Those people aren’t like us. They can’t be held responsible for what those not-like-us-people might do, even if it was disturbingly like what they were talking about on the TV/radio/internet/lectern/pulpit.
Folks, the population incidence of schizophrenia is approximately 1%. One. per. cent. Think about that for a minute. Think about going into a movie theatre… or a house of worship… or a football game… Now think about the population of the country (or the world). Move the decimal place two spots to the left. That’s a lot of people struggling with serious and persistent mental illness, typically in overburdened systems that rarely manage to provide the kind of help they need. I’ve worked with folks who are seriously affected by these disorders — I have a great deal of compassion for them.
Let me be crystal clear — not all people with thought disorders are violent. In fact, the research data is quite clear that the overwhelming majority are not. (In fact, they are no more likely to be violent than the general public, although most people massively underestimate how violent the general public is). Even those few who are dangerous are rarely dangerous to people they don’t know. (That’s true of the general population, too, by the way — the overwhelming majority of victims of violence know their attackers well… they’re often closely related to them.)
But “knowing” someone in this case can include being introduced to them by, say, a public authority figure who talks about them a whole lot and tells you that they know all about this person who is evil and they know that this person should be killed and they create a concrete image of how that could be done and they repeat the message over and over again or have lots of friends who repeat similar messages. Especially if part of your thought disorder includes the relatively common symptom of believing that the TV/radio/internet/movie/music has a special message just for you. When you’re having a hard time holding onto reality and making it make sense, then those nice, simple, consistent messages getting repeated over and over can feel comforting.
While I’m on the subject of relatively-common-symptoms of thought disorders, let me also point out that the belief that you are somehow important, special, have a special mission to carry out, have to sacrifice yourself, have to save the world, etc… is also on the list. Ahem.
Folks like to hide behind the “abstract language” thing. I can’t agree. You may speak abstractly, sure, but you’re speaking to lots of people in the population who are not abstract thinkers. (Think about how frustrating it can be at the DMV, or with the TSA, or on a telephone support line, or any of the other situations that provoke the typical Xtranormal video.) And when someone has a thought disorder, they often become highly concrete and not-quite-logical in how they process language. Words don’t quite mean what they usually mean, sounds start meaning more than the words, sentences can start in one place and end someplace very different, language and logic can start to feel like one of those water-snake toys that keeps slipping out of your hands. Understanding the niceties of figurative language and hyperbole and rhetorical flourish from the public authority figure on the TV/radio/internet/lectern/pulpit, figuring out what they really mean…? Go ahead and look up the writings of any of these “lone crazies” and tell me if you think a person who has that little control over language and thought can tell when an authority figure’s comments about the nobility of sacrifice and the necessity of violence and all that are really just entertaining and clever words, and when they’re concrete calls to real-world action. I like to think I’m a good consumer of the subtleties of language, and I am often unsure of what the shouting heads really mean.
The saving grace is that most of us, most of the time, have an observing ego. We think about what we’re going to do at least a little bit before we do it, and we judge our planned actions in light of whether they conform to the usual rules of the culture, what the likely outcomes would be, and so forth. And most of us, most of the time, have enough executive functioning to inhibit the acting out of the planned actions we judge to be poor choices. When I hear about something despicable, when I’m angry or scared, I might think, “Gee, I wish I could just… <fill in some random violent fantasy>.” But I don’t then actually do it… at least not most of the time (grin). But serious mental illness can wreak havoc with those self-controls.
What really concerns me is when the same authority figures who claim that they couldn’t possibly have known that they were sooo powerful… seem to feed their own sense of power by watching other people (who have less in the way of observing ego and executive function) carry out their own violent fantasies. I have known a few individuals who seemed to thrive on the chaos they caused within a community. Metaphorically, they would throw bricks high up in the air. When the “brick” came down and hurt someone or caused some other form of contention, they’d be as shocked as everyone else — perhaps more so. But there was also the sly smile, the subtle recognition of their own power to have caused that. Heh. I wonder if some of these folks who seem so often to step over the lines of appropriate authority-figure behavior are being repeatedly reinforced by how much crisis, both actual and feared, that they cause. That’s a problem, because even telling them how powerful they are and reminding them that with power comes responsibility feeds the narcissism.
I’m not sure how we as a civil community can address that effectively, not when the ranting is what makes money. To a great extent, I think we all have to get serious with ourselves about how our own fantasies are being fed by the violent talk. Personally, I’ve noticed a few shows that I enjoy and typically agree with politically, but I start to feel that I am getting too much pleasure out of the implied combat. When I stop liking who I become when I listen to them, I vote with my ears and whatever ratings statistics I might happen to be contributing to. I have stopped listening to them.
So all this is why I can’t accept that “no one could have predicted.” No one could have predicted precisely which person would react to precisely which turn of violent-fantasy speech or imagery in precisely which way at precisely which time and kill precisely which people. The population is too large to be keeping tabs on every person to the level that would permit experts to make such predictions, and I personally would not want to live in a country that kept tabs on its citizens in such a fashion. But the idea that someone would react in some violent way at some point was extremely predictable. And in fact, in most of these cases, as in the tragedy this morning, the prediction was in fact made and ignored, made and pooh-poohed, made and shouted down. And in most of these cases, there was that little “heh” coming from the background as those who agreed with the violent fantasy got the pleasure of having their fantasy gratified by someone else.
Those who have the attention of the public, on the TV, shouting on the radio, posting on the internet, ranting from the secular lectern or preaching from the religious pulpit, all have a responsibility. The more people who listen to you, the more you tell them what to do, the more responsibility you have when they act on what you tell them. Some of the blood spatters on you, too.
I just read this post:
and I think it’s really worth reading. So often, I think, parents have had their own educational or social traumas, or they’ve heard countless stories about the problems and pain other people have had, that they come in assuming that all teachers are dumb, hostile, clueless, nasty, evil, you name it. In the shrink biz, we’d call this a transference phenomenon, where your prior experience creates a distorted lens through which you view your current experience. Quite naturally, you expect that what happened before is going to happen again. But that often leads people to behave in ways that create the very problems they are afraid of happening. Thus, the trauma becomes re-enacted, proving, of course, that you were right to expect that sort of thing to happen, and continuing the cycle.
Are a lot of teachers clueless about giftedness? You bet. Are some of them hostile? Yes. But they’re not the norm. The overwhelming majority of educators do it because they like kids and because they like teaching. (After all, the pay is pretty crappy and the working conditions are terrible.) They don’t get up in the morning thinking, “Okay, how do I ruin little Freddie’s life today?” Build the relationship and see if you can find a different ending to the story.